Skip to contentSkip to conversation

Update to Debian stable (stretch)

PATCH In review Ian Jackson 6 participants 76 messages Apr 10, 2019 – May 1, 2019

lore
Share & tools
lore
#0 lore raw

cover Update to Debian stable (stretch)

This, finally, is the series to update osstest to Debian stretch. This is very urgent because Debian have been removing parts of jessie from their servers, in a rather unhelpful and not entirely controlled manner. Consequently, currently osstest is not reliable with jessie.

I am going to push this series to osstest pretest immediately without waiting for acks/reviews/etc. I expect it to trip two regressions in the osstest push gate, which I will force push.

One effect of this series is to put into service the two ThunderX arm64 machines, rochester[01], which are supportable in stretch but not jessie.

There seem to be a number of outstanding problems which I decided were not blockers:

* test-amd64-amd64-xl-qcow2 fails because the xvd driver fails to freeze during the Linux guest suspend. This seems to be a real bug of some kind and needs investigation.

* I experienced difficulties with the 4 Arndale devboards: high probability guest start failures. For now I have marked those nodes as unsuitable for use with stretch, which will, effectively, take them out of service - and leave us with a lack of armhf capacity. It is possible that this problem is due to the ifupdown-hotplug issue, now addressed, so I plan to retest.

* The serial output from the new rochester arm64 machines is missing some of the grub bootloader messages (and this is detected by one of the examine jobs). I may need help from someone familiar with these machines' hardware/firmware.

* stretch did not appear to work properly on joubertin[01] and merlot[01] (x86 boxes). I have marked them to be not used with stretch. They will need to be retested and the problems investigated (if indeed the problems still remain after other fixes which are now in this series).

* I have two bug reports / patches to go upstream to Debian: a boot ordering bug with ifupdown and udev, and a bug in fishdescriptor.

There are a couple of occurrences in this series of changes which break things followed by patches to fix them up. I have not bothered squashing/reorganising those.

Ian Jackson (41):
power: Fix uninitialised variable warning
Debian: Fix http:// url for bugs.xenproject.org
contents_make_cpio: Include symlinks
contents_make_cpio: Make files owned by root
overlay-persistent-net: Copy from jessie
persistent-net: Add overlay on installed systems >= stretch
persistent-net: Add overlay in installer >= stretch
ts-host-install: Break out $persistent_net_rules
ts-host-install: Unconditionally mkdir -p /etc/udev/rules.d
ts-host-install: Put canary in 70-persistent-net.rules
persistent-net: Include initramfs script to copy to target
persistent-net: Set net.ifnames=0 in di_installcmdline_core
ts-kernel-build: Enable some additional drivers for Thunder-X
TestSupport: Move stashed files' and next_unique_name' earlier
TestSupport: Move `target_somefile_getleaf' earlier
TestSupport: target_somefile_leaf rename and change a variable
TestSupport: Provide target_getfile_root_stash
ts-guests-nbd-mirror: Use target_getfile_root_stash
preseed_base: chmod ssh host private keys to placate sshd
preseed_base: break out debian_overlays_fixup_cmd
ts-debian-fixup: Use debian_overlays_fixup_cmd
preseed_hook_command: allow specifying di keys other than preseed/* preseed_hook_installscript: Use partman/early_command, not preseed/
Debian: preseed_hook_installscript: New $atonce option
Debian: partman scripts: Run right away too
Debian: set partman-lvm/device_remove_lvm_span
Debian: Add reference to bug numbers for erase-other-disks dm restrict audit: actually install right package for fishdescriptor
Debian: Move preseed_backports_packages earlier
dm restrict audit: actually install fishdescriptor in host dm restrict audit: always install (some) chiark-scripts
target_install_packages: Consistently use qw(...) rather than '...'
ts-xen-install: Install libpciaccess0
ts-debian-hvm-install: Honour linux_boot_append target var
make-flight: shadow test: Disable kpti in guests
platforms: Pass suite to get_arch_platforms
platforms: Honour suite in get_arch_platforms
dm restrict, fishdescriptor: Update to a fixed chiark-scripts
Debian: Fix /lib/udev/ifupdown-hotplug to not run if / is ro
Debian: Fix /lib/udev/ifupdown-hotplug in guests from debootstrap production-config-cambridge: Provide TftpDiVersion_stretch

Julien Grall (2):
stretch: Use chainloading when booting using GRUB on Arm64
ts-xen-build: Enable ITS driver in Xen

Wei Liu (19):
gitignore: ignore vim swap file
ts-xen-build-prep: only install w3c-dtd-xhtml for suites <Stretch
ts-xen-build-prep: install packages for suites >jessie
ts-xen-install: install some packages on stretch Debian.pm: use sysvinit-core instead of systemd
ts-leak-check: suppress systemd-shim, which leaks in stretch
ts-host-install: don't use the new nic naming scheme
ts-debian-fixup: merge origin extra= to our own if necessary
ts-debian-fixup: append noresume
ts-xen-build-prep: install e2fslibs-dev
ts-guests-nbd-mirror: make it work with stretch
Extend workaround `clk_ignore_unused' to stretch
adjust how to skip bootloader installation for arm32, in Stretch
ts-debian-di-install: use gho to pick d-i
make-flight: guest should use jessie to test pvgrub
Drop rumprun tests
ts-guests-nbd-mirror: use target_{get,put}file_root to transfter cfg
ts-kernel-build: disable host1x, which doesn't build Switch to Debian Stretch

 .gitignore                                         |   1 +
 Osstest.pm                                         |   2 +-
 Osstest/Debian.pm                                  | 205 ++++++++++++++-------
 Osstest/HostDB/Executive.pm                        |  10 +-
 Osstest/HostDB/Static.pm                           |   4 +-
 Osstest/RumpRun.pm                                 |  68 -------
 Osstest/TestSupport.pm                             | 115 ++++++------
 Osstest/Toolstack/rumprun.pm                       |  33 ----
 allow.all                                          |   1 -
 allow.rumprun                                      |   1 -
 ap-common                                          |   7 -
 ap-fetch-version                                   |   4 -
 ap-fetch-version-old                               |   5 -
 ap-print-url                                       |   3 -
 ap-push                                            |   5 -
 cr-daily-branch                                    |   9 -
 cri-common                                         |   1 -
 cri-getplatforms                                   |   2 +-
 crontab                                            |   4 +-
 daily-cron-email-real--rumpuserxen                 |   4 -
 daily-cron-email-real-bisectcomplete--rumpuserxen  |   4 -
 make-flight                                        |  48 +----
 mfi-common                                         |  37 ----
 overlay-persistent-net/etc/init.d/udev-finish      |  22 +++
 .../etc/udev/rule_generator.functions              | 113 ++++++++++++
 .../etc/udev/rules.d/73-usb-net-by-mac.rules       |   0
 .../etc/udev/rules.d/75-net-description.rules      |   0
 .../udev/rules.d/75-persistent-net-generator.rules | 143 ++++++++++++++
 overlay-persistent-net/etc/udev/udev-finish        |   9 +
 overlay-persistent-net/etc/udev/write_net_rules    | 152 +++++++++++++++
 .../lib/udev/rule_generator.functions              |   1 +
 overlay-persistent-net/lib/udev/udev-finish        |   1 +
 overlay-persistent-net/lib/udev/write_net_rules    |   1 +
 production-config                                  |   3 +
 production-config-cambridge                        |   1 +
 sg-run-job                                         |  26 ---
 ts-debian-di-install                               |   4 +-
 ts-debian-fixup                                    |  23 ++-
 ts-debian-hvm-install                              |   3 +
 ts-guests-nbd-mirror                               |  51 ++++-
 ts-host-install                                    |  23 ++-
 ts-kernel-build                                    |  14 ++
 ts-leak-check                                      |   1 +
 ts-rumprun-bake                                    |  88 ---------
 ts-rumprun-build                                   | 103 -----------
 ts-rumprun-demo-build                              |  67 -------
 ts-rumprun-demo-setup                              |  54 ------
 ts-rumprun-demo-xenstorels                         | 113 ------------
 ts-rumprun-test-prep                               |  37 ----
 ts-xen-build                                       |   4 +
 ts-xen-build-prep                                  |  16 +-
 ts-xen-install                                     |  12 +-
 52 files changed, 803 insertions(+), 855 deletions(-)
 delete mode 100644 Osstest/RumpRun.pm
 delete mode 100644 Osstest/Toolstack/rumprun.pm
 delete mode 100644 allow.rumprun
 delete mode 100644 daily-cron-email-real--rumpuserxen
 delete mode 100644 daily-cron-email-real-bisectcomplete--rumpuserxen
 create mode 100755 overlay-persistent-net/etc/init.d/udev-finish
 create mode 100644 overlay-persistent-net/etc/udev/rule_generator.functions
 create mode 100644 overlay-persistent-net/etc/udev/rules.d/73-usb-net-by-mac.rules
 create mode 100644 overlay-persistent-net/etc/udev/rules.d/75-net-description.rules
 create mode 100644 overlay-persistent-net/etc/udev/rules.d/75-persistent-net-generator.rules
 create mode 100755 overlay-persistent-net/etc/udev/udev-finish
 create mode 100755 overlay-persistent-net/etc/udev/write_net_rules
 create mode 120000 overlay-persistent-net/lib/udev/rule_generator.functions
 create mode 120000 overlay-persistent-net/lib/udev/udev-finish
 create mode 120000 overlay-persistent-net/lib/udev/write_net_rules
 delete mode 100755 ts-rumprun-bake
 delete mode 100755 ts-rumprun-build
 delete mode 100755 ts-rumprun-demo-build
 delete mode 100755 ts-rumprun-demo-setup
 delete mode 100755 ts-rumprun-demo-xenstorels
 delete mode 100755 ts-rumprun-test-prep
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH 1/62 gitignore: ignore vim swap file

From: Wei Liu <[email protected]>

---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)
.gitignore +1 -0
index 425506b6..f7e5b77f 100644
@@ -1,5 +1,6 @@
 *~
 *.bak
+*.swp
 tmp
 *.tmp
 bisection.ps
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#2 lore raw

PATCH 2/62 power: Fix uninitialised variable warning

In reply to Ian Jackson

In
power: Record approach used for power cycles in runvars
we introduced a reference to $r{$rv} which might be undef,
resulting in this:
Use of uninitialized value in concatenation (.) or string at Osstest/TestSupport.pm line 1069.

---
 Osstest/TestSupport.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/TestSupport.pm +1 -1
index 334cc2cb..d35a784b 100644
@@ -1023,7 +1023,7 @@ sub power_reboot_attempts ($$$;$$) {
 	    logm("power: rebooted $ho->{Name} (using $approach->{Name})");
 	    if (defined $record_runvar_tail) {
 		my $rv = "$ho->{Ident}_power_${record_runvar_tail}";
-		my $newval = $r{$rv}.(!!length($r{$rv}) and ',')
+		my $newval = ($r{$rv} // '').(!!length($r{$rv}) and ',')
 		    .$approach->{Name};
 		store_runvar($rv, $newval);
 	    }
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#3 lore raw

PATCH 3/62 ts-xen-build-prep: only install w3c-dtd-xhtml for suites <stretch

In reply to Ian Jackson

From: Wei Liu <[email protected]>

That package is not included in stretch.

That package was installed because the libvirt build needed it. However libvirt builds fine without it in stretch.

---
 ts-xen-build-prep | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
ts-xen-build-prep +4 -2
index 23bbbeb9..a15ab3df 100755
@@ -206,11 +206,13 @@ sub prep () {
                       libglib2.0-dev liblzma-dev pkg-config
                       autoconf automake libtool xsltproc
                       libxml2-utils libxml2-dev
-                      libdevmapper-dev w3c-dtd-xhtml libxml-xpath-perl
-                      libelf-dev
+                      libdevmapper-dev libxml-xpath-perl libelf-dev
                       ccache nasm checkpolicy ebtables
                       libgnutls28-dev);
 
+    if ($ho->{Suite} =~ m/squeeze|wheezy|jessie/) {
+	push(@packages, "w3c-dtd-xhtml");
+    }
     if ($ho->{Suite} !~ m/squeeze|wheezy/) {
 	push(@packages, qw(ocaml-nox ocaml-findlib));
     }
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#4 lore raw

PATCH 4/62 ts-xen-build-prep: install packages for suites >jessie

In reply to Ian Jackson

From: Wei Liu <[email protected]>

The stubdom build needs texinfo.
The libvirt build needs autopoint. The QEMU build needs libpciaccess-dev.

---
 ts-xen-build-prep | 4 ++++
 1 file changed, 4 insertions(+)
ts-xen-build-prep +4 -0
index a15ab3df..ca5735a1 100755
@@ -222,6 +222,10 @@ sub prep () {
 	# jessie (>jessie?)
 	push(@packages, "libnl-route-3-dev");
     }
+    if ($ho->{Suite} !~ m/squeeze|wheezy|jessie/) {
+        push(@packages, qw(texinfo autopoint libpciaccess-dev));
+    }
+
     target_install_packages($ho, @packages);
     target_cmd_root($ho, "chmod -R a+r /usr/share/git-core/templates");
     # workaround for Debian #595728
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#5 lore raw

PATCH 5/62 ts-xen-install: install some packages on stretch

In reply to Ian Jackson

From: Wei Liu <[email protected]>

The "route" command is now in that package.

libnl is needed when running xl.

---
v3: invert condition.

---
 ts-xen-install | 3 +++
 1 file changed, 3 insertions(+)
ts-xen-install +3 -0
index 8de94ac2..80952857 100755
@@ -52,6 +52,9 @@ sub packages () {
                                libsdl1.2debian libglib2.0-0 liblzma5
 			       qemu-utils
                                netcat-openbsd));
+    if ($ho->{Suite} !~ m/squeeze|wheezy|jessie/) {
+        target_install_packages($ho, 'net-tools libnl-route-3-200');
+    }
     if ($ho->{Suite} =~ m/jessie/) {
         target_install_packages($ho, 'libnl-route-3-200');
     }
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#6 lore raw

PATCH 6/62 Debian.pm: use sysvinit-core instead of systemd

In reply to Ian Jackson

From: Wei Liu <[email protected]>

Install that packages for suites >wheezy, because they use systemd as the default init.

---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
index 59c60d40..82b5fb40 100644
@@ -812,7 +812,7 @@ sub preseed_base ($$$;@) {
 
     # Systemd doesn't honor osstest-confirm-booted service, which
     # breaks ts-leak-check.  Fall back to SysV init for now.
-    if ( $suite =~ /jessie/ ) {
+    if ( $suite !~ /squeeze|wheezy/ ) {
        preseed_hook_command($ho, 'late_command', $sfx, <<END)
 in-target apt-get install -y sysvinit-core
 END
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#7 lore raw

PATCH 7/62 ts-leak-check: suppress systemd-shim, which leaks in stretch

In reply to Ian Jackson

From: Wei Liu <[email protected]>

---
 ts-leak-check | 1 +
 1 file changed, 1 insertion(+)
ts-leak-check +1 -0
index 678d0696..41e6245d 100755
@@ -202,6 +202,7 @@ xenstore /vm
 xenstore /libxl
 
 process .* udevd
+process .* /.+/systemd-shim
 
 file /var/run/xenstored/db
 file /var/run/xenstored/db.debug
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#8 lore raw

PATCH 8/62 ts-host-install: don't use the new nic naming scheme

In reply to Ian Jackson

From: Wei Liu <[email protected]>

---
 ts-host-install | 4 ++++
 1 file changed, 4 insertions(+)
ts-host-install +4 -0
index 292733ba..9ab3de62 100755
@@ -244,6 +244,10 @@ END
     # why this is repeated.
     push @hocmdline, "console=$console" unless $console eq "NONE";
 
+    # Don't use "Predictable Network Interface Names"
+    # https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
+    push @hocmdline, "net.ifnames=0" if $ho->{Suite} =~ m/stretch/;
+
     push @hocmdline,
         get_host_property($ho, "linux-boot-append $ho->{Suite}", ''),
         get_host_property($ho, "linux-boot-append $ho->{Suite} $r{arch}", '');
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#9 lore raw

PATCH 9/62 ts-debian-fixup: merge origin extra= to our own if necessary

In reply to Ian Jackson

From: Wei Liu <[email protected]>

The original extra= was not removed, so there were two extra= in the resulting config file.

It wasn't a problem for xl because the second extra= took precedence. However libvirt tests would only pick up the first extra= so they worked by chance.

Fix this issue by merging the original. If there isn't already extra= in $cfg, use our own.

---
v3: handle situation when no extra= is in $cfg

---
 ts-debian-fixup | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
ts-debian-fixup +5 -1
index 3e5cdb97..478dc2af 100755
@@ -178,7 +178,11 @@ sub otherfixupcfg () {
         $extra .= " iommu=soft";
     }
 
-    $cfg .= "\nextra='$extra'\n";
+    if ($cfg =~ m/extra\s*=\s*['"](.*)['"]/) {
+	$cfg =~ s/extra\s*=\s*['"](.*)['"]/extra = '$1 $extra'/;
+    } else {
+	$cfg .= "extra = '$extra'\n";
+    }
 };
 
 sub writecfg () {
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#10 lore raw

PATCH 10/62 ts-debian-fixup: append noresume

In reply to Ian Jackson

From: Wei Liu <[email protected]>

See code comment for explanation.

---
v4: Fix grammar error in comment.

---
 ts-debian-fixup | 10 ++++++++++
 1 file changed, 10 insertions(+)
ts-debian-fixup +10 -0
index 478dc2af..0e553d47 100755
@@ -178,6 +178,16 @@ sub otherfixupcfg () {
         $extra .= " iommu=soft";
     }
 
+    # There might be stale entries in /etc/initramfs-tools/conf.d/resume which
+    # get stored in the initramfs. That introduces delay in guest booting which
+    # might cause tests to fail.
+    #
+    # This is particularly prominent in stretch when it tries to scan for the
+    # nonexistent device(s) for a long time. See also Debian bug #784810.
+    #
+    # Append noresume to fix the issue.
+    $extra .= " noresume";
+
     if ($cfg =~ m/extra\s*=\s*['"](.*)['"]/) {
 	$cfg =~ s/extra\s*=\s*['"](.*)['"]/extra = '$1 $extra'/;
     } else {
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#11 lore raw

PATCH 11/62 ts-xen-build-prep: install e2fslibs-dev

In reply to Ian Jackson

From: Wei Liu <[email protected]>

The in-tree libfsimage ext2fs implementation can't handle 64bit enabled ext4, which is the default in stretch.

Installing e2fslibs-dev causes libfsimage to pick up the packaged ext2fs implementation.

---
 ts-xen-build-prep | 6 ++++++
 1 file changed, 6 insertions(+)
ts-xen-build-prep +6 -0
index ca5735a1..c38ab36d 100755
@@ -226,6 +226,12 @@ sub prep () {
         push(@packages, qw(texinfo autopoint libpciaccess-dev));
     }
 
+    # The in-tree ext4 support in libfsimage can't cope with 64bit ext4 on
+    # 32bit build. Use the packaged library.
+    if ($ho->{Suite} !~ m/squeeze|wheezy|jessie/) {
+        push(@packages, qw(e2fslibs-dev));
+    }
+
     target_install_packages($ho, @packages);
     target_cmd_root($ho, "chmod -R a+r /usr/share/git-core/templates");
     # workaround for Debian #595728
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#12 lore raw

PATCH 12/62 ts-guests-nbd-mirror: make it work with stretch

In reply to Ian Jackson

From: Wei Liu <[email protected]>

On the server side, only add oldstyle= and port= on wheezy and jessie. stretch doesn't support or need those anymore.

On the client side, generate new style configuration file.

Reorder nbd-client setup a bit. Install it first, then write our own configuration file, then start it. This stops dpkg asking what to do regarding configuration files.

---
v3: invert some tests, rearrange client setup code.
v4: Fix commit message grammar.

---
 ts-guests-nbd-mirror | 47 +++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 43 insertions(+), 4 deletions(-)
ts-guests-nbd-mirror +43 -4
index ca8300db..7ea31f7c 100755
@@ -60,15 +60,19 @@ sub configserver () {
 [generic]
     user = root
 END
-    $scfg .= <<END unless $sho->{Suite} =~ m/sarge|lenny|squeeze/;
+
+    $scfg .= <<END if $sho->{Suite} =~ m/wheezy|jessie/;
     oldstyle = true
 END
+
     foreach my $v (@vols) {
 	$v->{Port}= unique_incrementing_runvar("${srvhost}_nextport",4000);
 	$v->{Path}= "/dev/$v->{Gho}{Vg}/$v->{Lv}";
 	$scfg.=<<END;
 [export$v->{Ix}]
     exportname = $v->{Path}
+END
+	$scfg.=<<END if $sho->{Suite} =~ m/wheezy|jessie/;
     port = $v->{Port}
 END
     }
@@ -79,9 +83,7 @@ END
     target_install_packages($sho, qw(nbd-server));
 }
 
-sub configclient () {
-    target_cmd_root($cho, "dpkg --purge nbd-client ||:");
-
+sub configclient_pre_stretch () {
     my $mydaemon= '/root/nbd-client-async';
     target_putfilecontents_root_stash($cho,10,<<'END',$mydaemon);
 #!/bin/sh
@@ -107,7 +109,44 @@ NBD_PORT[$v->{Ix}]=$v->{Port}
 END
     }
     target_putfilecontents_root_stash($cho,10,$ccfg,"/etc/nbd-client");
+}
+
+sub configclient_stretch_and_later () {
+    my $ccfg = <<END;
+# generated by $0
+END
+
+    foreach my $v (@vols) {
+	my $nbddev = "nbd$v->{Ix}";
+	$ccfg .= <<END;
+$nbddev $sho->{Name} export$v->{Ix}
+END
+    }
+
+    target_putfilecontents_root_stash($cho,10,$ccfg,"/etc/nbdtab");
+}
+
+sub configclient () {
+    target_cmd_root($cho, "dpkg --purge nbd-client ||:");
+
     target_install_packages($cho, qw(nbd-client));
+
+    target_cmd_root($cho, "/etc/init.d/nbd-client stop ||:");
+
+    if ($cho->{Suite} =~ m/wheezy|jessie/) {
+        configclient_pre_stretch();
+    } else {
+        configclient_stretch_and_later();
+	foreach my $v (@vols) {
+	    my $nbddev = "nbd$v->{Ix}";
+	    target_cmd_root($cho, <<END);
+mkdir -p /dev/$v->{Gho}{Vg}
+if ! test -L $v->{Path}; then ln -s /dev/$nbddev $v->{Path}; fi
+END
+	}
+    }
+
+    target_cmd_root($cho, "/etc/init.d/nbd-client start");
 }
 
 sub shuffleconfigs () {
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#13 lore raw

PATCH 13/62 Extend workaround `clk_ignore_unused' to stretch

In reply to Ian Jackson

From: Wei Liu <[email protected]>

This is https://bugs.xenproject.org/xen/bug/45

Without that parameter we lose uart output.

CC Stefano Stabellini <[email protected]>
CC Julien Grall <[email protected]>
---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
index 82b5fb40..91bffdff 100644
@@ -241,7 +241,7 @@ END
 	push @xenkopt, $xenkopt;
 	# http://bugs.xenproject.org/xen/bug/45
 	push @xenkopt, "clk_ignore_unused"
-	    if $ho->{Suite} =~ m/wheezy|jessie/;
+	    if $ho->{Suite} =~ m/wheezy|jessie|stretch/;
 
 	$xenkopt = join ' ', @xenkopt;
 	logm("Dom0 Linux options: $xenkopt");
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Acked-by on v1 13/62
#14 lore raw

Re: [OSSTEST PATCH 13/62] Extend workaround `clk_ignore_unused' to stretch

Hi,

Quoted 3 lines · Ian Jackson

From: Wei Liu <[email protected]>

This is https://bugs.xenproject.org/xen/bug/45

Still no resolution for this one :/.

Quoted 6 lines

Without that parameter we lose uart output.

CC Stefano Stabellini <[email protected]>
CC Julien Grall <[email protected]>

Cheers,

Quoted 18 lines
---
  Osstest/Debian.pm | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
index 82b5fb40..91bffdff 100644
@@ -241,7 +241,7 @@ END
  	push @xenkopt, $xenkopt;
  	# http://bugs.xenproject.org/xen/bug/45
  	push @xenkopt, "clk_ignore_unused"
-	    if $ho->{Suite} =~ m/wheezy|jessie/;
+	    if $ho->{Suite} =~ m/wheezy|jessie|stretch/;
 
  	$xenkopt = join ' ', @xenkopt;
  	logm("Dom0 Linux options: $xenkopt");
Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#15 lore raw

PATCH 14/62 adjust how to skip bootloader installation for arm32, in Stretch

In reply to Ian Jackson

From: Wei Liu <[email protected]>

---
v4: Change case of suite names in comment.

---
 Osstest/Debian.pm | 4 ++++
 1 file changed, 4 insertions(+)
Osstest/Debian.pm +4 -0
index 91bffdff..85b1890d 100644
@@ -1041,7 +1041,11 @@ END
 	# actually does what we need, but the installer doesn't treat
 	# that as a "bootloader".
 	$preseed_file.= (<<END);
+        # jessie and earlier
 d-i     nobootloader/confirmation_common boolean true
+        # stretch and later
+d-i     grub-installer/skip boolean true
+d-i     lilo-installer/skip boolean true
 END
 
         # Debian Bug #771949 means that update-menu-list always
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#16 lore raw

PATCH 15/62 ts-debian-di-install: use gho to pick d-i

In reply to Ian Jackson

From: Wei Liu <[email protected]>

The original code used ho which gave us the host suite, but we wanted the guest suite.

---
 ts-debian-di-install | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
ts-debian-di-install +2 -2
index 60079712..361a1710 100755
@@ -152,8 +152,8 @@ sub setup_netboot($$$)
 	die if $r{ "$gho->{Guest}_netboot_kernel" }
 	||     $r{ "$gho->{Guest}_netboot_ramdisk" };
 
-	my $di_path = $c{TftpPath}.'/'.$ho->{Tftp}{DiBase}.'/'.${arch}.'/'.
-	    debian_guest_di_version($ho).'-'.$ho->{Suite};
+	my $di_path = $c{TftpPath}.'/'.$gho->{Tftp}{DiBase}.'/'.${arch}.'/'.
+	    debian_guest_di_version($gho).'-'.$gho->{Suite};
 
         if (${arch} =~ m/amd64|i386/) {
 	    $kernel = "$di_path/vmlinuz-xen";
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#17 lore raw

PATCH 16/62 make-flight: guest should use jessie to test pvgrub

In reply to Ian Jackson

From: Wei Liu <[email protected]>

stretch has 64bit feature enabled for ext4, which pvgrub can't cope. We want to continue to test pvgrub, so specify jessie in the guest suite field.

A consequence is that this test will test jessie forever. Eventually jessie will rot so badly that this test fails and then we will no longer be testing pvgrub1. Hopefully by then no-one will be using it.

---
v4: Adjust commit message slightly.

---
 make-flight | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
make-flight +4 -4
index 93c82a21..2f875125 100755
@@ -608,8 +608,8 @@ do_pvgrub_tests () {
   job_create_test test-$xenarch$kern-$dom0arch-amd64-pvgrub     \
     test-debian-di xl $xenarch $dom0arch                        \
       debian_arch=amd64                                         \
-      debian_suite=$guestsuite                                  \
-      debian_di_version=$guest_di_version                       \
+      debian_suite=jessie                                       \
+      debian_di_version=`getconfig_TftpDiVersion_suite jessie`  \
       debian_method=netboot                                     \
       debian_bootloader=pvgrub                                  \
       all_hostflags=$most_hostflags                             \
@@ -617,8 +617,8 @@ do_pvgrub_tests () {
   job_create_test test-$xenarch$kern-$dom0arch-i386-pvgrub      \
     test-debian-di xl $xenarch $dom0arch                        \
       debian_arch=i386                                          \
-      debian_suite=$guestsuite                                  \
-      debian_di_version=$guest_di_version                       \
+      debian_suite=jessie                                       \
+      debian_di_version=`getconfig_TftpDiVersion_suite jessie`  \
       debian_method=netboot                                     \
       debian_bootloader=pvgrub                                  \
       all_hostflags=$most_hostflags
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#18 lore raw

PATCH 17/62 stretch: Use chainloading when booting using GRUB on Arm64

In reply to Ian Jackson

From: Julien Grall <[email protected]>

The GRUB package in stretch is not able to boot Xen on Arm64. Use chainloading as we did for jessie for the time being.

Note that a bug has been filled on Debian to integrate Xen pactches for the next release (see [1]).

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=884770

---
v4: Correct suite name capitalisation in commit message and comment.

---
 Osstest/Debian.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Osstest/Debian.pm +2 -2
index 85b1890d..80b4cf37 100644
@@ -435,10 +435,10 @@ sub setupboot_grub2 ($$$$) {
     my $rmenu= '/boot/grub/grub.cfg';
     my $kernkey= (defined $xenhopt ? 'KernDom0' : 'KernOnly');
 
-    # Grub2 on Jessie/arm* doesn't do multiboot, so we must chainload.
+    # Grub2 on jessie/stretch ARM* doesn't do multiboot, so we must chainload.
     my $need_uefi_chainload =
         get_host_property($ho, "firmware", "") eq "uefi" &&
-        $ho->{Suite} =~ m/jessie/ && $r{arch} =~ m/^arm/;
+        $ho->{Suite} =~ m/jessie|stretch/ && $r{arch} =~ m/^arm/;
 
     my $parsemenu= sub {
         my $f= bl_getmenu_open($ho, $rmenu, "$stash/$ho->{Name}--grub.cfg.1");
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#19 lore raw

PATCH 18/62 Drop rumprun tests

In reply to Ian Jackson

From: Wei Liu <[email protected]>

These have been failing for some time and it doesn't any more look like this will be an attractive route to stub device models. (At least two Xen downstream projects are using Linux-based stub device models.)

---
v4: Expand commit message.

---
 Osstest/RumpRun.pm                                |  68 -------------
 Osstest/Toolstack/rumprun.pm                      |  33 -------
 allow.all                                         |   1 -
 allow.rumprun                                     |   1 -
 ap-common                                         |   7 --
 ap-fetch-version                                  |   4 -
 ap-fetch-version-old                              |   5 -
 ap-print-url                                      |   3 -
 ap-push                                           |   5 -
 cr-daily-branch                                   |   9 --
 cri-common                                        |   1 -
 crontab                                           |   4 +-
 daily-cron-email-real--rumpuserxen                |   4 -
 daily-cron-email-real-bisectcomplete--rumpuserxen |   4 -
 make-flight                                       |  35 -------
 mfi-common                                        |  37 -------
 sg-run-job                                        |  26 -----
 ts-rumprun-bake                                   |  88 -----------------
 ts-rumprun-build                                  | 103 --------------------
 ts-rumprun-demo-build                             |  67 -------------
 ts-rumprun-demo-setup                             |  54 -----------
 ts-rumprun-demo-xenstorels                        | 113 ----------------------
 ts-rumprun-test-prep                              |  37 -------
 23 files changed, 2 insertions(+), 707 deletions(-)
 delete mode 100644 Osstest/RumpRun.pm
 delete mode 100644 Osstest/Toolstack/rumprun.pm
 delete mode 100644 allow.rumprun
 delete mode 100644 daily-cron-email-real--rumpuserxen
 delete mode 100644 daily-cron-email-real-bisectcomplete--rumpuserxen
 delete mode 100755 ts-rumprun-bake
 delete mode 100755 ts-rumprun-build
 delete mode 100755 ts-rumprun-demo-build
 delete mode 100755 ts-rumprun-demo-setup
 delete mode 100755 ts-rumprun-demo-xenstorels
 delete mode 100755 ts-rumprun-test-prep
Osstest/RumpRun.pm deleted +0 -68
deleted file mode 100644
index f46d520b..00000000
@@ -1,68 +0,0 @@
-# This is part of "osstest", an automated testing framework for Xen.
-# Copyright (C) 2009-2013 Citrix Inc.
-# 
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-# 
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-# 
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-
-package Osstest::RumpRun;
-
-use strict;
-use warnings;
-
-use Osstest::TestSupport;
-
-BEGIN {
-    use Exporter ();
-    our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
-    $VERSION     = 1.00;
-    @ISA         = qw(Exporter);
-    @EXPORT      = qw(
-		      rumprun_guest_create
-		    );
-    %EXPORT_TAGS = ( );
-
-    @EXPORT_OK   = qw();
-}
-
-sub rumprun_guest_create ($) {
-    my ($gho) = @_;
-    my $ho = $gho->{Host};
-    my $gn = $gho->{Guest};
-    guest_prepare_disk($gho);
-
-    my $rumprun = target_var($gho, 'rumprun_path');
-    if (!$rumprun) {
-	logm("finding rumprun to use for $gho->{Name} on $ho->{Name}");
-	my $buildjob = $r{guests_rumprunbuildjob} // # todo: eliminate this
-	    target_var($gho, 'rumprunbuildjob');
-	my $rumprundist = target_extract_jobdistpath_subdir
-	    ($ho, "rumprun-rumprun-g-$gho->{Name}", "rumprun", $buildjob);
-	$rumprun = "$rumprundist/rumprun";
-	store_runvar("${gn}_rumprun_path", $rumprun);
-    }
-
-    my $imagepath = $r{"${gn}_imagepath"};
-    my $cmdline = guest_var($gho, 'cmdline', undef);
-
-    my $cmd = "$rumprun xen";
-    $cmd .= " -N $gn";
-    $cmd .= " -I xenif0,xenif,mac=$gho->{Ether}";
-    $cmd .= " -W xenif0,inet,dhcp";
-    $cmd .= " $imagepath";
-    $cmd .= " $cmdline";
-
-    target_cmd_root($ho, $cmd, 100);
-}
-
-1;
Osstest/Toolstack/rumprun.pm deleted +0 -33
deleted file mode 100644
index 74742c45..00000000
@@ -1,33 +0,0 @@
-# This is part of "osstest", an automated testing framework for Xen.
-# Copyright (C) 2014 Citrix Inc.
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-
-package Osstest::Toolstack::rumprun;
-
-use strict;
-use warnings;
-
-use Osstest::RumpRun;
-
-# Defer to xl driver for most things
-use parent qw(Osstest::Toolstack::xl);
-
-sub create ($$) {
-    my ($self,$gho) = @_;
-    rumprun_guest_create($gho);
-}
-
-1;
allow.all +0 -1
index 5d8d9cb7..894b69ea 100644
@@ -3,6 +3,5 @@ build-@@                        logs-capture@@
 test-@@-pcipt@@
 test-@@-qemuu-@@		guest-localmigrate
 test-@@-win7-@@			guest-stop
-test-@@-rumprun-@@		rumprun-demo-xenstorels/xenstorels.repeat
 @@-arm64			hosts-allocate
 @@-arm64-@@			hosts-allocate
allow.rumprun deleted +0 -1
deleted file mode 100644
index 436417b1..00000000
@@ -1 +0,0 @@
-!test-@@-rumprun-@@		rumprun-demo-xenstorels/xenstorels.repeat
ap-common +0 -7
index 87df7948..6ae7cce6 100644
@@ -45,12 +45,6 @@
 : ${PUSH_TREE_LIBVIRT:=$XENBITS:/home/xen/git/libvirt.git}
 : ${BASE_TREE_LIBVIRT:=git://xenbits.xen.org/libvirt.git}
 
-: ${TREE_RUMPRUN:=https://github.com/rumpkernel/rumprun}
-# ^ really http://repo.rumpkernel.org/rumprun
-: ${TREEVCS_RUMPRUN:=git}
-: ${BASE_TREE_RUMPRUN:=git://xenbits.xen.org/osstest/rumprun.git}
-: ${PUSH_TREE_RUMPRUN:=$XENBITS:/home/xen/git/osstest/rumprun.git}
-
 : ${TREE_SEABIOS_UPSTREAM:=https://git.seabios.org/seabios.git}
 : ${PUSH_TREE_SEABIOS:=$XENBITS:/home/xen/git/osstest/seabios.git}
 : ${BASE_TREE_SEABIOS:=git://xenbits.xen.org/osstest/seabios.git}
@@ -83,7 +77,6 @@ fi
 : ${LOCALREV_XEN:=daily-cron.$branch}
 : ${LOCALREV_LINUX:=daily-cron.$branch}
 : ${LOCALREV_LIBVIRT:=daily-cron.$branch}
-: ${LOCALREV_RUMPRUN:=daily-cron.$branch}
 : ${LOCALREV_SEABIOS:=daily-cron.$branch}
 : ${LOCALREV_OVMF:=daily-cron.$branch}
 : ${LOCALREV_XTF:=daily-cron.$branch}
ap-fetch-version +0 -4
index 87725bf0..9abfbcb5 100755
@@ -90,10 +90,6 @@ libvirt)
 	repo_tree_rev_fetch_git libvirt \
 		$TREE_LIBVIRT master $LOCALREV_LIBVIRT
 	;;
-rumprun)
-	repo_tree_rev_fetch_git rumprun \
-		$TREE_RUMPRUN master $LOCALREV_RUMPRUN
-	;;
 seabios)
 	repo_tree_rev_fetch_git seabios \
 		$TREE_SEABIOS_UPSTREAM master $LOCALREV_SEABIOS
ap-fetch-version-old +0 -5
index 18213e01..420db9f6 100755
@@ -30,7 +30,6 @@ check_ap_fetch_placeholders
 : ${BASE_TAG_LINUX2639:=tested/2.6.39.x}
 : ${BASE_LOCALREV_LINUX:=daily-cron.$branch.old}
 : ${BASE_LOCALREV_LIBVIRT:=daily-cron.$branch.old}
-: ${BASE_LOCALREV_RUMPRUN:=daily-cron.$branch.old}
 : ${BASE_LOCALREV_SEABIOS:=daily-cron.$branch.old}
 : ${BASE_LOCALREV_XTF:=daily-cron.$branch.old}
 : ${BASE_LOCALREV_OVMF:=daily-cron.$branch.old}
@@ -99,10 +98,6 @@ libvirt)
 	repo_tree_rev_fetch_git libvirt \
 		$BASE_TREE_LIBVIRT $BASE_TAG_LIBVIRT $BASE_LOCALREV_LIBVIRT
 	;;
-rumprun)
-	repo_tree_rev_fetch_git rumprun \
-		$BASE_TREE_RUMPRUN xen-tested-master $BASE_LOCALREV_RUMPRUN
-	;;
 seabios)
 	repo_tree_rev_fetch_git seabios \
 		$BASE_TREE_SEABIOS xen-tested-master $BASE_LOCALREV_SEABIOS
ap-print-url +0 -3
index e9934bc0..27bdbfae 100755
@@ -52,9 +52,6 @@ linuxfirmware)
 libvirt)
 	echo $TREE_LIBVIRT
 	;;
-rumprun)
-	echo $TREE_RUMPRUN
-	;;
 seabios)
 	echo $TREE_SEABIOS_UPSTREAM
 	;;
ap-push +0 -5
index ded33f63..eb212c05 100755
@@ -37,7 +37,6 @@ TREE_LINUX=$PUSH_TREE_LINUX
 TREE_QEMU_UPSTREAM=$PUSH_TREE_QEMU_UPSTREAM
 TREE_XEN=$PUSH_TREE_XEN
 TREE_LIBVIRT=$PUSH_TREE_LIBVIRT
-TREE_RUMPRUN=$PUSH_TREE_RUMPRUN
 TREE_SEABIOS=$PUSH_TREE_SEABIOS
 TREE_OVMF=$PUSH_TREE_OVMF
 TREE_XTF=$PUSH_TREE_XTF
@@ -114,10 +113,6 @@ libvirt)
 	cd $repos/libvirt
 	git push $TREE_LIBVIRT $revision:refs/heads/xen-tested-master
 	;;
-rumprun)
-	cd $repos/rumprun
-	git push $TREE_RUMPRUN $revision:refs/heads/xen-tested-master
-	;;
 seabios)
 	cd $repos/seabios
 	git push $TREE_SEABIOS $revision:refs/heads/xen-tested-master
cr-daily-branch +0 -9
index 49b8ad8e..5698b083 100755
@@ -223,10 +223,6 @@ if [ "x$REVISION_LIBVIRT" = x ]; then
 	determine_version REVISION_LIBVIRT libvirt LIBVIRT
 	export REVISION_LIBVIRT
 fi
-if [ "x$REVISION_RUMPRUN" = x ]; then
-	determine_version REVISION_RUMPRUN rumprun RUMPRUN
-	export REVISION_RUMPRUN
-fi
 if [ "x$REVISION_XTF" = x ]; then
 	determine_version REVISION_XTF xtf XTF
 	export REVISION_XTF
@@ -278,11 +274,6 @@ libvirt)
 	realtree=libvirt
 	NEW_REVISION=$REVISION_LIBVIRT
 	;;
-rumprun)
-	realtree=rumprun
-	NEW_REVISION=$REVISION_RUMPRUN
-	export REVISION_LIBVIRT=disable
-	;;
 seabios)
 	realtree=seabios
 	NEW_REVISION=$REVISION_SEABIOS
cri-common +0 -1
index 8d2d26cf..5a425824 100644
@@ -76,7 +76,6 @@ select_xenbranch () {
 	linux-*)		tree=linux;	xenbranch=xen-unstable ;;
 	linuxfirmware)	    tree=linuxfirmware;	xenbranch=xen-unstable ;;
 	libvirt)		tree=libvirt;	xenbranch=xen-unstable ;;
-	rumprun)	      tree=rumprun; xenbranch=xen-unstable ;;
 	seabios)		tree=seabios;	xenbranch=xen-unstable ;;
 	xtf)			tree=xtf;	xenbranch=xen-unstable ;;
 	ovmf)			tree=ovmf;	xenbranch=xen-unstable ;;
crontab +2 -2
index e1e798a5..8d9b31f1 100755
@@ -14,8 +14,8 @@ [email protected]
 18		9	* * 1,3,5	cd testing.git && BRANCHES='linux-next freebsd-master' 	./cr-for-branches branches -w "./cr-daily-branch --real"
 18		9	* * 3,7		cd testing.git && BRANCHES=xen-unstable-coverity ./cr-for-branches branches -w "./cr-daily-branch --real"
 34		15	23 * *		cd testing.git && BRANCHES=examine		./cr-for-branches branches -w "./cr-daily-branch --real"
-18		4	* * *		cd testing.git && BRANCHES='linux-3.0 libvirt rumprun' ./cr-for-branches branches -w "./cr-daily-branch --real"
-6-59/15   	*	* * *		cd testing.git && EXTRA_BRANCHES='xen-unstable-smoke linux-3.0 rumprun libvirt freebsd-master' ./cr-for-branches bisects -w "./cr-try-bisect --real"
+18		4	* * *		cd testing.git && BRANCHES='linux-3.0 libvirt' ./cr-for-branches branches -w "./cr-daily-branch --real"
+6-59/15   	*	* * *		cd testing.git && EXTRA_BRANCHES='xen-unstable-smoke linux-3.0 libvirt freebsd-master' ./cr-for-branches bisects -w "./cr-try-bisect --real"
 #8-59/5		*	* * *		cd bisects/adhoc.git &&	with-lock-ex -q data-tree-lock bash -c "./cr-try-bisect-adhoc; exit $?"
 22		8	* * *		cd testing.git && BRANCHES=maintjobs		./cr-for-branches . -w ./cr-all-branch-statuses ''
 3		4	* * *		savelog -c28 testing.git/tmp/cr-for-branches.log >/dev/null
daily-cron-email-real--rumpuserxen deleted +0 -4
deleted file mode 100644
index 8f950780..00000000
@@ -1,4 +0,0 @@
-To: [email protected],
-    [email protected],
-    [email protected]
-Bcc: [email protected]
daily-cron-email-real-bisectcomplete--rumpuserxen deleted +0 -4
deleted file mode 100644
index 8f950780..00000000
@@ -1,4 +0,0 @@
-To: [email protected],
-    [email protected],
-    [email protected]
-Bcc: [email protected]
make-flight +0 -35
index 2f875125..151b1435 100755
@@ -85,14 +85,6 @@ job_create_build_filter_callback () {
         *)			return 1 ;;
       esac
     ;;
-    rumprun)
-      case "$job" in
-        build-*-pvops)		;;
-        build-*-rumprun)	;;
-        build-*-*)		return 1 ;;
-        *)			;;
-      esac
-    ;;
   esac
   return 0
 }
@@ -139,15 +131,6 @@ job_create_test_filter_callback () {
     libvirt)
       if [ x$toolstack != xlibvirt ] ; then return 1; fi
       ;;
-    rumprun)
-      case "$job" in
-      *-rumprun-*)
-          ;;
-      *)
-          : "suppressed $job"
-          return 1;;
-      esac
-      ;;
     xtf)
       case "$xenarch:$dom0arch" in
           amd64:amd64) ;;
@@ -211,7 +194,6 @@ arch_branch_filter_callback () {
         qemu-upstream-4.2-testing) return 1;;
         qemu-upstream-4.3-testing) return 1;;
         qemu-upstream-4.4-testing) return 1;;
-        rumprun) return 1;;
         seabios) return 1;;
         ovmf) return 1;;
         esac
@@ -235,21 +217,6 @@ test_matrix_branch_filter_callback () {
   return 0
 }
 
-do_rumpkernel_tests () {
-  branch_wants_rumpkernel_tests || return 0
-
-  rumparch=$dom0arch
-  job_create_test test-$xenarch$kern-$dom0arch-rumprun-$rumparch \
-                      test-rumprun xl \
-            $xenarch $dom0arch                                       \
-            guests_rumprunbuildjob=${bfi}build-$rumparch-rumprun   \
-            nettest_builtimage=rumpimages:nettest \
-            nettest_cmdline=4096 \
-            xenstorels_builtimage=rumpimages:xenstorels              \
-            xenstorels_cmdline='ls -fp device'                       \
-            all_hostflags=$most_hostflags
-}
-
 do_freebsd_tests () {
 
   if [ $xenarch != amd64 -o $dom0arch != i386 -o "$kern" != "" ]; then
@@ -769,8 +736,6 @@ test_matrix_do_one () {
   *)                test_shim=y ;;
   esac
 
-  do_rumpkernel_tests
-
   # xend PV guest test on x86 only
   if [ x$test_xend = xy -a \( $dom0arch = "i386" -o $dom0arch = "amd64" \) ]; then
     job_create_test test-$xenarch$kern-$dom0arch-pv test-debian xend \
mfi-common +0 -37
index 83d3c713..f91156fe 100644
@@ -31,16 +31,6 @@ stripy () {
   eval "$out_vn=\"\$out_$out_val\""
 }
 
-branch_wants_rumpkernel_tests () {
-  case "$branch" in
-  osstest*) return 0;;
-  rump*)    return 0;;
-  xen-*)    return 0;;
-  linux*)   return 0;;
-  *)        return 1;;
-  esac
-}
-
 branch_wants_migrupgrade_tests () {
   case "$branch" in
     xen-3.*-testing) return 1 ;;
@@ -445,33 +435,6 @@ create_build_jobs () {
         fi
     fi
 
-    if branch_wants_rumpkernel_tests; then
-
-    case $arch in
-    i386|amd64)
-    job_create_build build-$arch-rumprun build-rumprun               \
-                arch=$arch                                                   \
-        tree_xen=$TREE_XEN                                                   \
-                $RUNVARS $BUILD_RUNVARS $BUILD_RUMPRUN_RUNVARS $arch_runvars \
-                $hostos_runvars                                              \
-                host_hostflags=$build_hostflags                              \
-                buildjob=${bfi}build-$arch                                   \
-                tree_rumprun=$TREE_RUMPRUN			\
-          ${TREEVCS_RUMPRUN:+treevcs_rumprun=}${TREEVCS_RUMPRUN} \
-                revision_rumprun=$REVISION_RUMPRUN		\
-                tree_xen=$TREE_XEN                                           \
-                tree_qemuu=$TREE_QEMU_UPSTREAM                               \
-                revision_xen=$REVISION_XEN                                   \
-                revision_qemuu=$REVISION_QEMU_UPSTREAM                       \
-                tree_rumprun_rumpsrc=$TREE_RUMPRUN_RUMPSRC	\
-                ${TREEVCS_RUMPRUN_RUMPSRC:+treevcs_rumprun_rumpsrc=}${TREEVCS_RUMPRUN_RUMPSRC}
-                # rumpsrc-related runvars needed only for old rumpuser-xen
-                # (ie ones which need $bodges=1 in ts-rumprun-build)
-		;;
-    esac
-
-    fi
-
     case "$arch" in
     arm*) continue;; # don't do any other kernel builds
     esac
sg-run-job +0 -26
index 7d27f415..56b6384a 100755
@@ -649,22 +649,6 @@ proc test-guest-nomigr {g} {
     run-ts . =   ts-guest-destroy + host $g
 }
 
-proc need-hosts/test-rumprun {} { return host }
-proc run-job/test-rumprun {} {
-    run-ts . =   ts-rumprun-test-prep       + host
-    set g nettest
-    run-ts . =   ts-rumprun-demo-setup      + host $g
-    run-ts . =   ts-guest-start             + host $g
-    run-ts . =   ts-guest-destroy           + host $g
-    set g xenstorels
-    run-ts . =   ts-rumprun-demo-setup      + host + $g
-    run-ts . =   ts-rumprun-demo-xenstorels + host + $g
-    run-ts . =   ts-guest-destroy-hard      + host + $g
-    repeat-ts 150 =.repeat \
-                 ts-rumprun-demo-xenstorels + host + $g   + \; \
-                 ts-guest-destroy-hard        host   $g   +
-}
-
 if {[file exists sg-run-job-adhoc]} {
     source sg-run-job-adhoc
 }
@@ -712,7 +696,6 @@ proc need-hosts/host-examine-linux {} { examine-host-examine debian }
 proc need-hosts/build {}                { return BUILD_LINUX }
 proc need-hosts/build-kern {}           { return BUILD_LINUX }
 proc need-hosts/build-libvirt {}        { return BUILD_LINUX }
-proc need-hosts/build-rumprun {}        { return BUILD_LINUX }
 proc need-hosts/build-xtf {}            { return BUILD_LINUX }
 proc need-hosts/build-freebsd {}        { return BUILD_FREEBSD }
 proc need-hosts/build-xen-freebsd {}    { return BUILD_FREEBSD }
@@ -729,15 +712,6 @@ proc run-job/build-libvirt {} {
     run-ts . = ts-libvirt-build
 }
 
-proc run-job/build-rumprun {} {
-    run-ts . = ts-rumprun-build
-    run-ts . = ts-rumprun-demo-build + host + nettest rump-test-net
-    run-ts . xen-build ts-xen-build-rump + host --no-kconfig --
-    run-ts . = ts-rumprun-bake + host \
-        nettest :nettest:/rump-test-net \
-        xenstorels ::/usr/local/bin/xenstore-ls
-}
-
 proc run-job/build-xtf {} {
     run-ts . = ts-xtf-build
 }
ts-rumprun-bake deleted +0 -88

88 lines not on this page · raw message

ts-rumprun-build deleted +0 -103

103 lines not on this page · raw message

ts-rumprun-demo-build deleted +0 -67

67 lines not on this page · raw message

ts-rumprun-demo-setup deleted +0 -54

54 lines not on this page · raw message

ts-rumprun-demo-xenstorels deleted +0 -113

113 lines not on this page · raw message

ts-rumprun-test-prep deleted +0 -37
deleted file mode 100755
index 67371710..00000000
@@ -1,37 +0,0 @@
-#!/usr/bin/perl -w
-# This is part of "osstest", an automated testing framework for Xen.
-# Copyright (C) 2009-2013 Citrix Inc.
-# 
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-# 
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-# 
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-use strict qw(vars);
-use DBI;
-BEGIN { unshift @INC, qw(.); }
-use Osstest;
-use POSIX;
-use Osstest::TestSupport;
-use Osstest::Debian;
-
-tsreadconfig();
-
-our ($whhost) = @ARGV;
-$whhost ||= 'host';
-our $ho= selecthost($whhost);
-
-sub packages () {
-    target_install_packages($ho,
-			    qw(binutils));
-}
-
-packages();
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#20 lore raw

PATCH 19/62 Debian: Fix http:// url for bugs.xenproject.org

In reply to Ian Jackson
---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
index 80b4cf37..414cd897 100644
@@ -239,7 +239,7 @@ END
 	# Dom0 specific kernel options
 	my @xenkopt = @kopt;
 	push @xenkopt, $xenkopt;
-	# http://bugs.xenproject.org/xen/bug/45
+	# https://bugs.xenproject.org/xen/bug/45
 	push @xenkopt, "clk_ignore_unused"
 	    if $ho->{Suite} =~ m/wheezy|jessie|stretch/;
 
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#21 lore raw

PATCH 20/62 ts-guests-nbd-mirror: use target_{get, put}file_root to transfter cfg

In reply to Ian Jackson

From: Wei Liu <[email protected]>

The original code used target_cmd_output_root which caused a trailing new line to be deleted, which caused libvirt converter to fail.

It wasn't discovered until now because we appended too many "\n".

Use target_{get,put}file_root to do the job.

---
 ts-guests-nbd-mirror | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
ts-guests-nbd-mirror +5 -2
index 7ea31f7c..06903aaa 100755
@@ -154,8 +154,11 @@ sub shuffleconfigs () {
 	my $gn= $gns[$i];
 	my $gho= $ghos[$i];
 	my $cfgpath= $r{ "$gho->{Guest}_cfgpath" };
-	my $cfgdata= target_cmd_output_root($sho,"cat $cfgpath");
-	target_putfilecontents_root_stash($cho,10,$cfgdata,$cfgpath);
+	my $file= $cfgpath;
+	$file=~ s,/,-,g;
+	$file= "$stash/".hostnamepath($cho)."--$file";
+	target_getfile_root($sho, 60, $cfgpath, $file);
+	target_putfile_root($cho, 60, $file, $cfgpath);
     }
 }
 
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#22 lore raw

PATCH 21/62 ts-kernel-build: disable host1x, which doesn't build

In reply to Ian Jackson

From: Wei Liu <[email protected]>

Empirically, on stretch armhf:

drivers/gpu/host1x/cdma.c: In function `host1x_pushbuffer_init':
drivers/gpu/host1x/cdma.c:94:48: error: passing argument 3 of `dma_alloc_wc' from incompatible pointer type [-Werror=incompatible-pointer-types]

     pb->mapped = dma_alloc_wc(host1x->dev, size, &pb->phys,
						  ^

etc.

This is blocking the upgrade of the Xen Project CI to Debian stretch so disable it for now.

CC Julien Grall <[email protected]>
CC Stefano Stabellini <[email protected]>
CC Thierry Reding <[email protected]>
---
 ts-kernel-build | 3 +++
 1 file changed, 3 insertions(+)
ts-kernel-build +3 -0
index 21b8f78a..0bc443de 100755
@@ -594,6 +594,9 @@ case ${XEN_TARGET_ARCH} in
     *) ;;
 esac
 
+# Disable components that don't build
+setopt CONFIG_TEGRA_HOST1X n
+
 exit 0
 END
 }
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#23 lore raw

Re: [OSSTEST PATCH 21/62] ts-kernel-build: disable host1x, which doesn't build

Quoted 9 lines · Ian Jackson

From: Wei Liu <[email protected]>

Empirically, on stretch armhf:

drivers/gpu/host1x/cdma.c: In function `host1x_pushbuffer_init':
drivers/gpu/host1x/cdma.c:94:48: error: passing argument 3 of `dma_alloc_wc' from incompatible pointer type [-Werror=incompatible-pointer-types]

     pb->mapped = dma_alloc_wc(host1x->dev, size, &pb->phys,
						  ^

etc.

This was fixed in v4.18 by this commit:

commit 2f8a6da866eff746a9f8c7745790f3765baeb589
Author: Emil Goode <[email protected]>
Date: Wed May 16 12:22:04 2018 +0200

gpu: host1x: Fix compiler errors by converting to dma_addr_t

The compiler is complaining with the following errors:

drivers/gpu/host1x/cdma.c:94:48: error:
passing argument 3 of ‘dma_alloc_wc’ from incompatible pointer type [-Werror=incompatible-pointer-types]

drivers/gpu/host1x/cdma.c:113:48: error:
passing argument 3 of ‘dma_alloc_wc’ from incompatible pointer type [-Werror=incompatible-pointer-types]

The expected pointer type of the third argument to dma_alloc_wc() is dma_addr_t but phys_addr_t is passed.

Change the phys member of struct push_buffer to be dma_addr_t so that we pass the correct type to dma_alloc_wc().
Also check pb->mapped for non-NULL in the destroy function as that is the right way of checking if dma_alloc_wc() was successful.

It should be fairly easy to backport this to older releases, though I'm not sure exactly what made this trigger. This wasn't causing any build errors for a very long time, since this type mismatch has existed ever since the driver was merged all the way back in v3.10.

Thierry

Quoted 29 lines

This is blocking the upgrade of the Xen Project CI to Debian stretch so disable it for now.

CC Julien Grall <[email protected]>
CC Stefano Stabellini <[email protected]>
CC Thierry Reding <[email protected]>
---
 ts-kernel-build | 3 +++
 1 file changed, 3 insertions(+)
ts-kernel-build +3 -0
index 21b8f78a..0bc443de 100755
@@ -594,6 +594,9 @@ case ${XEN_TARGET_ARCH} in
     *) ;;
 esac
 
+# Disable components that don't build
+setopt CONFIG_TEGRA_HOST1X n
+
 exit 0
 END
 }
Signature · 1 line
2.11.0

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#24 lore raw

Re: [OSSTEST PATCH 21/62] ts-kernel-build: disable host1x, which doesn't build

Hi,

Quoted 37 lines · Thierry Reding
Quoted 9 lines · Ian Jackson

From: Wei Liu <[email protected]>

Empirically, on stretch armhf:

drivers/gpu/host1x/cdma.c: In function `host1x_pushbuffer_init':

   drivers/gpu/host1x/cdma.c:94:48: error: passing argument 3 of `dma_alloc_wc' from incompatible pointer type [-Werror=incompatible-pointer-types]
      pb->mapped = dma_alloc_wc(host1x->dev, size, &pb->phys,
						  ^

etc.

This was fixed in v4.18 by this commit:

commit 2f8a6da866eff746a9f8c7745790f3765baeb589
Author: Emil Goode <[email protected]>
Date: Wed May 16 12:22:04 2018 +0200

gpu: host1x: Fix compiler errors by converting to dma_addr_t

The compiler is complaining with the following errors:

drivers/gpu/host1x/cdma.c:94:48: error:
passing argument 3 of ‘dma_alloc_wc’ from incompatible pointer type [-Werror=incompatible-pointer-types]

drivers/gpu/host1x/cdma.c:113:48: error:
passing argument 3 of ‘dma_alloc_wc’ from incompatible pointer type [-Werror=incompatible-pointer-types]

The expected pointer type of the third argument to dma_alloc_wc() is dma_addr_t but phys_addr_t is passed.

Change the phys member of struct push_buffer to be dma_addr_t so that we pass the correct type to dma_alloc_wc().
Also check pb->mapped for non-NULL in the destroy function as that is the right way of checking if dma_alloc_wc() was successful.

Quoted 7 lines

It should be fairly easy to backport this to older releases, though I'm not sure exactly what made this trigger. This wasn't causing any build errors for a very long time, since this type mismatch has existed ever since the driver was merged all the way back in v3.10.

Likely no-one ever tried to build this module with CONFIG_XEN=y. Xen requires dma_addr_t to be 64-bit regardless the page-tables format selected (e.g short vs LPAE).

AFAIK, this is the only case on Arm32 where phys_addr_t and dma_addr_t would mismatched.

Ian are we using a different config between Jessie and Stretch?

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#25 lore raw

Re: [OSSTEST PATCH 21/62] ts-kernel-build: disable host1x, which doesn't build

Julien Grall writes ("Re: [OSSTEST PATCH 21/62] ts-kernel-build: disable host1x, which doesn't build"):

Quoted 1 line

Ian are we using a different config between Jessie and Stretch?

Not very [1], but we *are* using a different compiler (since we just use the one from the distro).

[1] There are some changes to add some ThunderX drivers, in a patch that you suggested :-):

+setopt CONFIG_PCI_HOST_THUNDER_PEM=y
+setopt CONFIG_PCI_HOST_THUNDER_ECAM=y
+setopt CONFIG_THUNDER_NIC_PF=m
+setopt CONFIG_THUNDER_NIC_VF=m
+setopt CONFIG_THUNDER_NIC_BGX=m
+setopt CONFIG_THUNDER_NIC_RGX=m
+setopt CONFIG_MDIO_THUNDER=m
+setopt CONFIG_I2C_THUNDERX=m
+setopt CONFIG_SPI_THUNDERX=m

I doubt these are related.

It's nice to know that it's fixed in 4.18. I don't know how disruptive it would be to backport it. I could perhaps make the config option to disable host1x conditional on the Linux kernel version but it seems easier to just leave things as they are and take the workaround out at some point in the distant future when all kernels we care about have the fix.

Thanks all,
Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#26 lore raw

Re: [OSSTEST PATCH 21/62] ts-kernel-build: disable host1x, which doesn't build

Hi Ian,

Quoted 19 lines · Ian Jackson

Julien Grall writes ("Re: [OSSTEST PATCH 21/62] ts-kernel-build: disable host1x, which doesn't build"):

Quoted 1 line

Ian are we using a different config between Jessie and Stretch?

Not very [1], but we *are* using a different compiler (since we just use the one from the distro).

[1] There are some changes to add some ThunderX drivers, in a patch that you suggested :-):

(quoted diff) +9 -0
+setopt CONFIG_PCI_HOST_THUNDER_PEM=y
+setopt CONFIG_PCI_HOST_THUNDER_ECAM=y
+setopt CONFIG_THUNDER_NIC_PF=m
+setopt CONFIG_THUNDER_NIC_VF=m
+setopt CONFIG_THUNDER_NIC_BGX=m
+setopt CONFIG_THUNDER_NIC_RGX=m
+setopt CONFIG_MDIO_THUNDER=m
+setopt CONFIG_I2C_THUNDERX=m
+setopt CONFIG_SPI_THUNDERX=m
Quoted 1 line

I doubt these are related.

Right, this should not change the selection of the host1x driver. So this is mostly like due to the compiler change.

Quoted 7 lines

It's nice to know that it's fixed in 4.18. I don't know how disruptive it would be to backport it. I could perhaps make the config option to disable host1x conditional on the Linux kernel version but it seems easier to just leave things as they are and take the workaround out at some point in the distant future when all kernels we care about have the fix.

I agree that this is the most sensible approach for now.

Looking at [1], I guess we would need to request backport up to 3.18?

Cheers,

[1] http://logs.test-lab.xenproject.org/osstest/results/all-branch-statuses.txt

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#27 lore raw

Re: [OSSTEST PATCH 21/62] ts-kernel-build: disable host1x, which doesn't build

In reply to Ian Jackson
Quoted 9 lines · Ian Jackson

From: Wei Liu <[email protected]>

Empirically, on stretch armhf:

drivers/gpu/host1x/cdma.c: In function `host1x_pushbuffer_init':
drivers/gpu/host1x/cdma.c:94:48: error: passing argument 3 of `dma_alloc_wc' from incompatible pointer type [-Werror=incompatible-pointer-types]

     pb->mapped = dma_alloc_wc(host1x->dev, size, &pb->phys,
						  ^

etc.

This was fixed in v4.18 by this commit:

commit 2f8a6da866eff746a9f8c7745790f3765baeb589
Author: Emil Goode <[email protected]>
Date: Wed May 16 12:22:04 2018 +0200

gpu: host1x: Fix compiler errors by converting to dma_addr_t

The compiler is complaining with the following errors:

drivers/gpu/host1x/cdma.c:94:48: error:
passing argument 3 of ‘dma_alloc_wc’ from incompatible pointer type [-Werror=incompatible-pointer-types]

drivers/gpu/host1x/cdma.c:113:48: error:
passing argument 3 of ‘dma_alloc_wc’ from incompatible pointer type [-Werror=incompatible-pointer-types]

The expected pointer type of the third argument to dma_alloc_wc() is dma_addr_t but phys_addr_t is passed.

Change the phys member of struct push_buffer to be dma_addr_t so that we pass the correct type to dma_alloc_wc().
Also check pb->mapped for non-NULL in the destroy function as that is the right way of checking if dma_alloc_wc() was successful.

It should be fairly easy to backport this to older releases, though I'm not sure exactly what made this trigger. This wasn't causing any build errors for a very long time, since this type mismatch has existed ever since the driver was merged all the way back in v3.10.

Thierry

Quoted 29 lines

This is blocking the upgrade of the Xen Project CI to Debian stretch so disable it for now.

CC Julien Grall <[email protected]>
CC Stefano Stabellini <[email protected]>
CC Thierry Reding <[email protected]>
---
 ts-kernel-build | 3 +++
 1 file changed, 3 insertions(+)
ts-kernel-build +3 -0
index 21b8f78a..0bc443de 100755
@@ -594,6 +594,9 @@ case ${XEN_TARGET_ARCH} in
     *) ;;
 esac
 
+# Disable components that don't build
+setopt CONFIG_TEGRA_HOST1X n
+
 exit 0
 END
 }
Signature · 1 line
2.11.0

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Acked-by on v1 21/62
#28 lore raw

Re: [OSSTEST PATCH 21/62] ts-kernel-build: disable host1x, which doesn't build

In reply to Ian Jackson

Hi,

Quoted 10 lines · Ian Jackson

From: Wei Liu <[email protected]>

Empirically, on stretch armhf:

drivers/gpu/host1x/cdma.c: In function `host1x_pushbuffer_init':

   drivers/gpu/host1x/cdma.c:94:48: error: passing argument 3 of `dma_alloc_wc' from incompatible pointer type [-Werror=incompatible-pointer-types]
      pb->mapped = dma_alloc_wc(host1x->dev, size, &pb->phys,
						  ^

etc.

Quoted 8 lines

This is blocking the upgrade of the Xen Project CI to Debian stretch so disable it for now.

CC Julien Grall <[email protected]>
CC Stefano Stabellini <[email protected]>
CC Thierry Reding <[email protected]>

Cheers,

Quoted 19 lines
---
  ts-kernel-build | 3 +++
  1 file changed, 3 insertions(+)
ts-kernel-build +3 -0
index 21b8f78a..0bc443de 100755
@@ -594,6 +594,9 @@ case ${XEN_TARGET_ARCH} in
      *) ;;
  esac
 
+# Disable components that don't build
+setopt CONFIG_TEGRA_HOST1X n
+
  exit 0
  END
  }
Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#29 lore raw

PATCH 22/62 contents_make_cpio: Include symlinks

In reply to Ian Jackson

We are going to introduce some symlinks into one of our preprepared overlays. We must therefore arrange to copy them as appropriate.

The syntax `-type f,l' is an extension in GNU find. If this causes trouble in the future we will then have to introduce the obvious circumlocution involving ( ).

---
 Osstest/TestSupport.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/TestSupport.pm +1 -1
index d35a784b..1f01ac6a 100644
@@ -1518,7 +1518,7 @@ sub contents_make_cpio ($$$) {
     if (!$child) {
         postfork();
         chdir($srcdir) or die $!;
-        open STDIN, 'find ! -name "*~" ! -name "#*" -type f -print0 |'
+        open STDIN, 'find ! -name "*~" ! -name "#*" -type f,l -print0 |'
             or die $!;
         open STDOUT, '>&', $fh or die $!;
         system "cpio -H$format -o --quiet -0 -R 1000:1000";
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#30 lore raw

PATCH 23/62 contents_make_cpio: Make files owned by root

In reply to Ian Jackson

This function is called to generate overlays for use, mainly, by the initramfs.

We are going to use it to ship udev rules. Annoyingly, udev hates files which aren't owned by root - it simply ignores them.

---
 Osstest/TestSupport.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/TestSupport.pm +1 -1
index 1f01ac6a..41f6f5f8 100644
@@ -1521,7 +1521,7 @@ sub contents_make_cpio ($$$) {
         open STDIN, 'find ! -name "*~" ! -name "#*" -type f,l -print0 |'
             or die $!;
         open STDOUT, '>&', $fh or die $!;
-        system "cpio -H$format -o --quiet -0 -R 1000:1000";
+        system "cpio -H$format -o --quiet -0 -R 0:0";
         $? and die $?;
         $!=0; close STDIN; die "$! $?" if $! or $?;
         exit 0;
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#31 lore raw

PATCH 24/62 overlay-persistent-net: Copy from jessie

In reply to Ian Jackson

These were copied from a system running Debian jessie.

The nontrivial files are:
# Copyright (C) 2006 Marco d'Itri <[email protected]>
# Copyright (C) 2007 Kay Sievers <[email protected]>
and licenced GPLv2+. That is compatible with osstest's AGPLv3+.

Right now we do nothing with these.

---
 overlay-persistent-net/etc/init.d/udev-finish      |  22 +++
 .../etc/udev/rule_generator.functions              | 113 +++++++++++++++
 .../etc/udev/rules.d/73-usb-net-by-mac.rules       |   0
 .../etc/udev/rules.d/75-net-description.rules      |   0
 .../udev/rules.d/75-persistent-net-generator.rules | 143 +++++++++++++++++++
 overlay-persistent-net/etc/udev/udev-finish        |   9 ++
 overlay-persistent-net/etc/udev/write_net_rules    | 152 +++++++++++++++++++++
 .../lib/udev/rule_generator.functions              |   1 +
 overlay-persistent-net/lib/udev/udev-finish        |   1 +
 overlay-persistent-net/lib/udev/write_net_rules    |   1 +
 10 files changed, 442 insertions(+)
 create mode 100755 overlay-persistent-net/etc/init.d/udev-finish
 create mode 100644 overlay-persistent-net/etc/udev/rule_generator.functions
 create mode 100644 overlay-persistent-net/etc/udev/rules.d/73-usb-net-by-mac.rules
 create mode 100644 overlay-persistent-net/etc/udev/rules.d/75-net-description.rules
 create mode 100644 overlay-persistent-net/etc/udev/rules.d/75-persistent-net-generator.rules
 create mode 100755 overlay-persistent-net/etc/udev/udev-finish
 create mode 100755 overlay-persistent-net/etc/udev/write_net_rules
 create mode 120000 overlay-persistent-net/lib/udev/rule_generator.functions
 create mode 120000 overlay-persistent-net/lib/udev/udev-finish
 create mode 120000 overlay-persistent-net/lib/udev/write_net_rules
overlay-persistent-net/etc/init.d/udev-finish new file +22 -0
new file mode 100755
index 00000000..10602017
@@ -0,0 +1,22 @@
+#!/bin/sh -e
+### BEGIN INIT INFO
+# Provides:          udev-finish
+# Required-Start:    udev $local_fs
+# Required-Stop:     
+# Default-Start:     S
+# Default-Stop:
+# Short-Description: Copy rules generated while the root was ro
+### END INIT INFO
+
+PATH="/sbin:/bin"
+
+. /lib/lsb/init-functions
+
+case "$1" in
+  start) ;;
+  stop|restart|force-reload) exit 0 ;;
+  *) echo "Usage: $0 {start|stop|restart|force-reload}" >&2; exit 1 ;;
+esac
+
+exec /lib/udev/udev-finish
+
overlay-persistent-net/etc/udev/rule_generator.functions new file +113 -0

113 lines not on this page · raw message

overlay-persistent-net/etc/udev/rules.d/73-usb-net-by-mac.rules new file +0 -0
new file mode 100644
index 00000000..e69de29b
overlay-persistent-net/etc/udev/rules.d/75-net-description.rules new file +0 -0
new file mode 100644
index 00000000..e69de29b
overlay-persistent-net/etc/udev/rules.d/75-persistent-net-generator.rules new file +143 -0

143 lines not on this page · raw message

overlay-persistent-net/etc/udev/udev-finish new file +9 -0
new file mode 100755
index 00000000..85e3c720
@@ -0,0 +1,9 @@
+#!/bin/sh -e
+
+# Copy any rules generated while the root filesystem was read-only
+for file in /run/udev/tmp-rules--*; do
+  [ -e "$file" ] || continue
+  cat "$file" >> "/etc/udev/rules.d/${file##*tmp-rules--}"
+  rm -f "$file"
+done
+
overlay-persistent-net/etc/udev/write_net_rules new file +152 -0

152 lines not on this page · raw message

overlay-persistent-net/lib/udev/rule_generator.functions new file +1 -0
new file mode 120000
index 00000000..3b225ebf
@@ -0,0 +1 @@
+/etc/udev/rule_generator.functions
\ No newline at end of file
overlay-persistent-net/lib/udev/udev-finish new file +1 -0
new file mode 120000
index 00000000..2c2e3042
@@ -0,0 +1 @@
+/etc/udev/udev-finish
\ No newline at end of file
overlay-persistent-net/lib/udev/write_net_rules new file +1 -0
new file mode 120000
index 00000000..bfe37cc3
@@ -0,0 +1 @@
+/etc/udev/write_net_rules
\ No newline at end of file
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#32 lore raw

PATCH 25/62 persistent-net: Add overlay on installed systems >= stretch

In reply to Ian Jackson

This prepares the ground for turning off net.ifnames. The actual rules are gated on net.ifnames so right now there is no change.

---
 Osstest/Debian.pm | 7 +++++++
 1 file changed, 7 insertions(+)
Osstest/Debian.pm +7 -0
index 414cd897..e246c012 100644
@@ -798,6 +798,13 @@ sub debian_overlays ($$) {
     };
     my $suite = $ho->{Suite};
     $maybe->('overlay');
+
+    # In stretch and later, net.ifnames=1, the fdo-style `predictable
+    # network device names', is the default; but it is anything but
+    # predictable, so we disable it.  Instead, we restore the
+    # 75-persistent-net-generator mechanism from jessie and earlier.
+    $maybe->("overlay-persistent-net") if $ho->{Suite} !~ m/wheezy|jessie/;
+
     $maybe->("overlay-$suite");
     $maybe->($c{OverlayLocal}, 'overlay-local.tar');
     $maybe->("$c{OverlayLocal}-$suite", 'overlay-local-$suite.tar');
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#33 lore raw

PATCH 26/62 persistent-net: Add overlay in installer >= stretch

In reply to Ian Jackson

We are going to need this in the installer so that the interface names from the installer environment are captured so that they can be the same on the host.

This prepares the ground for turning off net.ifnames. The actual rules are gated on net.ifnames so right now there is no change.

---
 ts-host-install | 7 +++++++
 1 file changed, 7 insertions(+)
ts-host-install +7 -0
index 9ab3de62..8ea81e24 100755
@@ -188,6 +188,13 @@ sub setup_netboot_firstboot($) {
     system qw(rm -rf --),"$initrd_overlay.d";
     mkdir "$initrd_overlay.d" or die "$initrd_overlay.d: $!";
 
+    if ($ho->{Suite} !~ m/wheezy|jessie/) {
+	my @cmd = (qw(cp -dR overlay-persistent-net/.),
+		   "$initrd_overlay.d/.");
+	logm("using persistent-net-generator: @cmd");
+	system_checked @cmd;
+    }
+
     my $ipappend = 2;
     my $wantphysif= get_host_property($ho,'interface force','auto');
     logm("Forcing interface $wantphysif");
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#34 lore raw

PATCH 27/62 ts-host-install: Break out $persistent_net_rules

In reply to Ian Jackson

We're going to want to reuse this value.

---
 ts-host-install | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
ts-host-install +4 -1
index 8ea81e24..f80a151c 100755
@@ -195,6 +195,9 @@ sub setup_netboot_firstboot($) {
 	system_checked @cmd;
     }
 
+    my $persistent_net_rules =
+	"$initrd_overlay.d/etc/udev/rules.d/70-persistent-net.rules";
+
     my $ipappend = 2;
     my $wantphysif= get_host_property($ho,'interface force','auto');
     logm("Forcing interface $wantphysif");
@@ -206,7 +209,7 @@ sub setup_netboot_firstboot($) {
 	# ip(8) moved to /sbin in Jessie
 	my $ipcmd = $ho->{Suite} =~ m/wheezy/ ? "/bin/ip" : "/sbin/ip";
         file_simple_write_contents
-            ("$initrd_overlay.d/etc/udev/rules.d/70-persistent-net.rules",
+            ($persistent_net_rules,
              $ho->{Flags}{'force-mac-address'} ? <<END : <<END);
 SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", KERNEL=="$wantphysif", RUN += "$ipcmd link set $wantphysif address $ho->{Ether}"
 END
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#35 lore raw

PATCH 28/62 ts-host-install: Unconditionally mkdir -p /etc/udev/rules.d

In reply to Ian Jackson

We are going to want this directory to exist so that we can put a canary in 70-persistent-net.rules.

In the cases where the behaviour of osstest changes, the empty directory does not result in any overall change.

---
 ts-host-install | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
ts-host-install +2 -1
index f80a151c..7423eb9b 100755
@@ -198,6 +198,8 @@ sub setup_netboot_firstboot($) {
     my $persistent_net_rules =
 	"$initrd_overlay.d/etc/udev/rules.d/70-persistent-net.rules";
 
+    system_checked(qw(mkdir -p --), "$initrd_overlay.d/etc/udev/rules.d");
+
     my $ipappend = 2;
     my $wantphysif= get_host_property($ho,'interface force','auto');
     logm("Forcing interface $wantphysif");
@@ -205,7 +207,6 @@ sub setup_netboot_firstboot($) {
 	$ipappend = 0;
 	die "need Ether for $ho->{Name} ($wantphysif)"
 	    unless defined $ho->{Ether};
-        system_checked(qw(mkdir -p --), "$initrd_overlay.d/etc/udev/rules.d");
 	# ip(8) moved to /sbin in Jessie
 	my $ipcmd = $ho->{Suite} =~ m/wheezy/ ? "/bin/ip" : "/sbin/ip";
         file_simple_write_contents
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#36 lore raw

PATCH 29/62 ts-host-install: Put canary in 70-persistent-net.rules

In reply to Ian Jackson

This will allow us to see if the initramfs's network names are being properly copied to the installed system. Ie, this is just a debugging aid.

---
 ts-host-install | 4 ++++
 1 file changed, 4 insertions(+)
ts-host-install +4 -0
index 7423eb9b..ea087a25 100755
@@ -218,6 +218,10 @@ SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="$ho->{Ether}", A
 END
     }
 
+    open CANARY, '>>', "$persistent_net_rules" or die $!;
+    print CANARY "\n# - canary - came via initramfs\n" or die $!;
+    close CANARY or die $!;
+
     my %xopts;
 
     di_special_kernel($ho, sub {
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#37 lore raw

PATCH 30/62 persistent-net: Include initramfs script to copy to target

In reply to Ian Jackson

This is the piece which actually copies the installer's network names to the target. It should not appear on the installed system, so it's not in overlay-persistent-net.

Technically this is only useful when the installer has the overlay-persistent-net in it, which is done only in ts-host-install and not in all the places where setup_netboot_firstboot is used. But without overlay-persistent-net it is harmless, and it is most convenient to put it here.

The little script fragment was copied out of a jessie debian-installer initramfs environment.

---
 Osstest/Debian.pm | 13 +++++++++++++
 1 file changed, 13 insertions(+)
Osstest/Debian.pm +13 -0
index e246c012..6309b246 100644
@@ -848,6 +848,19 @@ if [ -f /target$grub ] ; then
 fi
 END
 
+    # See comment in debian_overlays re net.ifnames=1
+    $ho->{Suite} =~ m/jessie|wheezy/ or
+	preseed_hook_installscript($ho, $sfx,
+            '/usr/lib/base-installer.d/', '05udev', <<'END');
+#!/bin/sh -e
+
+RULESDIR=etc/udev/rules.d
+
+mkdir -p /target/$RULESDIR
+cp /$RULESDIR/70-persistent-*.rules /target/$RULESDIR 2>/dev/null || true
+
+END
+
     debian_overlays($ho, sub {
 	my ($srcdir, $tfilename) = @_;
 	preseed_hook_overlay($ho, $sfx, $srcdir, $tfilename);
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#38 lore raw

PATCH 31/62 persistent-net: Set net.ifnames=0 in di_installcmdline_core

In reply to Ian Jackson
---
 Osstest/Debian.pm | 5 +++++
 1 file changed, 5 insertions(+)
Osstest/Debian.pm +5 -0
index 6309b246..aff5acd5 100644
@@ -81,6 +81,11 @@ sub debian_boot_setup ($$$$$;$) {
         $kopt .= ' '.$targkopt;
     }
 
+    # https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
+    # In fact these are anything but predictable.  We use the scheme
+    # from Debian jessie and earlier, persistent-net-generator etc.
+    $kopt .= ' net.ifnames=0';
+
     foreach my $hook ($hooks ? @$hooks : ()) {
         my $bo_hook= $hook->{EditBootOptions};
         $bo_hook->($ho, \$xenhopt, \$kopt) if $bo_hook;
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#39 lore raw

PATCH 32/62 ts-kernel-build: Enable some additional drivers for Thunder-X

In reply to Ian Jackson

Without this, our kernels do not find the storage.

---
 ts-kernel-build | 11 +++++++++++
 1 file changed, 11 insertions(+)
ts-kernel-build +11 -0
index 0bc443de..3dad7d36 100755
@@ -242,6 +242,17 @@ setopt CONFIG_MMC_DW m
 setopt CONFIG_MMC_DW_EXYNOS m
 setopt CONFIG_REGULATOR_S5M8767 m
 
+# Enable some additonal drivers for Thunder-X
+setopt CONFIG_PCI_HOST_THUNDER_PEM=y
+setopt CONFIG_PCI_HOST_THUNDER_ECAM=y
+setopt CONFIG_THUNDER_NIC_PF=m
+setopt CONFIG_THUNDER_NIC_VF=m
+setopt CONFIG_THUNDER_NIC_BGX=m
+setopt CONFIG_THUNDER_NIC_RGX=m
+setopt CONFIG_MDIO_THUNDER=m
+setopt CONFIG_I2C_THUNDERX=m
+setopt CONFIG_SPI_THUNDERX=m
+
 ####
 
 END
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#40 lore raw

PATCH 33/62 ts-xen-build: Enable ITS driver in Xen

In reply to Ian Jackson

From: Julien Grall <[email protected]>

The ITS driver was added in Xen 4.10 as a technical preview feature. However, it is required in order to boot Xen as Thunder-X because PCI devices don't support legacy interrupt.

So enable CONFIG_ITS in our Xen build.

---
 ts-xen-build | 4 ++++
 1 file changed, 4 insertions(+)
ts-xen-build +4 -0
index 6ddfc533..1762cd61 100755
@@ -128,6 +128,10 @@ END
 		echo >>xen/.config CONFIG_EXPERT=y
 		echo >>xen/.config CONFIG_HVM_FEP=y
 		echo >>xen/.config CONFIG_VERBOSE_DEBUG=y
+		# ITS driver is required to boot the Hardware Domain
+		# on Xen. For now (Xen 4.10/4.11 at at least),
+		# will be not built by default and gated by expert mode
+		echo >>xen/.config CONFIG_HAS_ITS=y
 	fi
 END
                );
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#41 lore raw

PATCH 34/62 TestSupport: Move `stashed files' and `next_unique_name' earlier

In reply to Ian Jackson

We are going to make more use of this in intervening code.

Pure code motion.

---
 Osstest/TestSupport.pm | 78 +++++++++++++++++++++++++-------------------------
 1 file changed, 39 insertions(+), 39 deletions(-)
Osstest/TestSupport.pm +39 -39
index 41f6f5f8..ce346097 100644
@@ -300,6 +300,45 @@ sub get_filecontents ($;$) {
     return $data;
 }
 
+#---------- stashed files ----------
+
+sub next_unique_name ($) {
+    my ($fnref) = @_;
+    my $num = $$fnref =~ s/\+([1-9]\d*)$// ? $1 : 0;
+    $$fnref .= '+'.($num+1);
+}
+
+sub open_unique_stashfile ($) {
+    my ($leafref) = @_;
+    my $dh;
+    for (;;) {
+        my $df= $$leafref;
+        $dh= new IO::File "$stash/$df", O_RDWR|O_EXCL|O_CREAT;
+        last if $dh;
+        die "$df $!" unless $!==&EEXIST;
+        next_unique_name $leafref;
+    }
+    return $dh;
+}
+
+sub get_stashed ($$) {
+    my ($param, $otherflightjob) = @_; 
+    # may be run outside transaction, or with flights locked
+    my ($oflight, $ojob) = otherflightjob($otherflightjob);
+    my $path= get_runvar($param, $otherflightjob);
+    die "$path $& " if
+        $path =~ m,[^-+._0-9a-zA-Z/], or
+        $path =~ m/\.\./;
+    return "$c{Stash}/$oflight/$ojob/$path";
+}
+
+sub compress_stashed($) {
+    my ($path) = @_;
+    return unless -e "$stash/$path";
+    my $r= system 'gzip','-9vf','--',"$stash/$path";
+    die "$r $!" if $r;
+}
+
 #---------- runvars ----------
 
 sub store_runvar ($$) {
@@ -597,12 +636,6 @@ sub target_file_exists ($$) {
     die "$rfile $out ?";
 }
 
-sub next_unique_name ($) {
-    my ($fnref) = @_;
-    my $num = $$fnref =~ s/\+([1-9]\d*)$// ? $1 : 0;
-    $$fnref .= '+'.($num+1);
-}
-
 our $target_editfile_cancel_exception =
     bless { }, 'Osstest::TestSupport::TargetEditfileCancelException';
 
@@ -1369,39 +1402,6 @@ sub hostnamepath ($) {
     join '_', reverse @l;
 }
 
-#---------- stashed files ----------
-
-sub open_unique_stashfile ($) {
-    my ($leafref) = @_;
-    my $dh;
-    for (;;) {
-        my $df= $$leafref;
-        $dh= new IO::File "$stash/$df", O_RDWR|O_EXCL|O_CREAT;
-        last if $dh;
-        die "$df $!" unless $!==&EEXIST;
-        next_unique_name $leafref;
-    }
-    return $dh;
-}
-
-sub get_stashed ($$) {
-    my ($param, $otherflightjob) = @_; 
-    # may be run outside transaction, or with flights locked
-    my ($oflight, $ojob) = otherflightjob($otherflightjob);
-    my $path= get_runvar($param, $otherflightjob);
-    die "$path $& " if
-        $path =~ m,[^-+._0-9a-zA-Z/], or
-        $path =~ m/\.\./;
-    return "$c{Stash}/$oflight/$ojob/$path";
-}
-
-sub compress_stashed($) {
-    my ($path) = @_;
-    return unless -e "$stash/$path";
-    my $r= system 'gzip','-9vf','--',"$stash/$path";
-    die "$r $!" if $r;
-}
-
 #---------- other stuff ----------
 
 sub common_toolstack ($) {
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#42 lore raw

PATCH 35/62 TestSupport: Move `target_somefile_getleaf' earlier

In reply to Ian Jackson

We are going to make more use of this in intervening code.

Pure code motion.

---
 Osstest/TestSupport.pm | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
Osstest/TestSupport.pm +9 -9
index ce346097..4e2f120a 100644
@@ -504,6 +504,15 @@ sub remote_perl_script_done ($) {
     !$? or die "$thing->{What} $?";
 }
 
+sub target_somefile_getleaf ($$$) {
+    my ($lleaf_ref, $rdest, $ho) = @_;
+    if (!defined $$lleaf_ref) {
+        $$lleaf_ref= $rdest;
+        $$lleaf_ref =~ s,.*/,,;
+    }
+    $$lleaf_ref= hostnamepath($ho)."--$$lleaf_ref";
+}
+
 sub sshuho ($$) { my ($user,$ho)= @_; return "$user\@$ho->{Ip}"; }
 
 sub sshopts () {
@@ -591,15 +600,6 @@ sub target_install_packages_norec ($@) {
     target_run_pkgmanager_install($ho,\@packages,1);
 }
 
-sub target_somefile_getleaf ($$$) {
-    my ($lleaf_ref, $rdest, $ho) = @_;
-    if (!defined $$lleaf_ref) {
-        $$lleaf_ref= $rdest;
-        $$lleaf_ref =~ s,.*/,,;
-    }
-    $$lleaf_ref= hostnamepath($ho)."--$$lleaf_ref";
-}
-
 sub tpfcs_core {
     my ($tputfilef,$ho,$timeout,$filedata, $rdest,$lleaf) = @_;
     target_somefile_getleaf(\$lleaf,$rdest,$ho);
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#43 lore raw

PATCH 36/62 TestSupport: target_somefile_leaf rename and change a variable

In reply to Ian Jackson

Rename this function. getleaf' contains get' which makes it sound like the function copies something, or returns answers suitable for getting, or something.

Also rename $rdest' to $rfile' since it might be a source too. (Although we are not about to make it a source...)

---
 Osstest/TestSupport.pm | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
Osstest/TestSupport.pm +5 -5
index 4e2f120a..a5870e4d 100644
@@ -504,10 +504,10 @@ sub remote_perl_script_done ($) {
     !$? or die "$thing->{What} $?";
 }
 
-sub target_somefile_getleaf ($$$) {
-    my ($lleaf_ref, $rdest, $ho) = @_;
+sub target_somefile_leaf ($$$) {
+    my ($lleaf_ref, $rfile, $ho) = @_;
     if (!defined $$lleaf_ref) {
-        $$lleaf_ref= $rdest;
+        $$lleaf_ref= $rfile;
         $$lleaf_ref =~ s,.*/,,;
     }
     $$lleaf_ref= hostnamepath($ho)."--$$lleaf_ref";
@@ -602,7 +602,7 @@ sub target_install_packages_norec ($@) {
 
 sub tpfcs_core {
     my ($tputfilef,$ho,$timeout,$filedata, $rdest,$lleaf) = @_;
-    target_somefile_getleaf(\$lleaf,$rdest,$ho);
+    target_somefile_leaf(\$lleaf,$rdest,$ho);
 
     my $h= new IO::File "$stash/$lleaf", 'w' or die "$lleaf $!";
     print $h $filedata or die $!;
@@ -654,7 +654,7 @@ sub teditfileex {
     if (!defined $rdest) {
         $rdest= $rfile;
     }
-    target_somefile_getleaf(\$lleaf,$rdest,$ho);
+    target_somefile_leaf(\$lleaf,$rdest,$ho);
     my $lfile;
     
     for (;;) {
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#44 lore raw

PATCH 37/62 TestSupport: Provide target_getfile_root_stash

In reply to Ian Jackson

This convenient function selects a local filename based on a target filename, and copies the target file to the selected stash file.

---
 Osstest/TestSupport.pm | 9 +++++++++
 1 file changed, 9 insertions(+)
Osstest/TestSupport.pm +9 -0
index a5870e4d..ec867e4f 100644
@@ -61,6 +61,7 @@ BEGIN {
                       target_cmd_inputfh_root sshuho
                       target_getfile target_getfile_root
                       target_putfile target_putfile_root
+		      target_getfile_root_stash
                       target_putfilecontents_stash
 		      target_putfilecontents_root_stash
                       target_put_guest_image target_editfile
@@ -553,6 +554,14 @@ sub target_getfile_root ($$$$) {
     my ($ho,$timeout, $rsrc,$ldst) = @_;
     tgetfileex('root', @_);
 }
+sub target_getfile_root_stash ($$$;$) {
+    my ($ho,$timeout,$rsrc, $lleaf) = @_; # => full path of local file
+    target_somefile_leaf(\$lleaf, $rsrc, $ho);
+    open_unique_stashfile(\$lleaf); # discard filehandle, leave file
+    my $lfile = "$stash/$lleaf";
+    target_getfile_root($ho,$timeout,$rsrc,$lfile);
+    return $lfile;
+}
 
 sub tputfileex {
     my ($ruser, $ho,$timeout, $lsrc,$rdst, $rsync) = @_;
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#45 lore raw

PATCH 38/62 ts-guests-nbd-mirror: Use target_getfile_root_stash

In reply to Ian Jackson

That removes the rather ad-hoc open-coding.

---
 ts-guests-nbd-mirror | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
ts-guests-nbd-mirror +1 -4
index 06903aaa..0365c47b 100755
@@ -154,10 +154,7 @@ sub shuffleconfigs () {
 	my $gn= $gns[$i];
 	my $gho= $ghos[$i];
 	my $cfgpath= $r{ "$gho->{Guest}_cfgpath" };
-	my $file= $cfgpath;
-	$file=~ s,/,-,g;
-	$file= "$stash/".hostnamepath($cho)."--$file";
-	target_getfile_root($sho, 60, $cfgpath, $file);
+	my $file = target_getfile_root_stash($sho, 60, $cfgpath);
 	target_putfile_root($cho, 60, $file, $cfgpath);
     }
 }
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#46 lore raw

PATCH 39/62 preseed_base: chmod ssh host private keys to placate sshd

In reply to Ian Jackson

Otherwise:
Could not load host key: /etc/ssh/ssh_host_ecdsa_key @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Permissions 0640 for '/etc/ssh/ssh_host_ed25519_key' are too open. It is recommended that your private key files are NOT accessible by others. This private key will be ignored.
key_load_private: bad permissions
Could not load host key: /etc/ssh/ssh_host_ed25519_key

This seems to start happening with stretch. Presumably stretch is more annoyingly picky than jessie.

---
 Osstest/Debian.pm | 8 ++++++++
 1 file changed, 8 insertions(+)
Osstest/Debian.pm +8 -0
index aff5acd5..d76dd03d 100644
@@ -871,6 +871,14 @@ END
 	preseed_hook_overlay($ho, $sfx, $srcdir, $tfilename);
     });
 
+    # Host private keys in the overlays have to be group-readable
+    # at least, or no-one can use them.  But ssh is very fussy.
+    preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+chmod 600 /target/etc/ssh/ssh_host_*_key ||:
+END
+
     my $preseed = <<"END";
 d-i debian-installer/locale string en_GB
 d-i console-keymaps-at/keymap select gb
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#47 lore raw

PATCH 40/62 preseed_base: break out debian_overlays_fixup_cmd

In reply to Ian Jackson

We are going to want this for guests too.

---
 Osstest/Debian.pm | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
Osstest/Debian.pm +11 -3
index d76dd03d..78d242e4 100644
@@ -36,7 +36,7 @@ BEGIN {
     @EXPORT      = qw(debian_boot_setup
                       di_installer_path di_special_kernel
                       setupboot_bootloader_edited_rune
-                      debian_overlays
+                      debian_overlays debian_overlays_fixup_cmd
                       debian_guest_suite debian_guest_di_version
                       %preseed_cmds
                       preseed_base
@@ -815,6 +815,14 @@ sub debian_overlays ($$) {
     $maybe->("$c{OverlayLocal}-$suite", 'overlay-local-$suite.tar');
 }
 
+sub debian_overlays_fixup_cmd ($;$) {
+    my ($ho, $subdir) = @_;
+    $subdir //= '';
+    return <<END;
+chmod 600 $subdir/etc/ssh/ssh_host_*_key ||:
+END
+}
+
 sub preseed_base ($$$;@) {
     my ($ho,$sfx,$extra_packages,%xopts) = @_;
 
@@ -873,10 +881,10 @@ END
 
     # Host private keys in the overlays have to be group-readable
     # at least, or no-one can use them.  But ssh is very fussy.
-    preseed_hook_command($ho, 'late_command', $sfx, <<END);
+    preseed_hook_command($ho, 'late_command', $sfx,
+			 <<END. debian_overlays_fixup_cmd($ho, '/target'));
 #!/bin/sh
 set -ex
-chmod 600 /target/etc/ssh/ssh_host_*_key ||:
 END
 
     my $preseed = <<"END";
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#48 lore raw

PATCH 41/62 ts-debian-fixup: Use debian_overlays_fixup_cmd

In reply to Ian Jackson

Otherwise we get the same error for guests as was prevented for hosts by "preseed_base: chmod ssh host private keys to placate sshd".

---
 ts-debian-fixup | 5 +++++
 1 file changed, 5 insertions(+)
ts-debian-fixup +5 -0
index 0e553d47..7d9d0398 100755
@@ -206,7 +206,12 @@ savecfg();
 ether();
 target_kernkind_check($gho);
 access();
+
 debian_overlays($ho, \&overlay);
+target_cmd_root($ho, <<END.debian_overlays_fixup_cmd($ho, $mountpoint));
+set -ex
+END
+
 console();
 filesystems();
 otherfixupcfg();
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#49 lore raw

PATCH 42/62 preseed_hook_command: allow specifying di keys other than preseed/*

In reply to Ian Jackson

Ie, only add preseed/ if there is not already a slash.

No functional change with existing call sites other than urls and temporary filenames.

---
 Osstest/Debian.pm | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
Osstest/Debian.pm +6 -3
index 78d242e4..f8ff4f46 100644
@@ -1472,9 +1472,12 @@ END
 
 sub preseed_hook_command ($$$$) {
     my ($ho, $di_key, $sfx, $text) = @_;
+    $di_key = "preseed/$di_key" unless $di_key =~ m{/};
+    my $basename = $di_key;
+    $basename =~ s{/}{--};
     my $ix= $#{ $preseed_cmds{$di_key} } + 1;
-    my $url= create_webfile($ho, "$di_key-$ix$sfx", $text);
-    my $file= "/tmp/$di_key-$ix";
+    my $url= create_webfile($ho, "$basename-$ix$sfx", $text);
+    my $file= "/tmp/$basename-$ix";
     my $cmd_cmd= "$preseed_wget -O $file '$url' && chmod +x $file && $file";
     push @{ $preseed_cmds{$di_key} }, $cmd_cmd;
 }
@@ -1521,7 +1524,7 @@ END
 sub preseed_hook_cmds () {
     my $preseed;
     foreach my $di_key (keys %preseed_cmds) {
-        $preseed .= "d-i preseed/$di_key string ".
+        $preseed .= "d-i $di_key string ".
             (join ' && ', @{ $preseed_cmds{$di_key} }). "\n";
     }
     return $preseed;
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#50 lore raw

PATCH 43/62 preseed_hook_installscript: Use partman/early_command, not preseed/

In reply to Ian Jackson

On iso-based installs, with stretch, preseed/early_command runs before the network is up. This causes the install to fail.

Our existing call sites add things to
/usr/lib/base-installer.d/
/lib/partman/init.d/
for which this is still early enough.

---
 Osstest/Debian.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Osstest/Debian.pm +2 -1
index f8ff4f46..3afea62b 100644
@@ -1484,11 +1484,12 @@ sub preseed_hook_command ($$$$) {
 
 sub preseed_hook_installscript ($$$$$) {
     my ($ho, $sfx, $installer_dir, $installer_leaf, $data) = @_;
+    # the specified script is installed via partman/early_command
     my $installer_pathname= "$installer_dir/$installer_leaf";
     my $urlfile= $installer_pathname;
     $urlfile =~ s/[^-_0-9a-z]/ sprintf "X%02x", ord($&) /ge;
     my $url= create_webfile($ho, $urlfile, $data);
-    preseed_hook_command($ho, 'early_command', $sfx, <<END);
+    preseed_hook_command($ho, 'partman/early_command', $sfx, <<END);
 #!/bin/sh
 set -ex
 mkdir -p '$installer_dir'
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#51 lore raw

PATCH 44/62 Debian: preseed_hook_installscript: New $atonce option

In reply to Ian Jackson

This runs the script right away as well as merely installing it.

---
 Osstest/Debian.pm | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
Osstest/Debian.pm +6 -3
index 3afea62b..d1b52b5b 100644
@@ -1482,20 +1482,23 @@ sub preseed_hook_command ($$$$) {
     push @{ $preseed_cmds{$di_key} }, $cmd_cmd;
 }
 
-sub preseed_hook_installscript ($$$$$) {
-    my ($ho, $sfx, $installer_dir, $installer_leaf, $data) = @_;
+sub preseed_hook_installscript ($$$$$;$) {
+    my ($ho, $sfx, $installer_dir, $installer_leaf, $data, $atonce) = @_;
     # the specified script is installed via partman/early_command
     my $installer_pathname= "$installer_dir/$installer_leaf";
     my $urlfile= $installer_pathname;
     $urlfile =~ s/[^-_0-9a-z]/ sprintf "X%02x", ord($&) /ge;
     my $url= create_webfile($ho, $urlfile, $data);
-    preseed_hook_command($ho, 'partman/early_command', $sfx, <<END);
+    preseed_hook_command($ho, 'partman/early_command', $sfx,
+			 <<END.($atonce?<<END_ATONCE:''));
 #!/bin/sh
 set -ex
 mkdir -p '$installer_dir'
 $preseed_wget -O '$installer_pathname' '$url'
 chmod +x '$installer_pathname'
 END
+'$installer_pathname'
+END_ATONCE
 }
 
 sub preseed_hook_overlay ($$$$) {
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#52 lore raw

PATCH 45/62 Debian: partman scripts: Run right away too

In reply to Ian Jackson

We are switching the installation of these to partman/early_command which runs as a result of a /lib/partman/init.d hook. That means that things we install don't get picked up, so run them right away (too).

---
 Osstest/Debian.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Osstest/Debian.pm +2 -2
index d1b52b5b..9dec321f 100644
@@ -1157,7 +1157,7 @@ sub preseed_create ($$;@) {
     my $d_i = $ho->{Tftp}{Path}.'/'.di_installer_path($ho);
 
     preseed_hook_installscript($ho, $sfx,
-          '/lib/partman/init.d', '000override-parted-devices', <<END);
+          '/lib/partman/init.d', '000override-parted-devices', <<END, 1);
 #!/bin/sh
 set -ex
 cd /bin
@@ -1171,7 +1171,7 @@ chmod +x parted_devices
 END
 
     preseed_hook_installscript($ho, $sfx,
-          '/lib/partman/init.d', '25erase-other-disks', <<END);
+          '/lib/partman/init.d', '25erase-other-disks', <<END, 1);
 #!/bin/sh
 set -ex
 stamp=/var/erase-other-disks.stamp
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#53 lore raw

PATCH 46/62 Debian: set partman-lvm/device_remove_lvm_span

In reply to Ian Jackson

Web searching[1] suggests that this suppresses this error:

!! ERROR: Unable to automatically remove LVM data Because the volume group(s) on the selected device also consist of physical volumes on other devices, it is not considered safe to remove its LVM data automatically. If you wish to use this device for partitioning, please remove its LVM data first.

[1] eg https://serverfault.com/questions/571363/unable-to-automatically-remove-lvm-data

It doesn't, though. I am only experiencing it now because the ad-hoc disk-erasing (25erase-other-disks) is broken for other reasons. But let's have it anyway as it sounds like a thing we might want.

---
 Osstest/Debian.pm | 1 +
 1 file changed, 1 insertion(+)
Osstest/Debian.pm +1 -0
index 9dec321f..073b776c 100644
@@ -907,6 +907,7 @@ d-i partman-partitioning/confirm_write_new_label boolean true
 d-i partman/choose_partition select finish
 d-i partman/confirm boolean true
 d-i partman-lvm/confirm boolean true
+d-i partman-lvm/device_remove_lvm_span boolean true
 
 d-i partman/confirm_nooverwrite true
 d-i partman-lvm/confirm_nooverwrite true
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#54 lore raw

PATCH 47/62 Debian: Add reference to bug numbers for erase-other-disks

In reply to Ian Jackson
---
 Osstest/Debian.pm | 1 +
 1 file changed, 1 insertion(+)
Osstest/Debian.pm +1 -0
index 073b776c..79b7960d 100644
@@ -1171,6 +1171,7 @@ END2
 chmod +x parted_devices
 END
 
+    # Work around Debian bugs #601363 #687160
     preseed_hook_installscript($ho, $sfx,
           '/lib/partman/init.d', '25erase-other-disks', <<END, 1);
 #!/bin/sh
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#55 lore raw

PATCH 48/62 dm restrict audit: actually install right package for fishdescriptor

In reply to Ian Jackson

In
dm restrict audit: install newer chiark-scripts for fishdescriptor
a locally-provided chiark-scripts_6.0.2_all.deb was installed for jessie. For stretch a backport was installed, but mistakenly of chiark-utils-bin rather than chiark-scripts.

---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
index 79b7960d..9aa88822 100644
@@ -1060,7 +1060,7 @@ sub preseed_create_guest ($$$;@) {
     if (grep { m/_dmrestrict$/ && $r{$_} } keys %r and
 	$suite =~ m/stretch/) {
 	preseed_backports_packages($ho, $sfx, \%xopts, $suite,
-				   qw(chiark-utils-bin));
+				   qw(chiark-scripts));
     }
 
     my $preseed_file= preseed_base($ho, $sfx, $extra_packages, %xopts);
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#56 lore raw

PATCH 49/62 Debian: Move preseed_backports_packages earlier

In reply to Ian Jackson

No functional change.

---
 Osstest/Debian.pm | 88 +++++++++++++++++++++++++++----------------------------
 1 file changed, 44 insertions(+), 44 deletions(-)
Osstest/Debian.pm +44 -44
index 9aa88822..25bf8e88 100644
@@ -823,6 +823,50 @@ chmod 600 $subdir/etc/ssh/ssh_host_*_key ||:
 END
 }
 
+sub preseed_backports_packages ($$$$@) {
+    my ($ho, $sfx, $xopts, $suite, @pkgs) = @_;
+
+    if (! $xopts->{BackportsSourcesAlreadyAdded}++) {
+	my $bp_url = $c{"DebianSnapshotBackports_$suite"};
+	$bp_url ||= "http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath}";
+
+	my $apt_insert='';
+	my $extra_rune='';
+	if ($suite =~ m/wheezy|jessie/) {
+	    # this has global effect, unfortunately
+	    $extra_rune = <<END;
+d=/target/etc/apt/apt.conf.d
+mkdir -p \$d
+cat >\$d/50osstestsnapshot <<EOF
+Acquire::Check-Valid-Until false;
+EOF
+END
+	} else {
+	    $apt_insert = '[check-valid-until=no]';
+	}
+
+	preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+
+cat <<EOF >>/target/etc/apt/sources.list
+
+# $suite backports
+deb $apt_insert $bp_url $suite-backports main
+EOF
+$extra_rune
+in-target apt-get update
+END
+    }
+
+    preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+
+in-target apt-get install -y -t $suite-backports @pkgs
+END
+}
+
 sub preseed_base ($$$;@) {
     my ($ho,$sfx,$extra_packages,%xopts) = @_;
 
@@ -986,50 +1030,6 @@ END
     return $preseed;
 }
 
-sub preseed_backports_packages ($$$$@) {
-    my ($ho, $sfx, $xopts, $suite, @pkgs) = @_;
-
-    if (! $xopts->{BackportsSourcesAlreadyAdded}++) {
-	my $bp_url = $c{"DebianSnapshotBackports_$suite"};
-	$bp_url ||= "http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath}";
-
-	my $apt_insert='';
-	my $extra_rune='';
-	if ($suite =~ m/wheezy|jessie/) {
-	    # this has global effect, unfortunately
-	    $extra_rune = <<END;
-d=/target/etc/apt/apt.conf.d
-mkdir -p \$d
-cat >\$d/50osstestsnapshot <<EOF
-Acquire::Check-Valid-Until false;
-EOF
-END
-	} else {
-	    $apt_insert = '[check-valid-until=no]';
-	}
-
-	preseed_hook_command($ho, 'late_command', $sfx, <<END);
-#!/bin/sh
-set -ex
-
-cat <<EOF >>/target/etc/apt/sources.list
-
-# $suite backports
-deb $apt_insert $bp_url $suite-backports main
-EOF
-$extra_rune
-in-target apt-get update
-END
-    }
-
-    preseed_hook_command($ho, 'late_command', $sfx, <<END);
-#!/bin/sh
-set -ex
-
-in-target apt-get install -y -t $suite-backports @pkgs
-END
-}
-
 sub preseed_create_guest ($$$;@) {
     my ($ho, $arch, $sfx, %xopts) = @_;
 
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#57 lore raw

PATCH 50/62 dm restrict audit: actually install fishdescriptor in host

In reply to Ian Jackson

In
dm restrict audit: install newer chiark-scripts for fishdescriptor
arrangements were made to install a backport of chiark-scripts but the code was mistakenly placed in preseed_create_guest but of course it's needed in the host.

---
 Osstest/Debian.pm | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
Osstest/Debian.pm +6 -5
index 25bf8e88..8d53bbe1 100644
@@ -1022,6 +1022,12 @@ END
     # security.d.o CDN seems unreliable right now
     # and jessie-updates is no more
 
+    if (grep { m/_dmrestrict$/ && $r{$_} } keys %r and
+	$suite =~ m/stretch/) {
+	preseed_backports_packages($ho, $sfx, \%xopts, $suite,
+				   qw(chiark-scripts));
+    }
+
     $preseed .= <<"END";
 
 ### END OF DEBIAN PRESEED BASE
@@ -1057,11 +1063,6 @@ sub preseed_create_guest ($$$;@) {
             $extra_packages = "pv-grub-menu";
         }
     }
-    if (grep { m/_dmrestrict$/ && $r{$_} } keys %r and
-	$suite =~ m/stretch/) {
-	preseed_backports_packages($ho, $sfx, \%xopts, $suite,
-				   qw(chiark-scripts));
-    }
 
     my $preseed_file= preseed_base($ho, $sfx, $extra_packages, %xopts);
     $preseed_file.= (<<END);
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#58 lore raw

PATCH 51/62 dm restrict audit: always install (some) chiark-scripts

In reply to Ian Jackson

In
dm restrict audit: install newer chiark-scripts for fishdescriptor
arrangements were made to install suitable chiark-scripts for for jessie and stretch.

For buster and later, the mainline Debian version of chiark-scripts is indeed sufficient, but nothing installed it. Do that.

---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
index 8d53bbe1..9d7d1518 100644
@@ -997,7 +997,7 @@ d-i apt-setup/another boolean false
 d-i apt-setup/non-free boolean false
 d-i apt-setup/contrib boolean false
 
-d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, chiark-utils-bin, wget, $extra_packages
+d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, chiark-utils-bin, chiark-scripts, wget, $extra_packages
 
 d-i grub-installer/force-efi-extra-removable boolean true
 
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#59 lore raw

PATCH 52/62 target_install_packages: Consistently use qw(...) rather than '...'

In reply to Ian Jackson

qw(...) splits its argument into words.

There is one semantic change, where two package names were passed in a single argument. That worked by accident.

---
 ts-xen-install | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
ts-xen-install +5 -5
index 80952857..03f8c03e 100755
@@ -53,18 +53,18 @@ sub packages () {
 			       qemu-utils
                                netcat-openbsd));
     if ($ho->{Suite} !~ m/squeeze|wheezy|jessie/) {
-        target_install_packages($ho, 'net-tools libnl-route-3-200');
+        target_install_packages($ho, qw(net-tools libnl-route-3-200));
     }
     if ($ho->{Suite} =~ m/jessie/) {
-        target_install_packages($ho, 'libnl-route-3-200');
+        target_install_packages($ho, qw(libnl-route-3-200));
     }
     target_install_packages($ho,
 			    $ho->{Suite} =~ /squeeze/ ? "libyajl1" : "libyajl2");
     if ($ho->{Suite} !~ m/lenny|squeeze/) {
-        target_install_packages($ho, 'libfdt1');
+        target_install_packages($ho, qw(libfdt1));
     }
     if ($r{arch} eq 'i386') {
-	target_install_packages($ho, 'libc6-xen');
+	target_install_packages($ho, qw(libc6-xen));
     }
     target_install_packages($ho, @{toolstack($ho)->{ExtraPackages}})
         if toolstack($ho)->{ExtraPackages};
@@ -88,7 +88,7 @@ sub some_extradebs ($) {
 	    $ontarget = "extrapackages-$cfgvar-$counter"; $counter++;
 	    $dpkgopts = '-iGROEB';
 	    logm("$cfgvar: updating packages from directory $path");
-	    target_install_packages($ho, 'rsync') unless $rsync_installed++;
+	    target_install_packages($ho, qw(rsync)) unless $rsync_installed++;
 	    target_putfile_root($ho,300, "$path/.", $ontarget, '-r');
 	} elsif ($path =~ m{\.deb$}) {
 	    $path =~ s{_\.deb}{ "_$r{arch}.deb" }e;
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#60 lore raw

PATCH 53/62 ts-xen-install: Install libpciaccess0

In reply to Ian Jackson

In "ts-xen-build-prep: install packages for suites >jessie", libpciaccess-dev was added for the benefit of qemu. libvirt needs it too.

We also need the runtime library. Without it, libvirt does not start:

  2019-04-04 22:35:36.760+0000: 3623: error : virModuleLoadFile:53 : internal error: Failed to load module '/usr/local/lib/libvirt/connection-driver/libvirt_driver_nodedev.so': libpciaccess.so.0: cannot open shared object file: No such file or directory
---
 ts-xen-install | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
ts-xen-install +2 -1
index 03f8c03e..9f78a75f 100755
@@ -53,7 +53,8 @@ sub packages () {
 			       qemu-utils
                                netcat-openbsd));
     if ($ho->{Suite} !~ m/squeeze|wheezy|jessie/) {
-        target_install_packages($ho, qw(net-tools libnl-route-3-200));
+        target_install_packages($ho, qw(net-tools libnl-route-3-200 
+					libpciaccess0));
     }
     if ($ho->{Suite} =~ m/jessie/) {
         target_install_packages($ho, qw(libnl-route-3-200));
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#61 lore raw

PATCH 54/62 ts-debian-hvm-install: Honour linux_boot_append target var

In reply to Ian Jackson

This looks for:
<guest>_linux_boot_append
all_guest_linux_boot_append

Nothing sets these yet.

---
 ts-debian-hvm-install | 3 +++
 1 file changed, 3 insertions(+)
ts-debian-hvm-install +3 -0
index 54d5d1c2..ed8803ed 100755
@@ -129,6 +129,9 @@ sub gcmdline (;$) {
 					    PreseedScheme => 'file');
     push @dicmdline, $extra if $extra;
 
+    my $append = target_var($gho,'linux_boot_append');
+    push @dicmdline, $append if $append;
+
     push @dicmdline, "--";
     # See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762007 for
     # why console= is repeated.
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#62 lore raw

PATCH 55/62 make-flight: shadow test: Disable kpti in guests

In reply to Ian Jackson

Since Spectre/Meltdown, shadow has been a lot slower, especially with KPTI in the guest. Empirically, too slow (with the kernel from Debian stretch).

---
 make-flight | 1 +
 1 file changed, 1 insertion(+)
make-flight +1 -0
index 151b1435..1e3ebd5c 100755
@@ -493,6 +493,7 @@ do_shadow_tests () {
             $debian_runvars all_hostflags=$most_hostflags
 
   do_hvm_debian_test_one debianhvm xl seabios false '' -shadow \
+                         all_guest_linux_boot_append=nopti \
                          "xen_boot_append=hap=false"
 }
 
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#63 lore raw

Re: [OSSTEST PATCH 55/62] make-flight: shadow test: Disable kpti in guests

Quoted 3 lines · Ian Jackson

Since Spectre/Meltdown, shadow has been a lot slower, especially with KPTI in the guest. Empirically, too slow (with the kernel from Debian stretch).

The speed of shadow pagetables hasn't changed - I don't think we even touched the shadow code at all for XSA-254.

The problem is the change in guest behaviour as a consequence of needing KPTI for a Meltdown mitigation.

The guest now flushes its pagetables on every syscall/interrupt/exception rather than once on a process=>process context switch, which is why running a guest using KPTI in shadow mode is boarderline unusable.

The actual change to use nopti looks fine.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#64 lore raw

Re: [OSSTEST PATCH 55/62] make-flight: shadow test: Disable kpti in guests

Andrew Cooper writes ("Re: [OSSTEST PATCH 55/62] make-flight: shadow test: Disable kpti in guests"):

Quoted 11 lines · Ian Jackson
Quoted 3 lines

Since Spectre/Meltdown, shadow has been a lot slower, especially with KPTI in the guest. Empirically, too slow (with the kernel from Debian stretch).

The speed of shadow pagetables hasn't changed - I don't think we even touched the shadow code at all for XSA-254.

The problem is the change in guest behaviour as a consequence of needing KPTI for a Meltdown mitigation.

Quoted 6 lines

The guest now flushes its pagetables on every syscall/interrupt/exception rather than once on a process=>process context switch, which is why running a guest using KPTI in shadow mode is boarderline unusable.

The actual change to use nopti looks fine.

Thanks for the explanation. Unfortunately for the reasons I've explained I won't change the commit message because that would involve rewriting the commit, but it is useful to have this correction in the archives.

Regards,
Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#65 lore raw

PATCH 56/62 platforms: Pass suite to get_arch_platforms

In reply to Ian Jackson

The available platforms may depend on the suite to be used.

For now we use $defsuite from make-flight, which is not entirely right but it will do for now because we don't use other suites much.

No functional change yet since neither implementation uses it.

---
 Osstest/HostDB/Executive.pm | 4 ++--
 Osstest/HostDB/Static.pm    | 4 ++--
 cri-getplatforms            | 2 +-
 make-flight                 | 4 ++--
 4 files changed, 7 insertions(+), 7 deletions(-)
Osstest/HostDB/Executive.pm +2 -2
index d42250f6..bf6ec57d 100644
@@ -90,8 +90,8 @@ END
     return $flags;
 }
 
-sub get_arch_platforms ($$) {
-    my ($hd, $blessing, $arch) = @_;
+sub get_arch_platforms ($$$) {
+    my ($hd, $blessing, $arch, $suite) = @_;
 
     my @plats = ( );
     my $platsq = $dbh_tests->prepare(<<END);
Osstest/HostDB/Static.pm +2 -2
index ab9542dd..0c6be3ee 100644
@@ -72,8 +72,8 @@ sub get_flags ($$) { #method
     return $flags;
 }
 
-sub get_arch_platforms ($$) {
-    my ($hd, $blessing, $arch) = @_;
+sub get_arch_platforms ($$$) {
+    my ($hd, $blessing, $arch, $suite) = @_;
 
     my $prop = "Platforms".ucfirst($arch);
 
cri-getplatforms +1 -1
index 2bf13483..2b8cee0b 100755
@@ -20,6 +20,6 @@ getplatforms () {
         perl -e '
                 use Osstest;
                 csreadconfig();
-                print join " ", $mhostdb->get_arch_platforms("'$blessing'", "'$1'") or die $!;
+                print join " ", $mhostdb->get_arch_platforms("'$blessing'", "'$1'", "'$2'") or die $!;
         '
 }
make-flight +2 -2
index 1e3ebd5c..92dacb35 100755
@@ -27,9 +27,9 @@ buildflight=$4
 flight=`./cs-flight-create $blessing $branch`
 
 . ./cri-common
-. ./cri-getplatforms
 . ./ap-common
 . ./mfi-common
+. ./cri-getplatforms
 
 # Older versions of Xen may not build with the current default.  Note
 # that branches older than 4.3 might need something even older than
@@ -612,7 +612,7 @@ do_pv_debian_tests () {
 
   for xsm in $xsms ; do
     # Basic PV Linux test with xl
-    for platform in '' `getplatforms $xenarch` ; do
+    for platform in '' `getplatforms $xenarch $defsuite` ; do
 
       # xsm test is not platform specific
       if [ x$xsm = xtrue -a x$platform != x ]; then
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#66 lore raw

PATCH 57/62 platforms: Honour suite in get_arch_platforms

In reply to Ian Jackson

The available platforms may depend on the suite to be used.

Actually implement that for HostDB::Executive.
For Static, we leave it to the user.

---
 Osstest/HostDB/Executive.pm | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
Osstest/HostDB/Executive.pm +4 -2
index bf6ec57d..7ffca6c4 100644
@@ -99,18 +99,20 @@ SELECT DISTINCT hostflag
            FROM hostflags h0
    WHERE EXISTS (
        SELECT *
-         FROM hostflags h1, hostflags h2, hostflags h3
+         FROM hostflags h1, hostflags h2, hostflags h3, hostflags h4
         WHERE h0.hostname = h1.hostname
           AND h1.hostname = h2.hostname
           AND h2.hostname = h3.hostname
+          AND h3.hostname = h4.hostname
           AND h1.hostflag = ?
           AND h2.hostflag = ?
           AND h3.hostflag = 'purpose-test'
+          AND h4.hostflag = ?
    )
    AND hostflag like 'platform-%';
 END
 
-    $platsq->execute("blessed-$blessing", "arch-$arch");
+    $platsq->execute("blessed-$blessing", "arch-$arch", "suite-$suite");
 
     while (my ($plat) = $platsq->fetchrow_array()) {
 	$plat =~ s/^platform-//g or die;
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#67 lore raw

PATCH 58/62 dm restrict, fishdescriptor: Update to a fixed chiark-scripts

In reply to Ian Jackson

I have only just fixed a bug which stops our test from working and the fix is not upstream yet.

---
 production-config | 1 +
 1 file changed, 1 insertion(+)
production-config +1 -0
index 59c74cca..fadfe8b9 100644
@@ -106,6 +106,7 @@ MicrocodeUpdateI386 microcode.x86.2015-06-12.cpio
 TftpGrubVersion XXXX-XX-XX
 
 DebianExtraPackages_jessie chiark-scripts_6.0.3~citrix1_all.deb
+DebianExtraPackages_stretch chiark-scripts_6.0.4~citrix1_all.deb
 
 DebianExtraPackages_uefi_i386_jessie   extradebs-uefi-i386-2018-04-01/
 DebianExtraPackages_uefi_amd64_jessie  extradebs-uefi-amd64-2018-04-01/
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#68 lore raw

PATCH 59/62 Debian: Fix /lib/udev/ifupdown-hotplug to not run if / is ro

In reply to Ian Jackson

Empirically, without this, on Debian stretch:

1. udev starts before / is remounted rw
2. udev spots eth0 and runs /lib/udev/ifupdown-hotplug
3. ifupdown-hotplug runs ifup which runs isc-dhcp-client
4. isc-dhcp-client talks to the dhcp server to get an address
5. isc-dhcp-client tries to write the lease and run the hook scripts but something here fails with EROFS
6. isc-dhcp-client sends DHCPDECLINE to the server
7. GOTO 4

This loop continues for several seconds, until / is remounted rw.

None of this is appears in any of the guest's logs because syslog is not running yet, and none of this stuff goes to the console either. But it does appear in the dhcp *server* logs so that a conscientious administrator will suffer consternation and concern.

It is not ever sensible to run /lib/udev/ifupdown-hotplug with / mounted ro. (Maybe it is not sensible to run udev so early.) Skipping it is fine because the boot sequence contains an explicit call to ifup which occurs *after* / is remounted, and that will collect any interfaces which were skipped earlier.

In my osstest stretch series development tests I don't think I saw any actual host install failures due to this situation. The timeouts are generous enough not to matter, and of course when we install Xen we reconfigure the host networking to have a static IP address so then the problem goes away.

In this patch we do this for the host. We provide a function to return the appropriate rune which we will use in a moment.

I have not yet reported this situation to the appropriate Debian channels. That's on my backlog. But in any case I have limited the workaround to stretch so we will revisit this for buster.

The approach to fixing this is somewhat awkward.

Firstly, since the bug is in /lib/udev/ifupdown-hotplug we want to edit that script. But we need to do it in the installer environment as a late_command, because after first boot, via ssh, is too late. The installer environment has no `patch'. I didn't want to just supply a whole new script. So instead we use sed and mv.

Secondly, as for the contents of /lib/udev/ifupdown-hotplug: I wasn't able to think of a convenient shell command which will tell us the errno value from trying to write a file. Plenty will print the strerror but I balked at LC_MESSAGES=C and grep. Perl, however, can do this, and is always available on Debian. So perl it is.

Thirdly, the code has a bad case of toothpicks (lots of \), because it needs to pass through (i) perl (ii) the shell (iii) sed's regexp syntax and/or i command.

---
 Osstest/Debian.pm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
Osstest/Debian.pm +22 -0
index 9d7d1518..41aa28b0 100644
@@ -48,6 +48,7 @@ BEGIN {
                       preseed_hook_cmds
                       di_installcmdline_core
                       di_vg_name
+                      debian_dhcp_rofs_fix
                       );
     %EXPORT_TAGS = ( );
 
@@ -931,6 +932,9 @@ END
 set -ex
 END
 
+    preseed_hook_command($ho, 'late_command', $sfx,
+			 debian_dhcp_rofs_fix($ho, '/target'));
+
     my $preseed = <<"END";
 d-i debian-installer/locale string en_GB
 d-i console-keymaps-at/keymap select gb
@@ -1563,4 +1567,22 @@ sub debian_guest_di_version ($) {
     return $gho->{DiVersion};
 }
 
+sub debian_dhcp_rofs_fix ($$) {
+    my ($ho, $rootdir) = @_;
+    # Works around bug where /lib/udev/ifupdown-hotplug runs while
+    # / is still ro.  In stretch, the isc dhcp client spins requesting
+    # an address and then sending a DHCPDECLINE (and then, usually,
+    # eventually works).
+    return '' unless $ho->{Suite} =~ m/stretch/;
+    my $script = "$rootdir/lib/udev/ifupdown-hotplug";
+    <<END.<<'ENDQ'.<<END
+set -ex
+sed -e <'$script' >'$script.new' \\
+END
+'/^if \[ -d \/run\/systemd\/system ]; then/ i perl -e '\''use POSIX; my $f="/var/lib/dhcp/rw-fs-check"; exit 0 if open T, ">", $f; die "quitting: $f: $!\\n" if $!==EROFS; warn "warning: $f: $!\\n"'\'' || exit 0'
+ENDQ
+mv '$script.new' '$script'
+END
+}
+
 1;
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#69 lore raw

PATCH 60/62 Debian: Fix /lib/udev/ifupdown-hotplug in guests from debootstrap

In reply to Ian Jackson

The ifupdown-hotplug problem just described affects guests too.

Empirically, this occurs only with the guests from xen-tools. In my osstest stretch series development tests this resulted in occasional failures of ts-guest-start. The problem is not deterministic; and it may be that it is a race between the guest's random dhcp retry and osstest's timeout, or something. The race affected both x86 and ARM.

I conjecture that it only affects xen-tools created guests because those guests use sysvinit. Presumably the other guests have systemd which has some different ordering. I conjecture that the sysvinit boot arrangements were damaged by some changes made in stretch to shared components (udev, probably) to support systemd.

An alternative explanation for seeing the failure only with xen-tools created guests might be that all guests are affected, but only xen-tools created guests are booted with a short timeout; whereas the d-i created ones have a long timeout for firstboot at least. I don't believe this theory because all guests are restarted with ts-guest-start and a short timeout.

So: in ts-debian-fixup, execute the sed rune to fix /lib/udev/ifupdown-hotplug. This then happens before first boot.

Debian guests installed via d-i are not affected by this patch.

---
 ts-debian-fixup | 2 ++
 1 file changed, 2 insertions(+)
ts-debian-fixup +2 -0
index 7d9d0398..fef9836e 100755
@@ -212,6 +212,8 @@ target_cmd_root($ho, <<END.debian_overlays_fixup_cmd($ho, $mountpoint));
 set -ex
 END
 
+target_cmd_root($ho, debian_dhcp_rofs_fix($ho, $mountpoint));
+
 console();
 filesystems();
 otherfixupcfg();
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#70 lore raw

PATCH 61/62 production-config-cambridge: Provide TftpDiVersion_stretch

In reply to Ian Jackson
---
 production-config-cambridge | 1 +
 1 file changed, 1 insertion(+)
production-config-cambridge +1 -0
index ce6239bd..a50b19b3 100644
@@ -77,6 +77,7 @@ TftpNetGrubTemplatesReal Netgrub.cfg/%ether%
 TftpNetbootGroup osstest
 TftpDiVersion_wheezy 2016-06-08
 TftpDiVersion_jessie 2018-06-26
+TftpDiVersion_stretch 2019-04-10
 
 DebianSnapshotBackports_jessie http://snapshot.debian.org/archive/debian/20190206T211314Z/
 
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#71 lore raw

PATCH 62/62 Switch to Debian Stretch

In reply to Ian Jackson

From: Wei Liu <[email protected]>

---
 Osstest.pm        | 2 +-
 production-config | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)
Osstest.pm +1 -1
index 92b1a0ea..7ce53fcb 100644
@@ -87,7 +87,7 @@ our %c = qw(
 
     Images images
 
-    DebianSuite jessie
+    DebianSuite stretch
     DebianMirrorSubpath debian
 
     TestHostKeypairPath id_rsa_osstest
production-config +2 -0
index fadfe8b9..5b9f5c82 100644
@@ -90,12 +90,14 @@ TftpNetbootGroup osstest
 # Update with ./mg-debian-installer-update(-all)
 TftpDiVersion_wheezy 2016-06-08
 TftpDiVersion_jessie 2018-06-26
+TftpDiVersion_stretch 2018-11-27
 
 DebianSnapshotBackports_jessie http://snapshot.debian.org/archive/debian/20190206T211314Z/
 
 # For ISO installs
 DebianImageVersion_wheezy 7.2.0
 DebianImageVersion_jessie 8.2.0
+DebianImageVersion_stretch 9.4.0
 
 # These should normally be the same.
 # Update with ./mg-cpu-microcode-update
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#72 lore raw

Re: [linux-arm-xen test] 134708: regressions - all pass

In reply to Ian Jackson

Julien Grall writes ("Re: [Xen-devel] [linux-arm-xen test] 134708: regressions - all pass"):

Quoted 3 lines · osstest service owner
Quoted 2 lines

flight 134708 linux-arm-xen real [real]
http://logs.test-lab.xenproject.org/osstest/logs/134708/

...

Quoted 8 lines
Quoted 1 line

test-arm64-arm64-examine 11 examine-serial/bootloader fail REGR. vs. 119195

I am a bit confused with the failure here [1]:

2019-04-13 22:36:55 Z ---------- substep 11 examine-serial/bootloader fail ---------- 

IIUC, we are looking for "osstest cookie". However this does not seem to appear on rochester0's logs. Any insights how "osstest cookie" will be output?

Hi. Yes, this is an issue I already discovered...

Ian Jackson writes ("[OSSTEST PATCH 00/62] Update to Debian stable (stretch)"):

Quoted 2 lines

There seem to be a number of outstanding problems which I decided were not blockers:

...

Quoted 4 lines

* The serial output from the new rochester arm64 machines is missing some of the grub bootloader messages (and this is detected by one of the examine jobs). I may need help from someone familiar with these machines' hardware/firmware.

I decided that this issue was not serious enough to be a blocker. It may need some force pushes, especially if we can't fix it soon, because osstest will try this test, in other flights, on the rochesters, where it fails, and that looks like a regression.

In particular, this is not a regression in the linux-arm-xen branch. So we could force push it.

As for the bug:

This check is checking that the osstest serial log captures the output from the bootloader. This can be quite important for debugging, especially to distinguish hardware or configuration problems from software problems. Seeing the bootloader output also provides a useful reference point to people unfamiliar with osstest.

This check is primarily a test on osstest itself; it is also part of machine commissioning. The test works by putting a magic string in the bootloader output.

Compare the failure above
http://logs.test-lab.xenproject.org/osstest/logs/134708/test-arm64-arm64-examine/info.html
with this one which ran on laxton1:
http://logs.test-lab.xenproject.org/osstest/logs/134643/test-arm64-arm64-examine/info.html

If you look at the final grub config
http://logs.test-lab.xenproject.org/osstest/logs/134643/test-arm64-arm64-examine/laxton1--grub.cfg.new
you can see things like this
menuentry 'osstest cookie 0b4b9311157321d55cca9d25ff49159d Debian GNU/Linux' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-a8e4e9c6-4829-402b-93aa-964f9cf39c41' {

In
http://logs.test-lab.xenproject.org/osstest/logs/134643/test-arm64-arm64-examine/serial-laxton1.log
near Apr 11 17:07:07.787491, you can see the osstest cookie coming out in the serial representation, in amongst the cursor positioning codes.

In
http://logs.test-lab.xenproject.org/osstest/logs/134708/test-arm64-arm64-examine/serial-rochester0.log
near Apr 13 22:34:32.442724 you see a banner from grub and then a bunch of "invalid utf-8 sequence: \304", and at the end some actual messages from grub. But the meat of the menu, including the cookie, is missing.

So the osstest considers the test a failure: the thing it arranged to appear in the bootloader output was not visible in the serial log.

I have not yet investigated this beyond making these observations. It is possible that there is a disagreement over character encoding or something, and what is happening is that line drawing characters are being misinterpreted by sympathy. It might be a firmware bug or a grub bug or a sympathy bug or a hardware design issue. I think the next steps would be to determine exactly what sequence of bytes are being sent by the host, and to investigate its BIOS options for controlling serial terminal output.

HTH.

Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#73 lore raw

rochester (Thunder-X) bootloader lost output issue

Hi. A couple of weeks ago I wrote a writeup about this kind of problem:

test-arm64-arm64-examine 11 examine-serial/bootloader fail REGR. vs. 119195

This is going to affect all of osstest's branches and it will be tiresome to force push it. Particularly, I think it may cause the examine jobs to become stuck on the rochesters and (even worse) if it doesn't it will pass by accident requiring another force push.

I would like to either fix this, or write a programmatic workaround. Would someone from the Xen ARM community be willing to debug this any time soon ?

Failing that I will probably make a hostflag that causes this test to be skipped, or something.

(Bottom-quoting my previous writeup for context.)

Thanks,
Ian.

Ian Jackson writes ("Re: [Xen-devel] [linux-arm-xen test] 134708: regressions - all pass"):

Quoted 83 lines

Julien Grall writes ("Re: [Xen-devel] [linux-arm-xen test] 134708: regressions - all pass"):

Quoted 3 lines · osstest service owner
Quoted 2 lines

flight 134708 linux-arm-xen real [real]
http://logs.test-lab.xenproject.org/osstest/logs/134708/

...

Quoted 8 lines
Quoted 1 line

test-arm64-arm64-examine 11 examine-serial/bootloader fail REGR. vs. 119195

I am a bit confused with the failure here [1]:

2019-04-13 22:36:55 Z ---------- substep 11 examine-serial/bootloader fail ---------- 

IIUC, we are looking for "osstest cookie". However this does not seem to appear on rochester0's logs. Any insights how "osstest cookie" will be output?

Hi. Yes, this is an issue I already discovered...

Ian Jackson writes ("[OSSTEST PATCH 00/62] Update to Debian stable (stretch)"):

Quoted 2 lines

There seem to be a number of outstanding problems which I decided were not blockers:

...

Quoted 4 lines

* The serial output from the new rochester arm64 machines is missing some of the grub bootloader messages (and this is detected by one of the examine jobs). I may need help from someone familiar with these machines' hardware/firmware.

I decided that this issue was not serious enough to be a blocker. It may need some force pushes, especially if we can't fix it soon, because osstest will try this test, in other flights, on the rochesters, where it fails, and that looks like a regression.

In particular, this is not a regression in the linux-arm-xen branch. So we could force push it.

As for the bug:

This check is checking that the osstest serial log captures the output from the bootloader. This can be quite important for debugging, especially to distinguish hardware or configuration problems from software problems. Seeing the bootloader output also provides a useful reference point to people unfamiliar with osstest.

This check is primarily a test on osstest itself; it is also part of machine commissioning. The test works by putting a magic string in the bootloader output.

Compare the failure above
http://logs.test-lab.xenproject.org/osstest/logs/134708/test-arm64-arm64-examine/info.html
with this one which ran on laxton1:
http://logs.test-lab.xenproject.org/osstest/logs/134643/test-arm64-arm64-examine/info.html

If you look at the final grub config
http://logs.test-lab.xenproject.org/osstest/logs/134643/test-arm64-arm64-examine/laxton1--grub.cfg.new
you can see things like this
menuentry 'osstest cookie 0b4b9311157321d55cca9d25ff49159d Debian GNU/Linux' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-a8e4e9c6-4829-402b-93aa-964f9cf39c41' {

In
http://logs.test-lab.xenproject.org/osstest/logs/134643/test-arm64-arm64-examine/serial-laxton1.log
near Apr 11 17:07:07.787491, you can see the osstest cookie coming out in the serial representation, in amongst the cursor positioning codes.

In
http://logs.test-lab.xenproject.org/osstest/logs/134708/test-arm64-arm64-examine/serial-rochester0.log
near Apr 13 22:34:32.442724 you see a banner from grub and then a bunch of "invalid utf-8 sequence: \304", and at the end some actual messages from grub. But the meat of the menu, including the cookie, is missing.

So the osstest considers the test a failure: the thing it arranged to appear in the bootloader output was not visible in the serial log.

I have not yet investigated this beyond making these observations. It is possible that there is a disagreement over character encoding or something, and what is happening is that line drawing characters are being misinterpreted by sympathy. It might be a firmware bug or a grub bug or a sympathy bug or a hardware design issue. I think the next steps would be to determine exactly what sequence of bytes are being sent by the host, and to investigate its BIOS options for controlling serial terminal output.

Quoted 3 lines

HTH.

Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#74 lore raw

[linux-arm-xen test] 134708: regressions - all pass

flight 134708 linux-arm-xen real [real]
http://logs.test-lab.xenproject.org/osstest/logs/134708/

Regressions :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
test-arm64-arm64-examine 11 examine-serial/bootloader fail REGR. vs. 119195

Tests which did not succeed, but are not blocking:

 test-armhf-armhf-libvirt     14 saverestore-support-check    fail  like 119195
 test-arm64-arm64-xl-xsm      13 migrate-support-check        fail   never pass
 test-arm64-arm64-xl-xsm      14 saverestore-support-check    fail   never pass
 test-arm64-arm64-libvirt     13 migrate-support-check        fail   never pass
 test-arm64-arm64-libvirt     14 saverestore-support-check    fail   never pass
 test-arm64-arm64-xl-credit1  13 migrate-support-check        fail   never pass
 test-arm64-arm64-xl-credit1  14 saverestore-support-check    fail   never pass
 test-arm64-arm64-xl-credit2  13 migrate-support-check        fail   never pass
 test-arm64-arm64-xl-credit2  14 saverestore-support-check    fail   never pass
 test-arm64-arm64-xl-rtds     13 migrate-support-check        fail   never pass
 test-arm64-arm64-xl-rtds     14 saverestore-support-check    fail   never pass
 test-arm64-arm64-libvirt-xsm 13 migrate-support-check        fail   never pass
 test-arm64-arm64-libvirt-xsm 14 saverestore-support-check    fail   never pass
 test-arm64-arm64-xl          13 migrate-support-check        fail   never pass
 test-arm64-arm64-xl          14 saverestore-support-check    fail   never pass
 test-arm64-arm64-xl-multivcpu 13 migrate-support-check        fail  never pass
 test-arm64-arm64-xl-multivcpu 14 saverestore-support-check    fail  never pass
 test-arm64-arm64-libvirt-qcow2 12 migrate-support-check        fail never pass
 test-arm64-arm64-libvirt-qcow2 13 saverestore-support-check    fail never pass
 test-armhf-armhf-xl-rtds     13 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-rtds     14 saverestore-support-check    fail   never pass
 test-armhf-armhf-xl-credit2  13 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-credit2  14 saverestore-support-check    fail   never pass
 test-armhf-armhf-xl-cubietruck 13 migrate-support-check        fail never pass
 test-armhf-armhf-xl-cubietruck 14 saverestore-support-check    fail never pass
 test-armhf-armhf-xl          13 migrate-support-check        fail   never pass
 test-armhf-armhf-xl          14 saverestore-support-check    fail   never pass
 test-armhf-armhf-xl-multivcpu 13 migrate-support-check        fail  never pass
 test-armhf-armhf-xl-multivcpu 14 saverestore-support-check    fail  never pass
 test-armhf-armhf-xl-credit1  13 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-credit1  14 saverestore-support-check    fail   never pass
 test-armhf-armhf-libvirt     13 migrate-support-check        fail   never pass
 test-armhf-armhf-libvirt-raw 12 migrate-support-check        fail   never pass
 test-armhf-armhf-libvirt-raw 13 saverestore-support-check    fail   never pass
 test-armhf-armhf-xl-vhd      12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-vhd      13 saverestore-support-check    fail   never pass

version targeted for testing:
linux e64ac26749dc2c0f390caccd04274608ab31c8cf
baseline version:
linux 50484ee133c2219a26fd98b39770187221b5e0bc

Last test of basis   119195  2018-02-14 15:21:44 Z  425 days
Testing same since   134708  2019-04-12 16:43:22 Z    3 days    1 attempts

------------------------------------------------------------
2867 people touched revisions under test,
not listing them all

jobs:

 build-arm64-xsm                                              pass    
 build-arm64                                                  pass    
 build-armhf                                                  pass    
 build-arm64-libvirt                                          pass    
 build-armhf-libvirt                                          pass    
 build-arm64-pvops                                            pass    
 build-armhf-pvops                                            pass    
 test-arm64-arm64-xl                                          pass    
 test-armhf-armhf-xl                                          pass    
 test-arm64-arm64-libvirt-xsm                                 pass    
 test-arm64-arm64-xl-xsm                                      pass    
 test-arm64-arm64-xl-credit1                                  pass    
 test-armhf-armhf-xl-credit1                                  pass    
 test-arm64-arm64-xl-credit2                                  pass    
 test-armhf-armhf-xl-credit2                                  pass    
 test-armhf-armhf-xl-cubietruck                               pass    
 test-arm64-arm64-examine                                     pass    
 test-armhf-armhf-examine                                     pass    
 test-arm64-arm64-libvirt                                     pass    
 test-armhf-armhf-libvirt                                     pass    
 test-arm64-arm64-xl-multivcpu                                pass    
 test-armhf-armhf-xl-multivcpu                                pass    
 test-arm64-arm64-libvirt-qcow2                               pass    
 test-armhf-armhf-libvirt-raw                                 pass    
 test-arm64-arm64-xl-rtds                                     pass    
 test-armhf-armhf-xl-rtds                                     pass    
 test-armhf-armhf-xl-vhd                                      pass    

------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images

Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs

Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master

Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary

Not pushing.

(No revision log; it would be 260579 lines long.)

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#75 lore raw

Re: [linux-arm-xen test] 134708: regressions - all pass

Hi,

Quoted 8 lines · osstest service owner

flight 134708 linux-arm-xen real [real]
http://logs.test-lab.xenproject.org/osstest/logs/134708/

Regressions :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
test-arm64-arm64-examine 11 examine-serial/bootloader fail REGR. vs. 119195

I am a bit confused with the failure here [1]:

2019-04-13 22:36:55 Z ---------- substep 11 examine-serial/bootloader fail ---------- 

IIUC, we are looking for "osstest cookie". However this does not seem to appear on rochester0's logs. Any insights how "osstest cookie" will be output?

Quoted 93 lines

Tests which did not succeed, but are not blocking:

  test-armhf-armhf-libvirt     14 saverestore-support-check    fail  like 119195
  test-arm64-arm64-xl-xsm      13 migrate-support-check        fail   never pass
  test-arm64-arm64-xl-xsm      14 saverestore-support-check    fail   never pass
  test-arm64-arm64-libvirt     13 migrate-support-check        fail   never pass
  test-arm64-arm64-libvirt     14 saverestore-support-check    fail   never pass
  test-arm64-arm64-xl-credit1  13 migrate-support-check        fail   never pass
  test-arm64-arm64-xl-credit1  14 saverestore-support-check    fail   never pass
  test-arm64-arm64-xl-credit2  13 migrate-support-check        fail   never pass
  test-arm64-arm64-xl-credit2  14 saverestore-support-check    fail   never pass
  test-arm64-arm64-xl-rtds     13 migrate-support-check        fail   never pass
  test-arm64-arm64-xl-rtds     14 saverestore-support-check    fail   never pass
  test-arm64-arm64-libvirt-xsm 13 migrate-support-check        fail   never pass
  test-arm64-arm64-libvirt-xsm 14 saverestore-support-check    fail   never pass
  test-arm64-arm64-xl          13 migrate-support-check        fail   never pass
  test-arm64-arm64-xl          14 saverestore-support-check    fail   never pass
  test-arm64-arm64-xl-multivcpu 13 migrate-support-check        fail  never pass
  test-arm64-arm64-xl-multivcpu 14 saverestore-support-check    fail  never pass
  test-arm64-arm64-libvirt-qcow2 12 migrate-support-check        fail never pass
  test-arm64-arm64-libvirt-qcow2 13 saverestore-support-check    fail never pass
  test-armhf-armhf-xl-rtds     13 migrate-support-check        fail   never pass
  test-armhf-armhf-xl-rtds     14 saverestore-support-check    fail   never pass
  test-armhf-armhf-xl-credit2  13 migrate-support-check        fail   never pass
  test-armhf-armhf-xl-credit2  14 saverestore-support-check    fail   never pass
  test-armhf-armhf-xl-cubietruck 13 migrate-support-check        fail never pass
  test-armhf-armhf-xl-cubietruck 14 saverestore-support-check    fail never pass
  test-armhf-armhf-xl          13 migrate-support-check        fail   never pass
  test-armhf-armhf-xl          14 saverestore-support-check    fail   never pass
  test-armhf-armhf-xl-multivcpu 13 migrate-support-check        fail  never pass
  test-armhf-armhf-xl-multivcpu 14 saverestore-support-check    fail  never pass
  test-armhf-armhf-xl-credit1  13 migrate-support-check        fail   never pass
  test-armhf-armhf-xl-credit1  14 saverestore-support-check    fail   never pass
  test-armhf-armhf-libvirt     13 migrate-support-check        fail   never pass
  test-armhf-armhf-libvirt-raw 12 migrate-support-check        fail   never pass
  test-armhf-armhf-libvirt-raw 13 saverestore-support-check    fail   never pass
  test-armhf-armhf-xl-vhd      12 migrate-support-check        fail   never pass
  test-armhf-armhf-xl-vhd      13 saverestore-support-check    fail   never pass

version targeted for testing:
linux e64ac26749dc2c0f390caccd04274608ab31c8cf
baseline version:
linux 50484ee133c2219a26fd98b39770187221b5e0bc

Last test of basis   119195  2018-02-14 15:21:44 Z  425 days
Testing same since   134708  2019-04-12 16:43:22 Z    3 days    1 attempts

------------------------------------------------------------
2867 people touched revisions under test,
not listing them all

jobs:

  build-arm64-xsm                                              pass
  build-arm64                                                  pass
  build-armhf                                                  pass
  build-arm64-libvirt                                          pass
  build-armhf-libvirt                                          pass
  build-arm64-pvops                                            pass
  build-armhf-pvops                                            pass
  test-arm64-arm64-xl                                          pass
  test-armhf-armhf-xl                                          pass
  test-arm64-arm64-libvirt-xsm                                 pass
  test-arm64-arm64-xl-xsm                                      pass
  test-arm64-arm64-xl-credit1                                  pass
  test-armhf-armhf-xl-credit1                                  pass
  test-arm64-arm64-xl-credit2                                  pass
  test-armhf-armhf-xl-credit2                                  pass
  test-armhf-armhf-xl-cubietruck                               pass
  test-arm64-arm64-examine                                     pass
  test-armhf-armhf-examine                                     pass
  test-arm64-arm64-libvirt                                     pass
  test-armhf-armhf-libvirt                                     pass
  test-arm64-arm64-xl-multivcpu                                pass
  test-armhf-armhf-xl-multivcpu                                pass
  test-arm64-arm64-libvirt-qcow2                               pass
  test-armhf-armhf-libvirt-raw                                 pass
  test-arm64-arm64-xl-rtds                                     pass
  test-armhf-armhf-xl-rtds                                     pass
  test-armhf-armhf-xl-vhd                                      pass

------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images

Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs

Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master

Quoted 7 lines

Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary

Not pushing.

(No revision log; it would be 260579 lines long.)

Cheers,

[1] http://logs.test-lab.xenproject.org/osstest/logs/134708/test-arm64-arm64-examine/10.ts-examine-serial-post.log

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel