Skip to contentSkip to conversation

xen.git#staging smoke tests

PATCH Superseded Ian Jackson 2 participants 31 messages Sep 18, 2015 – Sep 21, 2015

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 xen.git#staging smoke tests

Thanks for the reviews.

My play cr-daily-branch run revealed that:

1. The cr-* branch plumbing was wrong. So I have redone that patch and dropped your ack.

2. test-amd64-amd64-xl-qemuu-debianhvm-i386 was broken due to problems mostly with ts-debian-hvm-install. This results in 9 new patches. (In principle we could deploy the smoke tests without fixing this test but this is probably not desirable.)

Patches v2 00-16 are in pretest now and not included in my series numbering here.

Patches v3 01-14 inclusive (v2 17-30) have been acked and are unchanged. So I have removed them from this patchbomb.

*   15   cr*: Support xen-unstable-smoke
 a  16   ap-*: Be able to fetch and push xen.git#smoke
 a  17   crontab: Enable xen-unstable-smoke branch every 3h at <1mod3>:51 UTC
+   18   ts-debian-hvm-install, etc.: Do not hardcode in-iso path
+   19   ts-debian-hvm-install: Cope with images containing only isolinux
+   20   ts-debian-hvm-install: Set $gsuite after $gho
+   21   ts-debian-hvm-install: Do not create EFI partition if EFI not in use
+   22   ts-debian-hvm-install: Use /dev/sda for i386, not /dev/xvda
+   23   Timeouts: Introduce target_adjust_timeout
+   24   Timeouts: Honour guest-related timeout-adjustment runvars
+   25   Debian i386 HVM tests: Increase installation timeout
+   26   cri-common: Add a missing semicolon

Ian.

#1 lore raw

PATCH 15/26 cr*: Support xen-unstable-smoke

Add this branch to select_xenbranch. This works like xen-unstable in most respects.

There are only two places in osstest where xenbranch `xen-unstable' is treated specially and only one of them needs adjusting to match xen-unstable-smoke too.

The new branch xen-unstable-smoke' has a prev' branch of `xen-unstable' according to cri-getprevxenbranch, which is technically wrong, but this is not important because xen-unstable-smoke has no prev tests.

We are going to sort out the push gate ref plumbing in xen.git in the next osstest patch.

Also, use a branch-settings file to set the new branch's resource priority to -20 to make it run ahead of anything else automatic.

---
v3: Dropped ack from Ian Campbell
Set qemuubranch to `qemu-xen-unstable'
Set xenbranch to xen-unstable-smoke', not xen-unstable'
cr-daily-branch: Update $realtree case for selecting qemuu revision

---
 README.planner                     |    1 +
 branch-settings.xen-unstable-smoke |    1 +
 cr-daily-branch                    |    2 +-
 cri-common                         |    1 +
 4 files changed, 4 insertions(+), 1 deletion(-)
 create mode 100644 branch-settings.xen-unstable-smoke
README.planner +1 -0
index dfa623e..3a491a4 100644
@@ -87,6 +87,7 @@ in the more distant future.
 Values for OSSTEST_RESOURCE_PRIORITY:
   mg-allocate                                    -1000000
   mg-blockage                                    -2000000
+  xen-unstable-smoke cr-daily-branch                  -20
   default Executive.pm with controlling terminal      -10
   mg-execute-flight                                    -8
   default                                               0
branch-settings.xen-unstable-smoke new file +1 -0
new file mode 100644
index 0000000..b73bc6a
@@ -0,0 +1 @@
+export OSSTEST_RESOURCE_PRIORITY=-20
cr-daily-branch +1 -1
index 013aeec..98fafdc 100755
@@ -340,7 +340,7 @@ case "$NEW_REVISION/$OLD_REVISION" in
                         "$treeurl#$OLD_REVISION-$NEW_REVISION" \
 
                 case "$realtree" in
-                xen-4*|xen-unstable)
+                xen-4*|xen-unstable*)
                     oldqemu=`./ap-qemu-revision $realtree $OLD_REVISION`
                     newqemu=`./ap-qemu-revision $realtree $NEW_REVISION`
                     if [ "$oldqemu" ] && [ "$newqemu" ]; then
cri-common +1 -0
index 2669485..45545d8 100644
@@ -66,6 +66,7 @@ select_prevxenbranch () {
 
 select_xenbranch () {
 	case "$branch" in
+	xen-unstable-smoke)	tree=xen;	xenbranch=$branch; qemuubranch=qemu-xen-unstable;;
 	xen-*)			tree=xen;	xenbranch=$branch ;;
 	qemu-mainline)		tree=qemuu;	xenbranch=xen-unstable	qemuubranch=qemu-mainline;;
         qemu-upstream-*)    tree=qemuu; xenbranch=xen-${branch#qemu-upstream-};;
Signature · 1 line
1.7.10.4
Acked-by on v1 15/26
#2 lore raw

Re: [OSSTEST PATCH 15/26] cr*: Support xen-unstable-smoke

Quoted 6 lines · Ian Jackson

Add this branch to select_xenbranch. This works like xen-unstable in most respects.

There are only two places in osstest where xenbranch `xen-unstable' is treated specially and only one of them needs adjusting to match xen-unstable-smoke too.

The included one is the one which adds the qemu revisions?

And the other one is, I think, the thing which sets
REVISION_LINUX_OLD=disable which we want (or maybe don't care about) for
the smoked tests. So good.

Quoted 5 lines

The new branch xen-unstable-smoke' has a prev' branch of `xen-unstable' according to cri-getprevxenbranch, which is technically wrong, but this is not important because xen-unstable-smoke has no prev tests.

Quoted 7 lines

We are going to sort out the push gate ref plumbing in xen.git in the next osstest patch.

Also, use a branch-settings file to set the new branch's resource priority to -20 to make it run ahead of anything else automatic.

#3 lore raw

Re: [OSSTEST PATCH 15/26] cr*: Support xen-unstable-smoke

Ian Campbell writes ("Re: [OSSTEST PATCH 15/26] cr*: Support xen-unstable-smoke"):

Quoted 9 lines · Ian Jackson
Quoted 6 lines

Add this branch to select_xenbranch. This works like xen-unstable in most respects.

There are only two places in osstest where xenbranch `xen-unstable' is treated specially and only one of them needs adjusting to match xen-unstable-smoke too.

The included one is the one which adds the qemu revisions?

No, this in cr-daily-branch:

    if [ "x$branch" != "xxen-unstable" ]; then
            export REVISION_LINUX_OLD=disable
    fi
Quoted 3 lines

And the other one is, I think, the thing which sets
REVISION_LINUX_OLD=disable which we want (or maybe don't care about) for
the smoked tests. So good.

Indeed.

I'm not sure what the qemu revisions thing is you're thinking of but I think all the places where the source code matches xen-unstable[^-]', the actual effect is to treat xen-unstable-smoke' the same way as `xen-unstable'.

Quoted 1 line

Thanks,
Ian.

#4 lore raw

Re: [OSSTEST PATCH 15/26] cr*: Support xen-unstable-smoke

Quoted 11 lines · Ian Jackson

Ian Campbell writes ("Re: [OSSTEST PATCH 15/26] cr*: Support xen-unstable -smoke"):

Quoted 9 lines · Ian Jackson
Quoted 7 lines

Add this branch to select_xenbranch. This works like xen-unstable in most respects.

There are only two places in osstest where xenbranch `xen-unstable' is
treated specially and only one of them needs adjusting to match xen-unstable-smoke too.

Quoted 7 lines
Quoted 1 line

The included one is the one which adds the qemu revisions?

No, this in cr-daily-branch:

    if [ "x$branch" != "xxen-unstable" ]; then
            export REVISION_LINUX_OLD=disable
    fi

That's the one which didn't need adjusting I think.

Quoted 11 lines
Quoted 4 lines

And the other one is, I think, the thing which sets
REVISION_LINUX_OLD=disable which we want (or maybe don't care about)
for
the smoked tests. So good.

Indeed.

I'm not sure what the qemu revisions thing is you're thinking of but I think all the places where the source code matches xen-unstable[^-]', the actual effect is to treat xen-unstable-smoke' the same way as `xen-unstable'.

"qemu revisions thing" is:

@@ -340,7 +340,7 @@ case "$NEW_REVISION/$OLD_REVISION" in
"$treeurl#$OLD_REVISION-$NEW_REVISION" \

case "$realtree" in

-                xen-4*|xen-unstable)
+                xen-4*|xen-unstable*)

oldqemu=./ap-qemu-revision $realtree $OLD_REVISION newqemu=./ap-qemu-revision $realtree $NEW_REVISION if [ "$oldqemu" ] && [ "$newqemu" ]; then

Quoted 5 lines
Quoted 2 lines

Thanks,
Ian.

#5 lore raw

Re: [OSSTEST PATCH 15/26] cr*: Support xen-unstable-smoke

Ian Campbell writes ("Re: [OSSTEST PATCH 15/26] cr*: Support xen-unstable-smoke"):

Quoted 6 lines · Ian Jackson
Quoted 3 lines
    if [ "x$branch" != "xxen-unstable" ]; then
            export REVISION_LINUX_OLD=disable
    fi

That's the one which didn't need adjusting I think.

Yes.

Quoted 8 lines
Quoted 4 lines

I'm not sure what the qemu revisions thing is you're thinking of but I think all the places where the source code matches xen-unstable[^-]', the actual effect is to treat xen-unstable-smoke' the same way as `xen-unstable'.

"qemu revisions thing" is:

cr-daily-branch case "$NEW_REVISION/$OLD_REVISION" in · line 346

Quoted 8 lines
cr-daily-branch +1 -1
@@ -340,7 +340,7 @@ case "$NEW_REVISION/$OLD_REVISION" in
                         "$treeurl#$OLD_REVISION-$NEW_REVISION" \
 
                 case "$realtree" in
-                xen-4*|xen-unstable)
+                xen-4*|xen-unstable*)
                     oldqemu=`./ap-qemu-revision $realtree $OLD_REVISION`
                     newqemu=`./ap-qemu-revision $realtree $NEW_REVISION`
                     if [ "$oldqemu" ] && [ "$newqemu" ]; then

Ah, OK. Yes. We agree.

Thanks,
Ian.

#6 lore raw

PATCH 16/26 ap-*: Be able to fetch and push xen.git#smoke

In reply to Ian Jackson

The branches and push gates are now:
xen.git#staging -[xen-unstable-smoke]-> #smoke -[xen-unstable]-> #master

Deployment note: When this passes the osstest self-push-gate, the main xen-unstable flight will start using smoke as an input. Therefore, until the new cronjob is installed to run the xen-unstable-smoke tests, an automatic process should keep xen.git#smoke up to date with xen.git#staging. Eg, running in screen in xen@xenbits:~/git/xen.git:
while sleep 1800; do git fetch . staging:smoke; done

---
v2: xen.git#smoked' branch name changed to #smoke'

---
 ap-fetch-version     |    3 +++
 ap-fetch-version-old |    3 +++
 ap-push              |    4 ++++
 3 files changed, 10 insertions(+)
ap-fetch-version +3 -0
index 488ea55..62adf79 100755
@@ -40,6 +40,9 @@ xen-3.*)
 	' x
 	;;
 xen-unstable)
+	repo_tree_rev_fetch_git xen $TREE_XEN smoke $LOCALREV_XEN
+	;;
+xen-unstable-smoke)
 	repo_tree_rev_fetch_git xen $TREE_XEN staging $LOCALREV_XEN
 	;;
 xen-4.*-testing)
ap-fetch-version-old +3 -0
index 7b918e3..716fc8f 100755
@@ -50,6 +50,9 @@ xen-3.*)
 xen-unstable)
 	repo_tree_rev_fetch_git xen $TREE_XEN master $LOCALREV_XEN
 	;;
+xen-unstable-smoke)
+	repo_tree_rev_fetch_git xen $TREE_XEN smoke $LOCALREV_XEN
+	;;
 xen-4.*-testing)
 	branchcore=${branch#xen-}
 	branchcore=${branchcore%-testing}
ap-push +4 -0
index 1dd5b05..aa0ec3d 100755
@@ -54,6 +54,10 @@ xen-unstable)
 	cd $repos/xen
 	git push $TREE_XEN $revision:refs/heads/master
 	;;
+xen-unstable-smoke)
+	cd $repos/xen
+	git push $TREE_XEN $revision:refs/heads/smoke
+	;;
 xen-*-testing)
 	cd $repos/xen
 	xenversion=$branch
Signature · 1 line
1.7.10.4
#7 lore raw

PATCH 17/26 crontab: Enable xen-unstable-smoke branch every 3h at <1mod3>:51 UTC

In reply to Ian Jackson
---
 crontab |    1 +
 1 file changed, 1 insertion(+)
crontab +1 -0
index 436645b..b9d4ad1 100644
@@ -2,6 +2,7 @@ PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
 [email protected],[email protected]
 # m		h	dom mon dow	command
 49		1	* * *		cd testing.git && BRANCHES_ALWAYS=xen-unstable	./cr-for-branches branches -w "./cr-daily-branch --real"
+51		1-23/3	* * *		cd testing.git && BRANCHES=xen-unstable-smoke	./cr-for-branches branches -w "./cr-daily-branch --real"
 4-59/30		*	* * *		cd testing.git &&				./cr-for-branches branches -q "./cr-daily-branch --real"
 18		9	* * 1,3,5	cd testing.git && BRANCHES=linux-next		./cr-for-branches branches -w "./cr-daily-branch --real"
 18		4	* * *		cd testing.git && BRANCHES='linux-linus linux-mingo-tip-master linux-3.0 libvirt rumpuserxen' ./cr-for-branches branches -w "./cr-daily-branch --real"
Signature · 1 line
1.7.10.4
#8 lore raw

PATCH 18/26 ts-debian-hvm-install, etc.: Do not hardcode in-iso path

In reply to Ian Jackson

ts-debian-hvm-install hardcoded `install.amd' as the directory in the .iso in which to find the kernel and initrd. This is wrong for architectures other than amd64.

Instead, pass this information in runvars (as is done for the netinst installs in make-distros-flight), and honour it in ts-debian-hvm-install.

If the runvars are not set, default to the previous hardcoded values. (This arranges that clones of old flights still work with new osstest, eg for bisection.)

---
 make-flight           |    8 ++++++++
 ts-debian-hvm-install |   25 +++++++++++++++++++------
 2 files changed, 27 insertions(+), 6 deletions(-)
make-flight +8 -0
index c1f40b9..accf922 100755
@@ -264,6 +264,12 @@ do_hvm_debian_test_one () {
 
   local arch=$(branch_debianhvm_arch)
 
+  case "$arch" in
+    amd64) iso_dir='install.amd' ;;
+    i386)  iso_dir='install.386' ;;
+    *)     iso_dir="install.$arch" ;;
+  esac
+
   stubdom_suffix=""
   stubdom_runvar=""
   if [ x$stubdom = xtrue ]; then
@@ -276,6 +282,8 @@ do_hvm_debian_test_one () {
     enable_xsm=$xsm                             \
     $stubdom_runvar                             \
     debianhvm_image=debian-7.2.0-$arch-CD-1.iso \
+    debianhvm_iso_kernel=/$iso_dir/vmlinuz \
+    debianhvm_iso_ramdisk=/$iso_dir/initrd.gz \
     bios=$bios \
     all_hostflags=$most_hostflags,hvm
 }
ts-debian-hvm-install +19 -6
index 4b1eca4..3b93ebd 100755
@@ -48,6 +48,8 @@ our $disk_mb= 10000;
 our $guesthost= "$gn.guest.osstest";
 our $gho;
 
+our ($kernel, $ramdisk);
+
 our $gsuite= guest_var($gho,'suite',$c{GuestDebianSuite});
 
 sub preseed () {
@@ -130,14 +132,14 @@ set default="0"
 set timeout=5
 
 menuentry 'debian guest auto Install' {
-    linux /install.amd/vmlinuz $cmdline
-    initrd /install.amd/initrd.gz
+    linux $kernel $cmdline
+    initrd $ramdisk
 }
 END
 }
 
 sub isolinux_cfg () {
-    my $cmdline = gcmdline("initrd=/install.amd/initrd.gz");
+    my $cmdline = gcmdline("initrd=$ramdisk");
 
     return <<"END";
     default autoinstall
@@ -145,7 +147,7 @@ sub isolinux_cfg () {
     timeout 0
 
     label autoinstall
-        kernel /install.amd/vmlinuz
+        kernel $kernel
         append $cmdline
 END
 }
@@ -156,9 +158,9 @@ sub prepare_initrd ($$$) {
       rm -rf $initrddir
       mkdir $initrddir
       cd $initrddir
-      gzip -d < $newiso/install.amd/initrd.gz | cpio --extract --make-directories --no-absolute-filename
+      gzip -d < $newiso$ramdisk | cpio --extract --make-directories --no-absolute-filename
       cp $preseed_file_path preseed.cfg
-      find . | cpio -H newc --create | gzip -9 > $newiso/install.amd/initrd.gz
+      find . | cpio -H newc --create | gzip -9 > $newiso$ramdisk
       cd -
       rm -rf $initrddir
       cd $newiso
@@ -169,6 +171,13 @@ END
 
 our $emptyiso= "/root/$flight.$job.$gn-empty.iso";
 
+sub iso_path ($$) {
+    my ($which, $deftail) = @_;
+    my $v = guest_var($gho, "iso_$which", "/install.amd/$deftail");
+    die "$gho->{Guest} $which $v ?" unless $v =~ m#^/#;
+    return $v;
+}
+
 sub prep () {
     target_install_packages_norec($ho, qw(lvm2 rsync genisoimage));
 
@@ -177,6 +186,10 @@ sub prep () {
     $gho= prepareguest($ho, $gn, $guesthost, 22,
                        $disk_mb + 1,
                        200);
+
+    $kernel = iso_path('kernel', 'vmlinuz');
+    $ramdisk = iso_path('ramdisk', 'initrd.gz');
+
     my $base = "/root/$flight.$job.$gn-";
     my $newiso= $base . "newiso";
     my $emptydir= $base . "empty-dir";
Signature · 1 line
1.7.10.4
Acked-by on v1 18/26
#9 lore raw

Re: [OSSTEST PATCH 18/26] ts-debian-hvm-install, etc.: Do not hardcode in-iso path

Quoted 8 lines · Ian Jackson

ts-debian-hvm-install hardcoded `install.amd' as the directory in the .iso in which to find the kernel and initrd. This is wrong for architectures other than amd64.

Instead, pass this information in runvars (as is done for the netinst installs in make-distros-flight), and honour it in ts-debian-hvm-install.

Quoted 5 lines

If the runvars are not set, default to the previous hardcoded values. (This arranges that clones of old flights still work with new osstest, eg for bisection.)

#10 lore raw

PATCH 19/26 ts-debian-hvm-install: Cope with images containing only isolinux

In reply to Ian Jackson

debian-7.2.0-i386-CD-1.iso contains no grub, only isolinux.

If the specified EFI grub file does not exist, fall back to isolinux. This requires a -c option as well, according to
https://wiki.debian.org/DebianInstaller/Modify/CD

Only try to set up a grub config if we are booting grub. (The i386 image in question does not contain a [debian]/boot/grub directory.)

If boot/grub/efi.img _does_ exist (ie, for other existing tests), the only difference in behaviour is to reorder slightly the options to
genisoimage: -b boot/grub/efi.img' now occurs after -no-emul-boot -r' rather than before.

---
 ts-debian-hvm-install |   13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
ts-debian-hvm-install +11 -2
index 3b93ebd..71ab1a5 100755
@@ -197,9 +197,16 @@ sub prep () {
     my $preseed_file_path = $base . "preseed";
 
     my @isogen_extra = qw(-eltorito-alt-boot
-                          -b boot/grub/efi.img
                           -no-emul-boot
                           -r);
+
+    my $bootfile = 'boot/grub/efi.img';
+    if (!target_file_exists($ho, "$newiso/$bootfile")) {
+	$bootfile = "isolinux/isolinux.bin";
+	push @isogen_extra, qw(-c isolinux/boot.cat);
+    }
+    push @isogen_extra, '-b', $bootfile;
+
     my @isogen_opts = (iso_gen_flags_basic(), @isogen_extra);
 
     iso_create_empty($ho, $emptyiso, $emptydir);
@@ -226,8 +233,10 @@ sub prep () {
         my $cmds = iso_copy_content_from_image($gho, $newiso);
         $cmds .= prepare_initrd($initrddir,$newiso,$preseed_file_path);
         target_cmd_root($ho, $cmds, $isotimeout);
+
         target_putfilecontents_root_stash($ho, 10, grub_cfg(),
-                                          "$newiso/debian/boot/grub/grub.cfg");
+                                          "$newiso/debian/boot/grub/grub.cfg")
+	    if $bootfile =~ m/grub/;
 
         target_putfilecontents_root_stash($ho, 10, isolinux_cfg(),
                                           "$newiso/isolinux/isolinux.cfg");
Signature · 1 line
1.7.10.4
Acked-by on v1 19/26
#11 lore raw

Re: [OSSTEST PATCH 19/26] ts-debian-hvm-install: Cope with images containing only isolinux

Quoted 6 lines · Ian Jackson

debian-7.2.0-i386-CD-1.iso contains no grub, only isolinux.

If the specified EFI grub file does not exist, fall back to isolinux. This requires a -c option as well, according to
https://wiki.debian.org/DebianInstaller/Modify/CD

Quoted 7 lines

Only try to set up a grub config if we are booting grub. (The i386 image in question does not contain a [debian]/boot/grub directory.)

If boot/grub/efi.img _does_ exist (ie, for other existing tests), the only difference in behaviour is to reorder slightly the options to
genisoimage: -b boot/grub/efi.img' now occurs after -no-emul-boot -r' rather than before.

I'm happy to determine experimentally (i.e. by pushing to pretest) if there is any meaning to the order of these (the wiki has them the new way around, so I presume not).

Quoted 12 lines
---
 ts-debian-hvm-install |   13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
ts-debian-hvm-install +0 -0
index 3b93ebd..71ab1a5 100755
@@ -197,9 +197,16 @@ sub prep () {
     my $preseed_file_path = $base . "preseed";
 

ts-debian-hvm-install sub prep () { · line 207

Quoted 10 lines
ts-debian-hvm-install +6 -1
     my @isogen_extra = qw(-eltorito-alt-boot
-                          -b boot/grub/efi.img
                           -no-emul-boot
                           -r);
+
+    my $bootfile = 'boot/grub/efi.img';
+    if (!target_file_exists($ho, "$newiso/$bootfile")) {
+	$bootfile = "isolinux/isolinux.bin";
+	push @isogen_extra, qw(-c isolinux/boot.cat);
+    }

My preference would have been to produce an iso which was bootable either via EFI (grub) or legacy (isolinux), but that would require more complex command lines and I'm sure neither of us wants to figure out what those are. So:

Quoted 20 lines
ts-debian-hvm-install +5 -1
+    push @isogen_extra, '-b', $bootfile;
+
     my @isogen_opts = (iso_gen_flags_basic(), @isogen_extra);
 
     iso_create_empty($ho, $emptyiso, $emptydir);
@@ -226,8 +233,10 @@ sub prep () {
         my $cmds = iso_copy_content_from_image($gho, $newiso);
         $cmds .= prepare_initrd($initrddir,$newiso,$preseed_file_path);
         target_cmd_root($ho, $cmds, $isotimeout);
+
         target_putfilecontents_root_stash($ho, 10, grub_cfg(),
-                                         
 "$newiso/debian/boot/grub/grub.cfg");
+                                         
 "$newiso/debian/boot/grub/grub.cfg")
+	    if $bootfile =~ m/grub/;
 

target_putfilecontents_root_stash($ho, 10, isolinux_cfg(),

"$newiso/isolinux/isolinux.cfg");

#12 lore raw

Re: [OSSTEST PATCH 19/26] ts-debian-hvm-install: Cope with images containing only isolinux

Ian Campbell writes ("Re: [OSSTEST PATCH 19/26] ts-debian-hvm-install: Cope with images containing only isolinux"):

Quoted 2 lines · Ian Jackson
Quoted 1 line

debian-7.2.0-i386-CD-1.iso contains no grub, only isolinux.

...

Quoted 3 lines

I'm happy to determine experimentally (i.e. by pushing to pretest) if there is any meaning to the order of these (the wiki has them the new way around, so I presume not).

My tests (which I left running) have determined that this patch was wrong in the amd64 case: it would work, but by falling back to isolinux.

This is because $newiso is not populated at the time $bootfile is determined. I think this is fairly easily fixed by reordering things and I will send an update when I have got something that actually works in both cases.

(This affects the EFI partition creation too, but shouldn't involve any change to that patch except maybe to context.)

Quoted 10 lines

ts-debian-hvm-install sub prep () { · line 207

Quoted 5 lines
ts-debian-hvm-install +5 -0
+    my $bootfile = 'boot/grub/efi.img';
+    if (!target_file_exists($ho, "$newiso/$bootfile")) {
+	$bootfile = "isolinux/isolinux.bin";
+	push @isogen_extra, qw(-c isolinux/boot.cat);
+    }

My preference would have been to produce an iso which was bootable either via EFI (grub) or legacy (isolinux), but that would require more complex command lines and I'm sure neither of us wants to figure out what those are. So:

Yes.

Quoted 1 line

Thanks, but given the circmstances I am not going to apply this ack to whatever revised version I produce.

Ian.

#13 lore raw

Re: [OSSTEST PATCH 19/26] ts-debian-hvm-install: Cope with images containing only isolinux

Ian Jackson writes ("Re: [OSSTEST PATCH 19/26] ts-debian-hvm-install: Cope with images containing only isolinux"):

Quoted 5 lines

Ian Campbell writes ("Re: [OSSTEST PATCH 19/26] ts-debian-hvm-install: Cope with images containing only isolinux"):

Quoted 1 line

Thanks, but given the circmstances I am not going to apply this ack to whatever revised version I produce.

FYI I have reviewed the interdiff between v3 and what is going to be v4, and there are no changes to changed lines, only to context. But of course that means the changes are now in a different place, following the new code motion patch which will precede this.

v4 will follow soon I think.

Ian.

#14 lore raw

PATCH 20/26 ts-debian-hvm-install: Set $gsuite after $gho

In reply to Ian Jackson

$gsuite was set from guest_var, but before $gho was set, leading to an undefined value warning from Perl.

This would ignore any guest-specific suite runvars. AFAICT these are set by some of the jobs in make-distros-flight. I think the effect of this change is to apply workarounds for the intended suite, rather than for wheezy.

(Although there is another assignment to $gho later in ts-debian-hvm-install, for stage 2, the stage 2 code does some trivial TestSupport calls and does not need $gsuite. So there is no need to make arrangements to assign to $gsuite - or, for that matter, $kernel or $ramdisk, in that path.)

---
 ts-debian-hvm-install |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
ts-debian-hvm-install +2 -1
index 71ab1a5..fb07293 100755
@@ -50,7 +50,7 @@ our $gho;
 
 our ($kernel, $ramdisk);
 
-our $gsuite= guest_var($gho,'suite',$c{GuestDebianSuite});
+our $gsuite;
 
 sub preseed () {
 
@@ -187,6 +187,7 @@ sub prep () {
                        $disk_mb + 1,
                        200);
 
+    $gsuite = guest_var($gho,'suite',$c{GuestDebianSuite});
     $kernel = iso_path('kernel', 'vmlinuz');
     $ramdisk = iso_path('ramdisk', 'initrd.gz');
 
Signature · 1 line
1.7.10.4
Acked-by on v1 20/26
#15 lore raw

Re: [OSSTEST PATCH 20/26] ts-debian-hvm-install: Set $gsuite after $gho

Quoted 8 lines · Ian Jackson

$gsuite was set from guest_var, but before $gho was set, leading to an undefined value warning from Perl.

This would ignore any guest-specific suite runvars. AFAICT these are set by some of the jobs in make-distros-flight. I think the effect of this change is to apply workarounds for the intended suite, rather than for wheezy.

Quoted 7 lines

(Although there is another assignment to $gho later in ts-debian-hvm-install, for stage 2, the stage 2 code does some trivial TestSupport calls and does not need $gsuite. So there is no need to make arrangements to assign to $gsuite - or, for that matter, $kernel or $ramdisk, in that path.)

#16 lore raw

PATCH 21/26 ts-debian-hvm-install: Do not create EFI partition if EFI not in use

In reply to Ian Jackson

If we are booting our install ISO using a non-EFI executable, don't try to provide an EFI for the installed system either.

---
 ts-debian-hvm-install |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
ts-debian-hvm-install +5 -2
index fb07293..9bc6cc8 100755
@@ -49,6 +49,7 @@ our $guesthost= "$gn.guest.osstest";
 our $gho;
 
 our ($kernel, $ramdisk);
+our $bootfile;
 
 our $gsuite;
 
@@ -56,7 +57,7 @@ sub preseed () {
 
     my $preseed_file = preseed_base($gho,$gsuite,'','',());
 
-    $preseed_file .= (<<END);
+    $preseed_file .= (<<END.($bootfile =~ m/\.efi/ ? <<END : '').<<END);
 d-i netcfg/get_hostname string $gn
 
 d-i partman-auto/disk string /dev/xvda
@@ -64,12 +65,14 @@ d-i partman-auto/method string  regular
 
 d-i partman-auto/expert_recipe string \\
         boot-root :: \\
+END
                 512 50 512 vfat \\
                         \$primary{ } \$bootable{ } \\
                         method{ efi } format{ } \\
                         use_filesystem{ } filesystem{ vfat } \\
                         mountpoint{ /boot/efi } \\
                 . \\
+END
                 5000 50 5000 ext4 \\
                         method{ format } format{ } \\
                         use_filesystem{ } filesystem{ ext4 } \\
@@ -201,7 +204,7 @@ sub prep () {
                           -no-emul-boot
                           -r);
 
-    my $bootfile = 'boot/grub/efi.img';
+    $bootfile = 'boot/grub/efi.img';
     if (!target_file_exists($ho, "$newiso/$bootfile")) {
 	$bootfile = "isolinux/isolinux.bin";
 	push @isogen_extra, qw(-c isolinux/boot.cat);
Signature · 1 line
1.7.10.4
Acked-by on v1 21/26
#17 lore raw

Re: [OSSTEST PATCH 21/26] ts-debian-hvm-install: Do not create EFI partition if EFI not in use

Quoted 4 lines · Ian Jackson

If we are booting our install ISO using a non-EFI executable, don't try to provide an EFI for the installed system either.

FWIW the default recipes used by d-i use "$iflabel{ gpt }" to achieve the same thing. Actually I'm a bit surprised that this isn't the affect of "method { efi }" too.

Anyway, this approach is good enough for our purposes so:

Quoted 49 lines
---
 ts-debian-hvm-install |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
ts-debian-hvm-install +5 -2
index fb07293..9bc6cc8 100755
@@ -49,6 +49,7 @@ our $guesthost= "$gn.guest.osstest";
 our $gho;
 
 our ($kernel, $ramdisk);
+our $bootfile;
 
 our $gsuite;
 
@@ -56,7 +57,7 @@ sub preseed () {
 
     my $preseed_file = preseed_base($gho,$gsuite,'','',());
 
-    $preseed_file .= (<<END);
+    $preseed_file .= (<<END.($bootfile =~ m/\.efi/ ? <<END : '').<<END);
 d-i netcfg/get_hostname string $gn
 
 d-i partman-auto/disk string /dev/xvda
@@ -64,12 +65,14 @@ d-i partman-auto/method string  regular
 
 d-i partman-auto/expert_recipe string \\
         boot-root :: \\
+END
                 512 50 512 vfat \\
                         \$primary{ } \$bootable{ } \\
                         method{ efi } format{ } \\
                         use_filesystem{ } filesystem{ vfat } \\
                         mountpoint{ /boot/efi } \\
                 . \\
+END
                 5000 50 5000 ext4 \\
                         method{ format } format{ } \\
                         use_filesystem{ } filesystem{ ext4 } \\
@@ -201,7 +204,7 @@ sub prep () {
                           -no-emul-boot
                           -r);
 
-    my $bootfile = 'boot/grub/efi.img';
+    $bootfile = 'boot/grub/efi.img';
     if (!target_file_exists($ho, "$newiso/$bootfile")) {
 	$bootfile = "isolinux/isolinux.bin";
 	push @isogen_extra, qw(-c isolinux/boot.cat);
#18 lore raw

Re: [OSSTEST PATCH 21/26] ts-debian-hvm-install: Do not create EFI partition if EFI not in use

Ian Campbell writes ("Re: [OSSTEST PATCH 21/26] ts-debian-hvm-install: Do not create EFI partition if EFI not in use"):

Quoted 3 lines · Ian Jackson
Quoted 2 lines

If we are booting our install ISO using a non-EFI executable, don't try to provide an EFI for the installed system either.

...

Quoted 3 lines

FWIW the default recipes used by d-i use "$iflabel{ gpt }" to achieve the same thing. Actually I'm a bit surprised that this isn't the affect of "method { efi }" too.

I'm happy to do this some other way but the way I did it avoided me having to look up how d-i partman-auto works...

Quoted 3 lines

Anyway, this approach is good enough for our purposes so:

Thanks,
Ian.

#19 lore raw

Re: [OSSTEST PATCH 21/26] ts-debian-hvm-install: Do not create EFI partition if EFI not in use

In reply to Ian Campbell

Ian Campbell writes ("Re: [OSSTEST PATCH 21/26] ts-debian-hvm-install: Do not create EFI partition if EFI not in use"):

Quoted 9 lines · Ian Jackson
Quoted 4 lines

If we are booting our install ISO using a non-EFI executable, don't try to provide an EFI for the installed system either.

FWIW the default recipes used by d-i use "$iflabel{ gpt }" to achieve the same thing. Actually I'm a bit surprised that this isn't the affect of "method { efi }" too.

...

Quoted 2 lines

ts-debian-hvm-install sub preseed () { · line 60

Quoted 2 lines
ts-debian-hvm-install +1 -1
-    $preseed_file .= (<<END);
+    $preseed_file .= (<<END.($bootfile =~ m/\.efi/ ? <<END : '').<<END);

My amd64 tests point out that this regexp does not (of course) match the actual filename `boot/grub/efi.img'. I will fix this.

Ian.

#20 lore raw

PATCH 22/26 ts-debian-hvm-install: Use /dev/sda for i386, not /dev/xvda

In reply to Ian Jackson

Empirically, the device shows up as /dev/sda.

---
 make-flight           |    6 ++++--
 ts-debian-hvm-install |    4 +++-
 2 files changed, 7 insertions(+), 3 deletions(-)
make-flight +4 -2
index accf922..2552a53 100755
@@ -263,10 +263,12 @@ do_hvm_debian_test_one () {
   stubdom=$5 # '' (or unset) or 'true'
 
   local arch=$(branch_debianhvm_arch)
+  local testvars
 
   case "$arch" in
     amd64) iso_dir='install.amd' ;;
-    i386)  iso_dir='install.386' ;;
+    i386)  iso_dir='install.386'
+           testvars+=' debianhvm_diskdevice=/dev/sda' ;;
     *)     iso_dir="install.$arch" ;;
   esac
 
@@ -280,7 +282,7 @@ do_hvm_debian_test_one () {
   job_create_test test-$xenarch$kern-$dom0arch-$toolstack$qemuu_suffix$stubdom_suffix-$testname-$arch\
     test-debianhvm $toolstack $xenarch $dom0arch $qemuu_runvar \
     enable_xsm=$xsm                             \
-    $stubdom_runvar                             \
+    $stubdom_runvar $testvars                   \
     debianhvm_image=debian-7.2.0-$arch-CD-1.iso \
     debianhvm_iso_kernel=/$iso_dir/vmlinuz \
     debianhvm_iso_ramdisk=/$iso_dir/initrd.gz \
ts-debian-hvm-install +3 -1
index 9bc6cc8..91c7107 100755
@@ -57,10 +57,12 @@ sub preseed () {
 
     my $preseed_file = preseed_base($gho,$gsuite,'','',());
 
+    my $disk = guest_var($gho,'diskdevice','/dev/xvda');
+
     $preseed_file .= (<<END.($bootfile =~ m/\.efi/ ? <<END : '').<<END);
 d-i netcfg/get_hostname string $gn
 
-d-i partman-auto/disk string /dev/xvda
+d-i partman-auto/disk string $disk
 d-i partman-auto/method string  regular
 
 d-i partman-auto/expert_recipe string \\
Signature · 1 line
1.7.10.4
Acked-by on v1 22/26
#21 lore raw

Re: [OSSTEST PATCH 22/26] ts-debian-hvm-install: Use /dev/sda for i386, not /dev/xvda

Quoted 1 line · Ian Jackson

Empirically, the device shows up as /dev/sda.

This is because the plain i386 kernel used by the installer does not support Xen (which requires PAE) and this affects even the options needed for PVHVM. amd64 on the other hand doesn't have this restriction and PVHVM works with the installer kernel.

Because of this the i386 install ends up using the emulated sda device instead of switching to the pv devices as amd64 does.

I think it's probably worth explaining some of this in the commit log rather than "Empirically".

IIRC someone was working on enabling PVHVM even for non-PAE i386 kernels, not sure what the status of that is, but clearly it isn't going to change things for Wheezy.

If you wanted instead to enable PV support for these installs then the multi-arch (i386+amd64) isoimages contain the Xen enabled kernels for i386 (as /install.386/xen/*, n.b. /install.amd/xen/* are suitable symlinks, so you can aoid special cases.

e.g. this one:

http://cdimage.debian.org/cdimage/archive/7.9.0/multi-arch/iso-cd/

(sorry for forgetting about this when we spoke IRL last week).

Anyway, that's clearly an "if you can be bothered thing" so with an expanded commit message:

#22 lore raw

Re: [OSSTEST PATCH 22/26] ts-debian-hvm-install: Use /dev/sda for i386, not /dev/xvda

Ian Campbell writes ("Re: [OSSTEST PATCH 22/26] ts-debian-hvm-install: Use /dev/sda for i386, not /dev/xvda"):

Quoted 2 lines

Because of this the i386 install ends up using the emulated sda device instead of switching to the pv devices as amd64 does.

Indeed.

Quoted 2 lines

I think it's probably worth explaining some of this in the commit log rather than "Empirically".

Fair enough.

Quoted 8 lines

IIRC someone was working on enabling PVHVM even for non-PAE i386 kernels, not sure what the status of that is, but clearly it isn't going to change things for Wheezy.

If you wanted instead to enable PV support for these installs then the multi-arch (i386+amd64) isoimages contain the Xen enabled kernels for i386 (as /install.386/xen/*, n.b. /install.amd/xen/* are suitable symlinks, so you can aoid special cases.

I think testing both the emulated and PVHVM devices, in a single test job, is quite nice, actually. And it doesn't seem too slow. So I will keep it this way.

Quoted 3 lines

Anyway, that's clearly an "if you can be bothered thing" so with an expanded commit message:

Thanks.

Ian.

#23 lore raw

PATCH 23/26 Timeouts: Introduce target_adjust_timeout

In reply to Ian Jackson

This function is now called for almost every timeout.

Specifically, it is called in the cases in TestSupport where a guest- or host-related timeout is passed from code which has a $ho or $gho, to code which does not: all callers of poll_loop, and tcmd.

Currently the function is a no-op. Its existence and use will allow us to introduce various provocations for adjusting timeouts, of which I have one planned.

---
 Osstest/TestSupport.pm |   14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
Osstest/TestSupport.pm +13 -1
index 0afaeab..f1f900e 100644
@@ -310,6 +310,10 @@ END
     return $value;
 }
 
+sub target_adjust_timeout ($$) {
+    my ($ho,$timeoutref) = @_; # $ho might be a $gho
+}
+
 #---------- running commands eg on targets ----------
 
 sub cmd {
@@ -608,6 +612,7 @@ sub target_ping_check_up ($) { return target_ping_check_core(@_,0); }
 
 sub target_await_down ($$) {
     my ($ho,$timeout) = @_;
+    target_adjust_timeout($ho,\$timeout);
     poll_loop($timeout,5,'reboot-down', sub {
         return target_ping_check_down($ho);
     });
@@ -616,6 +621,7 @@ sub target_await_down ($$) {
 sub tcmd { # $tcmd will be put between '' but not escaped
     my ($stdout,$user,$ho,$tcmd,$timeout,$extrasshopts) = @_;
     $timeout=30 if !defined $timeout;
+    target_adjust_timeout($ho,\$timeout);
     tcmdex($timeout,$stdout,
            'ssh', sshopts(), @{ $extrasshopts || [] },
            sshuho($user,$ho), $tcmd);
@@ -1018,7 +1024,9 @@ sub common_toolstack ($) {
 sub host_reboot ($) {
     my ($ho) = @_;
     target_reboot($ho);
-    poll_loop(40,2, 'reboot-confirm-booted', sub {
+    my $timeout = 40;
+    target_adjust_timeout($ho,\$timeout);
+    poll_loop($timeout,2, 'reboot-confirm-booted', sub {
         my $output;
         if (!eval {
             $output= target_cmd_output($ho, <<END, 40);
@@ -1397,6 +1405,7 @@ sub report_once ($$$) {
 sub guest_await_state ($$$$$) {
     my ($ho,$gho, $what,$wait_st,$timeout) = @_;
 
+    target_adjust_timeout($gho,\$timeout);
     poll_loop($timeout, 30, "await $what request from $gho->{Guest}", sub {
         my $st= guest_get_state($ho,$gho);
         return undef if $st eq $wait_st;
@@ -1850,6 +1859,7 @@ sub guest_checkrunning ($$) {
 sub guest_await_dhcp_tcp ($$) {
     my ($gho,$timeout) = @_;
     guest_find_tcpcheckport($gho);
+    target_adjust_timeout($gho,\$timeout);
     poll_loop($timeout,1,
               "guest $gho->{Name} ".visible_undef($gho->{Ether}).
 	      " $gho->{TcpCheckPort}".
@@ -1925,6 +1935,7 @@ sub target_tcp_check ($$) {
 
 sub await_tcp ($$$) {
     my ($maxwait,$interval,$ho) = @_;
+    target_adjust_timeout($ho,\$maxwait);
     poll_loop($maxwait,$interval,
               "await tcp $ho->{Name} $ho->{TcpCheckPort}",
               sub {
@@ -2106,6 +2117,7 @@ sub await_webspace_fetch_byleaf ($$$$$) {
     my ($maxwait,$interval,$logtailer, $ho, $url) = @_;
     my $leaf= $url;
     $leaf =~ s,.*/,,;
+    target_adjust_timeout($ho,\$maxwait);
     poll_loop($maxwait,$interval, "fetch $leaf", sub {
         my ($line, $last);
         $last= '(none)';
Signature · 1 line
1.7.10.4
Acked-by on v1 23/26
#24 lore raw

Re: [OSSTEST PATCH 23/26] Timeouts: Introduce target_adjust_timeout

Quoted 11 lines · Ian Jackson

This function is now called for almost every timeout.

Specifically, it is called in the cases in TestSupport where a guest- or host-related timeout is passed from code which has a $ho or $gho, to code which does not: all callers of poll_loop, and tcmd.

Currently the function is a no-op. Its existence and use will allow us to introduce various provocations for adjusting timeouts, of which I have one planned.

#25 lore raw

PATCH 24/26 Timeouts: Honour guest-related timeout-adjustment runvars

In reply to Ian Jackson

We look up a guest runvar GUEST_CONTEXT_timeoutfactor (according to the usual rules for guest runvars).

Here CONTEXT can currently be general' or install'. (`install' applies when the guest is being installed.) If the runvar exists, all timeouts relating to that guest in that context are increased by the specified factor.

---
 Osstest/TestSupport.pm |    6 ++++++
 1 file changed, 6 insertions(+)
Osstest/TestSupport.pm +6 -0
index f1f900e..c18f4a6 100644
@@ -312,6 +312,10 @@ END
 
 sub target_adjust_timeout ($$) {
     my ($ho,$timeoutref) = @_; # $ho might be a $gho
+    if ($ho->{Guest}) {
+	my $context = $ho->{TimeoutContext} // 'general';
+	$$timeoutref *= guest_var($ho,"${context}_timeoutfactor",1);
+    }
 }
 
 #---------- running commands eg on targets ----------
@@ -1604,6 +1608,8 @@ sub prepareguest ($$$$$$) {
 	}
     }
 
+    $gho->{TimeoutContext} = 'install';
+
     guest_find_lv($gho);
     guest_find_diskimg($gho);
     guest_find_ether($gho);
Signature · 1 line
1.7.10.4
Acked-by on v1 24/26
#26 lore raw

Re: [OSSTEST PATCH 24/26] Timeouts: Honour guest-related timeout-adjustment runvars

Quoted 9 lines · Ian Jackson

We look up a guest runvar GUEST_CONTEXT_timeoutfactor (according to the usual rules for guest runvars).

Here CONTEXT can currently be general' or install'. (`install' applies when the guest is being installed.) If the runvar exists, all timeouts relating to that guest in that context are increased by the specified factor.

#27 lore raw

PATCH 25/26 Debian i386 HVM tests: Increase installation timeout

In reply to Ian Jackson

The Debian i386 image boots a 32-bit non-PAE kernel which therefore cannot have any PV drivers on our 64-bit hypervisors. This makes it a bit slow: in my test on a machine under my desk it took 1400s out of the allowed 2000s.

With this change the timeout is 3000s instead.

---
 make-flight |    1 +
 1 file changed, 1 insertion(+)
make-flight +1 -0
index 2552a53..70c909d 100755
@@ -268,6 +268,7 @@ do_hvm_debian_test_one () {
   case "$arch" in
     amd64) iso_dir='install.amd' ;;
     i386)  iso_dir='install.386'
+           testvars+=' debianhvm_install_timeoutfactor=1.5'
            testvars+=' debianhvm_diskdevice=/dev/sda' ;;
     *)     iso_dir="install.$arch" ;;
   esac
Signature · 1 line
1.7.10.4
Acked-by on v1 25/26
#28 lore raw

Re: [OSSTEST PATCH 25/26] Debian i386 HVM tests: Increase installation timeout

Quoted 8 lines · Ian Jackson

The Debian i386 image boots a 32-bit non-PAE kernel which therefore cannot have any PV drivers on our 64-bit hypervisors. This makes it a bit slow: in my test on a machine under my desk it took 1400s out of the allowed 2000s.

With this change the timeout is 3000s instead.

(although see comments earlier regarding how you could enable PV drivers here...)

#29 lore raw

PATCH 26/26 cri-common: Add a missing semicolon

In reply to Ian Jackson

This is not technically needed as bash interprets `a=1 b=2' as settings of both a and b. But it's not idiomatic sh within osstest to use this syntax.

No functional change.

---
 cri-common |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
cri-common +1 -1
index 45545d8..d73dc4b 100644
@@ -68,7 +68,7 @@ select_xenbranch () {
 	case "$branch" in
 	xen-unstable-smoke)	tree=xen;	xenbranch=$branch; qemuubranch=qemu-xen-unstable;;
 	xen-*)			tree=xen;	xenbranch=$branch ;;
-	qemu-mainline)		tree=qemuu;	xenbranch=xen-unstable	qemuubranch=qemu-mainline;;
+	qemu-mainline)		tree=qemuu;	xenbranch=xen-unstable; qemuubranch=qemu-mainline;;
         qemu-upstream-*)    tree=qemuu; xenbranch=xen-${branch#qemu-upstream-};;
 	linux)			tree=linux;	xenbranch=xen-unstable ;;
 	linux-*)		tree=linux;	xenbranch=xen-unstable ;;
Signature · 1 line
1.7.10.4