Skip to contentSkip to conversation

Introduction of netsted HVM test job

PATCH In review Robert Ho 3 participants 137 messages Aug 28, 2015 – Nov 4, 2015

lore
Share & tools
lore

Robert Ho

author
#0 lore raw

v12 cover Introduction of netsted HVM test job

This patch set adds nested HVM test case for osstest.
In this test case, a Xen hypervisor (L1) runs on top of another Xen hypervisor (L0).
Upon L1 hypervisor, we will then create a nested guest (L2), and test if the Linux guest can then be installed and run well. About nested Xen virtualization, refer to
http://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen.

Test steps
0. To run osstest in standalone mode, write a config file in '~/.xen-osstest/config', and then create a standalone.config file to define 'TREE_LINUX', 'REVISION_LINUX' which will be used for nested test. The directory path of 'Debian ISO Images' which used for installing HVM guest VM could be defined in '~/.xen-osstest/config'.
1. run './standalone-reset' to generate standalone.db firstly then run 'build-amd64' job and then 'build-amd64-pvops', to prepare xen installation tarball and hvm guest kernel.
2. run 'test-amd64-amd6-qemuu-nested' job, it does following:
a. invoke test step of 'ts-debain-hvm-install' to install
a normal HVM guest
b. invoke test step of 'ts-nested-setup' to make some
appropriate runvars which selecthost() would recognise and prepare the configurations for installing L2 guest VM.
c. invoke 'nested-layer-descend' to manipulate
need_xen_host stack.
d. invoke test step of 'ts-debain-hvm-install' again, but
take the L1 hypervisor as host, install the L2 guest on it
e. invoke test step of 'ts-guest-stop', stop L2 guest. f. invoke test step of 'ts-guest-destroy' to destroy L1 guest.

This patch set reuse 'ts-debian-hvm-install' for both L1 installation and L2 installation, use 'l1' as L1's guestname and identity and use 'l2' as L2's guestname.

This patch series has been tested on test machines of amd64 arch, Debian-7.2.0-amd64 as guests OS, with hvm domain0 of Linux kernel 4.1.2, in standalone mode.
Also, we use linux-stable tree as domain0 kernel source.

Changes since v11:
1. Some refactor like format and code style, and a correction on grub menu manipulation in v11.
2. Adopt Ian Jackson's patch in make job runner framework (tcl part) accommodated to nested cases.
3. After Ian's runner framework change, some further accomodation changes in other parts.
--------------------------------------------------

Ian Jackson (4):
Tcl: Provide lunappend
sg-run-job: Declare Tcl (for the benefit of Emacs)
sg-run-job: Break out per-host-prep and per-host-finish
sg-run-job: Provide infrastructure for layers of nesting

Robert Ho (17):
Optimize and re-format previous code of 'submenu' parsing Correct a mistake in setboot_grup2() of Xen entry parsing pattern Allow runvars to specify guest disk and ram size (turning
previous values into defaults)
Comment out CDROM entry in HVM guest VM
Honour $xopts{ExtraConfig} and use it to enable nestedhvm Add new function of 'host_install_postboot_complete' Replace 'start osstest-confirm-booted' code by function Add new script to customize nested test configuration Wrapper and use core_dump_setup() for nested host and normal host to
setup coredump sysctl
Add test job for nest test case
Integrate Ian J. code and small corrections.
Add PDU power method for nested L1 and L2 guest
Compose the main recipe of nested test job
After Xen install, configure its xenbr0 interface as dhcp. Selecthost uses dynamic IP address if the host is not configured
static IP.
Don't lvextend if actually no more space to extend await_tcp(): check_ip for each loop iteration

 Osstest/Debian.pm           | 12 ++++---
 Osstest/DhcpWatch/leases.pm |  2 +-
 Osstest/PDU/guest.pm        | 63 ++++++++++++++++++++++++++++++++++++
 Osstest/TestSupport.pm      | 59 ++++++++++++++++++++++++++-------
 make-flight                 | 29 +++++++++++++++++
 sg-run-job                  | 61 +++++++++++++++++++++++++++-------
 tcl/osstestlib.tcl          |  7 ++++
 ts-debian-hvm-install       | 27 +++++++++++++---
 ts-host-install             | 17 ++--------
 ts-nested-setup             | 79 +++++++++++++++++++++++++++++++++++++++++++++
 ts-xen-build-prep           |  3 +-
 ts-xen-install              |  5 +--
 12 files changed, 310 insertions(+), 54 deletions(-)
 create mode 100755 Osstest/PDU/guest.pm
 create mode 100755 ts-nested-setup
Signature · 1 line
1.8.3.1
author
#1 lore raw

PATCH v12 1/21 Optimize and re-format previous code of 'submenu' parsing

* space between ')' and '{'; and after '='
* omit unnecessary 'define' and '!defined' usage
* break long '{}' into several lines

---
 Osstest/Debian.pm | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
Osstest/Debian.pm +6 -4
index f0bcf06..c6b4720 100644
@@ -417,10 +417,10 @@ sub setupboot_grub2 ($$$$) {
             next if m/^\s*\#/ || !m/\S/;
             if (m/^\s*\}\s*$/) {
                 die unless $entry || $submenu;
-                if (!defined $entry && defined $submenu) {
+                if (!$entry && $submenu) {
                     logm("Met end of a submenu $submenu->{StartLine}..$.. ".
                         "Our want kern is $want_kernver");
-                    $submenu=undef;
+                    $submenu= undef;
                     pop @offsets;
                     $offsets[$#offsets]++;
                     next;
@@ -461,11 +461,13 @@ sub setupboot_grub2 ($$$$) {
             }
             if (m/^\s*menuentry\s+[\'\"](.*)[\'\"].*\{\s*$/) {
                 die $entry->{StartLine} if $entry;
-                $entry= { Title => $1, StartLine => $., MenuEntryPath => join ">", @offsets };
+                $entry= { Title => $1,
+			  StartLine => $.,
+			  MenuEntryPath => join ">", @offsets };
                 $offsets[$#offsets]++;
             }
             if (m/^\s*submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {
-                $submenu={ StartLine =>$., MenuEntryPath => join ">", @offsets };
+                $submenu= { StartLine =>$. };
                 push @offsets,(0);
             }
             if (m/^\s*chainloader\s*\/EFI\/osstest\/xen.efi/) {
Signature · 1 line
1.8.3.1
#2 lore raw

Re: [OSSTest Nested v12 01/21] Optimize and re-format previous code of 'submenu' parsing

Robert Ho writes ("[OSSTest Nested v12 01/21] Optimize and re-format previous code of 'submenu' parsing"):

Quoted 3 lines

* space between ')' and '{'; and after '='
* omit unnecessary 'define' and '!defined' usage
* break long '{}' into several lines

These changes are all good.

But }

Osstest/Debian.pm sub setupboot_grub2 ($$$$) { · line 470

Quoted 3 lines
             if (m/^\s*submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {
Osstest/Debian.pm +1 -1
-                $submenu={ StartLine =>$., MenuEntryPath => join ">", @offsets };
+                $submenu= { StartLine =>$. };

This drops the setting of MenuEntryPath from $submenu. This isn't mentioned in the commit message, and (without looking at the code to double-check) I'm not sure that it's right.

Thanks,
Ian.

#3 lore raw

Re: [OSSTest Nested v12 01/21] Optimize and re-format previous code of 'submenu' parsing

Quoted 12 lines · Ian Jackson

Robert Ho writes ("[OSSTest Nested v12 01/21] Optimize and re-format previous code of 'submenu' parsing"):

Quoted 3 lines

* space between ')' and '{'; and after '='
* omit unnecessary 'define' and '!defined' usage
* break long '{}' into several lines

These changes are all good.

But }

Quoted 2 lines
             if (m/^\s*submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {

- $submenu={ StartLine =>$., MenuEntryPath => join

Quoted 8 lines

">", @offsets };

Quoted 1 line

+ $submenu= { StartLine =>$. };

This drops the setting of MenuEntryPath from $submenu. This isn't mentioned in the commit message, and (without looking at the code to double-check) I'm not sure that it's right.

From memory MenuEntryPath is unused, but I left it there because when

debugging it was very convenient to use Data::Dumper to dump $submenu and/or $entry at each iteration and in that case having that information to hand can be useful.

I suppose one could argue that when debugging you can add this field back, but personally I'd prefer to just leave it as it's harmless.

Ian.

Quoted 3 lines

Thanks,
Ian.

#4 lore raw

Re: [OSSTest Nested v12 01/21] Optimize and re-format previous code of 'submenu' parsing

Robert has passed me a v13 of this series (via an emailed `git bundle'!) which I am now working on. I'm currently polishing an initial subset for a push to osstest pretest today. In that context:

Ian Campbell writes ("Re: [OSSTest Nested v12 01/21] Optimize and re-format previous code of 'submenu' parsing"):

Quoted 14 lines · Ian Jackson
Quoted 12 lines

These changes are all good.

But }

Quoted 2 lines
             if (m/^\s*submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {

- $submenu={ StartLine =>$., MenuEntryPath => join

">", @offsets };

Quoted 1 line

+ $submenu= { StartLine =>$. };

This drops the setting of MenuEntryPath from $submenu. This isn't mentioned in the commit message, and (without looking at the code to double-check) I'm not sure that it's right.

Quoted 7 lines
Quoted 1 line

From memory MenuEntryPath is unused, but I left it there because when

debugging it was very convenient to use Data::Dumper to dump $submenu and/or $entry at each iteration and in that case having that information to hand can be useful.

I suppose one could argue that when debugging you can add this field back, but personally I'd prefer to just leave it as it's harmless.

I have dropped this hunk.

Ian.

author
#5 lore raw

PATCH v12 2/21 Correct a mistake in setboot_grup2() of Xen entry parsing pattern

In reply to Robert Ho

In previous adding of 'submenu' parsing code, a mistake was made. Now restore the match pattern to original.

---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
index c6b4720..74a99a3 100644
@@ -475,7 +475,7 @@ sub setupboot_grub2 ($$$$) {
                 $entry->{Hv}= $1;
                 $entry->{Chainload} = 1;
             }
-            if (m/^\s*multiboot\s*(?:\/boot)?\/(xen\S+)/) {
+            if (m/^\s*multiboot\s*(?:\/boot)?\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
                 die unless $entry;
                 $entry->{Hv}= $1;
                 $entry->{Chainload} = 0;
Signature · 1 line
1.8.3.1
#6 lore raw

Re: [OSSTest Nested v12 02/21] Correct a mistake in setboot_grup2() of Xen entry parsing pattern

Robert Ho writes ("[OSSTest Nested v12 02/21] Correct a mistake in setboot_grup2() of Xen entry parsing pattern"):

Quoted 1 line

In previous adding of 'submenu' parsing code, a mistake was made.

It would be conventional to mention the exact final commit message as found in the mainline tree, which seems to be

b77a6a2522d8 "Changes to support '/boot' leading paths of kernel, xen, in grub"

It would also be conventional to CC the author of the original commit, in case they have a good reason for having done that. (I have done that now, even though I think it's clear that the change was accidental.)

Thanks,
Ian.

author
#7 lore raw

PATCH v12 3/21 Allow runvars to specify guest disk and ram size (turning previous values into defaults)

In reply to Robert Ho

1. The default disk size for guest is '10000M' which is not sufficient for nested HVM guest, using larger disk size for nested guest to accommodate to nested test requirement, the specific disk_size is defined by make-flight.
2. Also, also allow ram size to be defined by runvar. It takes precedence over the calculation formula.
3. Add comment at the interface code.

---
 Osstest/TestSupport.pm |  3 ++-
 ts-debian-hvm-install  | 14 ++++++++++----
 2 files changed, 12 insertions(+), 5 deletions(-)
Osstest/TestSupport.pm +2 -1
index 5bf0284..b7963af 100644
@@ -1726,7 +1726,8 @@ sub target_put_guest_image ($$;$) {
 
 sub more_prepareguest_hvm ($$$$;@) {
     my ($ho, $gho, $ram_mb, $disk_mb, %xopts) = @_;
-    
+    # $ram_mb and $disk_mb are defaults, used if runvars don't say 
+
     my $passwd= 'xenvnc';
 
     prepareguest_part_lvmdisk($ho, $gho, $disk_mb);
ts-debian-hvm-install +10 -4
index 864f038..f9bc5a5 100755
@@ -200,13 +200,19 @@ sub prep () {
     my $host_freemem_mb = host_get_free_memory($ho);
     my $ram_minslop = 100;
     my $ram_lots = 5000;
-    if ($host_freemem_mb > $ram_lots * 2 + $ram_minslop) {
-        $ram_mb = $ram_lots;
-    } else {
-        $ram_mb = 768;
+
+    $ram_mb = guest_var($gho,'memsize',undef);
+    if (!$ram_mb) {
+         if ($host_freemem_mb > $ram_lots * 2 + $ram_minslop) {
+              $ram_mb = $ram_lots;
+         } else {
+              $ram_mb = 768;
+         }
     }
     logm("Host has $host_freemem_mb MB free memory, setting guest memory size to $ram_mb MB");
 
+    $disk_mb= guest_var($gho,'disksize',$disk_mb);
+
     more_prepareguest_hvm($ho,$gho, $ram_mb, $disk_mb,
                           OnReboot => 'preserve',
                           Bios => $r{bios},
Signature · 1 line
1.8.3.1
Acked-by on v12 3/21
#8 lore raw

Re: [OSSTest Nested v12 03/21] Allow runvars to specify guest disk and ram size (turning previous values into defaults)

Robert Ho writes ("[OSSTest Nested v12 03/21] Allow runvars to specify guest disk and ram size (turning previous values into defaults)"):

Quoted 9 lines

1. The default disk size for guest is '10000M' which is not sufficient for nested HVM guest, using larger disk size for nested guest to accommodate to nested test requirement, the specific disk_size is defined by make-flight.
2. Also, also allow ram size to be defined by runvar. It takes precedence over the calculation formula.
3. Add comment at the interface code.

#9 lore raw

Re: [OSSTest Nested v12 03/21] Allow runvars to specify guest disk and ram size (turning previous values into defaults)

Ian Jackson writes ("Re: [OSSTest Nested v12 03/21] Allow runvars to specify guest disk and ram size (turning previous values into defaults)"):

Quoted 9 lines

Robert Ho writes ("[OSSTest Nested v12 03/21] Allow runvars to specify guest disk and ram size (turning previous values into defaults)"):

Quoted 8 lines

1. The default disk size for guest is '10000M' which is not sufficient for nested HVM guest, using larger disk size for nested guest to accommodate to nested test requirement, the specific disk_size is defined by make-flight.
2. Also, also allow ram size to be defined by runvar. It takes precedence over the calculation formula.
3. Add comment at the interface code.

Quoted 3 lines
Quoted 1 line

Robert, I see that the v13 of this series you passed me under the table does not include this ack.

It is conventional to transcribe an ack when preparing a new version of a series. That helps reviewers and other contributors see which parts of a series still need attention.

I will record the ack in my own git tree.

Thanks,
Ian.

author
#10 lore raw

Re: [OSSTest Nested v12 03/21] Allow runvars to specify guest disk and ram size (turning previous values into defaults)

Quoted 25 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Friday, September 25, 2015 6:35 PM
To: Hu, Robert <[email protected]>; [email protected]; [email protected]; [email protected]; Jin, Gordon <[email protected]>; Zheng, Di <[email protected]>
Subject: Re: [OSSTest Nested v12 03/21] Allow runvars to specify guest disk and ram size (turning previous values into defaults)

Ian Jackson writes ("Re: [OSSTest Nested v12 03/21] Allow runvars to specify
guest disk and ram size (turning previous values into defaults)"):

Quoted 1 line

Robert Ho writes ("[OSSTest Nested v12 03/21] Allow runvars to specify

guest disk and ram size (turning previous values into defaults)"):

Quoted 3 lines
Quoted 3 lines

1. The default disk size for guest is '10000M' which is not sufficient for nested HVM guest, using larger disk size for nested guest to accommodate to nested test requirement, the specific

disk_size is

Quoted 2 lines
Quoted 2 lines

defined by make-flight.
2. Also, also allow ram size to be defined by runvar. It takes

precedence

Quoted 5 lines
Quoted 4 lines

over the calculation formula.
3. Add comment at the interface code.

Quoted 9 lines
Quoted 1 line

Robert, I see that the v13 of this series you passed me under the table does not include this ack.

It is conventional to transcribe an ack when preparing a new version of a series. That helps reviewers and other contributors see which parts of a series still need attention.

Sure. I will remember this in following series.

Quoted 1 line

I will record the ack in my own git tree.

Thanks.

And sorry I haven't got a chance to read your replies/patches until now. So many test tasks almost crushed me.
I'm going to read your mails in the coming week.

Quoted 3 lines

Thanks,
Ian.

Acked-by Robert Ho
#11 lore raw

Re: [OSSTest Nested v12 03/21] Allow runvars to specify guest disk and ram size (turning previous values into defaults) [and 2 more messages]

Hu, Robert writes ("RE: [OSSTest Nested v12 03/21] Allow runvars to specify guest disk and ram size (turning previous values into defaults)"):

Quoted 2 lines

And sorry I haven't got a chance to read your replies/patches until now. So many test tasks almost crushed me.

That's fine, of course. We all have other things we are doing.

Quoted 1 line

I'm going to read your mails in the coming week.

Thanks. I see Ian Campbell has replied to several already.

Hu, Robert writes ("RE: [OSSTest Nested v12 16/21] Add PDU power method for nested L1 and L2 guest"):

Quoted 6 lines

Ian Jackson [mailto:[email protected]]:

Quoted 5 lines

Robert Ho writes ("[OSSTest Nested v12 16/21] Add PDU power method for nested L1 and L2 guest"):

Quoted 2 lines

For nested host/guest, its power on/off method shall be its host invoke $(toolstack)->create/destroy method.

Quoted 6 lines
Quoted 4 lines

Thanks for this patch, which I have substantially edited for my v14.

However, I notice that it was missing a signed-off-by. Can you please confirm that I should add your s-o-b ?

Sorry I forgot to add my s-o-b. Yes, please add mine.

Of course only one of us ought to be rebasing this series at once, because otherwise it will be difficult for us to merge our work.

In my mail
Subject: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing
Date: Fri, 25 Sep 2015 20:15:05 +0100
I handed the series back to you.

So I think it is you who need to add your own s-o-b to that patch.

(I don't imply any criticism of you with this detailed explanation. I just don't want any misunderstandings; nor do I want this minor work item to be dropped.)

Likewise:

Quoted 10 lines
Quoted 10 lines

From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>; Ian Jackson <[email protected]>
Subject: [OSSTEST PATCH 15/26] DhcpWatch::leases: Fix a reporting message

This talks about `guest_check_ip', but this code is now factored out into a method. Use the correct method name in reporting.

...

Quoted 1 line

Ack.

If you meant

then it would be better if you wrote that explicitly.

And of course as current custodian of the branch, it is up to you to record your own ack in it. This may seem a little odd, but it is the standard approach when dealing with a series containing contributions from multiple people.

I've read your other emails and Ian's replies and I think you should be unblocked now ? So I'm expecting more questions, bug reports, etc., and hopefully eventually a v15. Do not hesitate to ask again for help.

Regards,
Ian.

author
#12 lore raw

Re: [OSSTest Nested v12 03/21] Allow runvars to specify guest disk and ram size (turning previous values into defaults) [and 2 more messages]

Quoted 41 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Tuesday, October 13, 2015 6:41 PM
To: Hu, Robert <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Jin, Gordon <[email protected]>; Zheng, Di <[email protected]>; [email protected]
Subject: RE: [OSSTest Nested v12 03/21] Allow runvars to specify guest disk and ram size (turning previous values into defaults) [and 2 more messages]

Hu, Robert writes ("RE: [OSSTest Nested v12 03/21] Allow runvars to specify
guest disk and ram size (turning previous values into defaults)"):

Quoted 2 lines

And sorry I haven't got a chance to read your replies/patches until now. So many test tasks almost crushed me.

That's fine, of course. We all have other things we are doing.

Quoted 1 line

I'm going to read your mails in the coming week.

Thanks. I see Ian Campbell has replied to several already.

Hu, Robert writes ("RE: [OSSTest Nested v12 16/21] Add PDU power method for nested L1 and L2 guest"):

Quoted 2 lines

Ian Jackson [mailto:[email protected]]:

Quoted 1 line

Robert Ho writes ("[OSSTest Nested v12 16/21] Add PDU power method

for

Quoted 10 lines
Quoted 8 lines

nested L1 and L2 guest"):

Quoted 2 lines

For nested host/guest, its power on/off method shall be its host invoke $(toolstack)->create/destroy method.

Thanks for this patch, which I have substantially edited for my v14.

However, I notice that it was missing a signed-off-by. Can you please confirm that I should add your s-o-b ?

Sorry I forgot to add my s-o-b. Yes, please add mine.

Of course only one of us ought to be rebasing this series at once, because otherwise it will be difficult for us to merge our work.

Quoted 6 lines

In my mail
Subject: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing
Date: Fri, 25 Sep 2015 20:15:05 +0100
I handed the series back to you.

So I think it is you who need to add your own s-o-b to that patch.

Yes I planned to do so.

Quoted 18 lines

(I don't imply any criticism of you with this detailed explanation. I just don't want any misunderstandings; nor do I want this minor work item to be dropped.)

Likewise:

Quoted 5 lines
Quoted 5 lines

From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>;

Ian

Quoted 2 lines
Quoted 2 lines

Jackson <[email protected]>
Subject: [OSSTEST PATCH 15/26] DhcpWatch::leases: Fix a reporting

message

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

This talks about `guest_check_ip', but this code is now factored out into a method. Use the correct method name in reporting.

...

Quoted 1 line

Ack.

If you meant

then it would be better if you wrote that explicitly.

Get it.

Quoted 5 lines

And of course as current custodian of the branch, it is up to you to record your own ack in it. This may seem a little odd, but it is the standard approach when dealing with a series containing contributions from multiple people.

Yes, understand. I'm learning the upstream rules.

Quoted 6 lines

I've read your other emails and Ian's replies and I think you should be unblocked now ? So I'm expecting more questions, bug reports, etc., and hopefully eventually a v15. Do not hesitate to ask again for help.

Yeah, I will report issues I found on each specific patch you posted.

Quoted 4 lines

Regards,
Ian.

author
#13 lore raw

PATCH v12 4/21 Comment out CDROM entry in HVM guest VM

In reply to Robert Ho

Comment out CDROM entry in sources.list to make HTTP URL entry available for L1 HVM guest VM.

---
 ts-debian-hvm-install | 7 +++++++
 1 file changed, 7 insertions(+)
ts-debian-hvm-install +7 -0
index f9bc5a5..2ec4717 100755
@@ -100,6 +100,13 @@ if [ -e \$b/debian/grubx64.efi ] ; then
 fi
 END
 
+preseed_hook_command($gho, 'late_command', '', <<END);
+#!/bin/sh
+set -ex
+
+in-target sed -i 's/^deb *cdrom/#&/g' /etc/apt/sources.list
+END
+
     $preseed_file .= preseed_hook_cmds();
 
     return $preseed_file;
Signature · 1 line
1.8.3.1
#14 lore raw

Re: [OSSTest Nested v12 04/21] Comment out CDROM entry in HVM guest VM

Robert Ho writes ("[OSSTest Nested v12 04/21] Comment out CDROM entry in HVM guest VM"):

Quoted 2 lines

Comment out CDROM entry in sources.list to make HTTP URL entry available for L1 HVM guest VM.

This commit message should probably mention that the installation iso is of course not generally inserted in the virtual cdrom, when the guest is running.

Ian.

author
#15 lore raw

PATCH v12 5/21 Honour $xopts{ExtraConfig} and use it to enable nestedhvm

In reply to Robert Ho
---
 ts-debian-hvm-install | 5 +++++
 1 file changed, 5 insertions(+)
ts-debian-hvm-install +5 -0
index 2ec4717..e271af8 100755
@@ -202,6 +202,10 @@ sub prep () {
     target_putfilecontents_root_stash($ho, 10, preseed(),
                                       $preseed_file_path);
 
+    my $extra_config = '';
+    $extra_config .= "nestedhvm=1\n"
+        if guest_var($gho,"enable_nestedhvm",'false') =~ m/true/;
+
     # If host has >8G free memory, create a guest with 4G memory to catch
     # any error that triggers cross 4G boundary
     my $host_freemem_mb = host_get_free_memory($ho);
@@ -223,6 +227,7 @@ sub prep () {
     more_prepareguest_hvm($ho,$gho, $ram_mb, $disk_mb,
                           OnReboot => 'preserve',
                           Bios => $r{bios},
+                          ExtraConfig => $extra_config,
                           PostImageHook => sub {
         my $cmds = iso_copy_content_from_image($gho, $newiso);
         $cmds .= prepare_initrd($initrddir,$newiso,$preseed_file_path);
Signature · 1 line
1.8.3.1
#16 lore raw

Re: [OSSTest Nested v12 05/21] Honour $xopts{ExtraConfig} and use it to enable nestedhvm

Robert Ho writes ("[OSSTest Nested v12 05/21] Honour $xopts{ExtraConfig} and use it to enable nestedhvm"):

Your summary line says `Honour $xopts{ExtraConfig}' but I think this was actually done in 2011 in dc801db7 "Osstest: prepareguest: honour xopt ExtraConfig".

I think this patch honours the enable_nestedhvm guest setting, ie the runvar GUEST_enable_nestedhvm or enable_nestedhvm.

Having thought about this, I think it would be better if the guest var were called `nestedhvm_enable'. Sorry to ask for this change now, but could you change that please (and the corresponding later parts that set it, of course) ?

ts-debian-hvm-install sub prep () { · line 207

Quoted 3 lines
ts-debian-hvm-install +3 -0
+    my $extra_config = '';
+    $extra_config .= "nestedhvm=1\n"
+        if guest_var($gho,"enable_nestedhvm",'false') =~ m/true/;

This should use guest_var_boolean (which I think was probably introduced after you wrote this code).

Thanks,
Ian.

#17 lore raw

Re: [OSSTest Nested v12 05/21] Honour $xopts{ExtraConfig} and use it to enable nestedhvm

Ian Jackson writes ("Re: [OSSTest Nested v12 05/21] Honour $xopts{ExtraConfig} and use it to enable nestedhvm"):

Quoted 4 lines

Having thought about this, I think it would be better if the guest var were called `nestedhvm_enable'. Sorry to ask for this change now, but could you change that please (and the corresponding later parts that set it, of course) ?

You didn't do this. Which is good, because I have done a git grep' of enable_ vs _enable and I think I was wrong. There are many more runvars enable_xxx' than `xxx_enable'.

Quoted 6 lines

ts-debian-hvm-install sub prep () { · line 207

Quoted 3 lines
ts-debian-hvm-install +3 -0
+    my $extra_config = '';
+    $extra_config .= "nestedhvm=1\n"
+        if guest_var($gho,"enable_nestedhvm",'false') =~ m/true/;

This should use guest_var_boolean (which I think was probably introduced after you wrote this code).

I will do this.

Ian.

author
#18 lore raw

PATCH v12 6/21 Add new function of 'host_install_postboot_complete'

In reply to Robert Ho

This function is called to add 'osstest-confirm-booted' service in target's start up services.
Previously, this was dircetly done by
target_cmd_root($ho, "update-rc.d osstest-confirm-booted start 99 2 .")

Here wrapper it because more than one place (ts-host-install
and ts-nested-setup) will need this, so wrapper it to avoid open code.

---
 Osstest/TestSupport.pm | 6 ++++++
 1 file changed, 6 insertions(+)
Osstest/TestSupport.pm +6 -0
index b7963af..f2ec9be 100644
@@ -115,6 +115,7 @@ BEGIN {
                       iso_gen_flags_basic
                       iso_copy_content_from_image
                       guest_editconfig_nocd
+                      host_install_postboot_complete
                       );
     %EXPORT_TAGS = ( );
 
@@ -2373,4 +2374,9 @@ sub guest_editconfig_nocd ($$) {
     });
 }
 
+sub host_install_postboot_complete ($) {
+    my ($ho) = @_;
+    target_cmd_root($ho, "update-rc.d osstest-confirm-booted start 99 2 .");
+}
+
 1;
Signature · 1 line
1.8.3.1
#19 lore raw

Re: [OSSTest Nested v12 06/21] Add new function of 'host_install_postboot_complete'

Robert Ho writes ("[OSSTest Nested v12 06/21] Add new function of 'host_install_postboot_complete'"):

Quoted 8 lines

This function is called to add 'osstest-confirm-booted' service in target's start up services.
Previously, this was dircetly done by
target_cmd_root($ho, "update-rc.d osstest-confirm-booted start 99 2 .")

Here wrapper it because more than one place (ts-host-install
and ts-nested-setup) will need this, so wrapper it to avoid open code.

The commit message describes what ought to be done, ore or less. (Although wrapper' is wrong: the terminology is break out'; wrapping refers to the design pattern of making a thin wrapper function for another function.)

But the actual code introduces another copy. You should combine this patch with the next patch. Ie, have one patch which moves the `confirm-booted' invocation into a subroutine.

Thanks,
Ian.

author
#20 lore raw

PATCH v12 7/21 Replace 'start osstest-confirm-booted' code by function

In reply to Robert Ho

Schedul start osstest-confirm-booted script via call `host_install_postboot_complete' fucntion; in order to avoid open code.

---
 ts-host-install | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
ts-host-install +1 -1
index f734a9c..3f7f060 100755
@@ -165,7 +165,7 @@ END
     target_putfilecontents_root_stash($ho,10,$coredumps_conf,
 				'/etc/security/limits.d/coredumps.conf');
 
-    target_cmd_root($ho, "update-rc.d osstest-confirm-booted start 99 2 .");
+    host_install_postboot_complete($ho);
 
     logm('OK: install completed');
 }   
Signature · 1 line
1.8.3.1
#21 lore raw

Re: [OSSTest Nested v12 07/21] Replace 'start osstest-confirm-booted' code by function

Robert Ho writes ("[OSSTest Nested v12 07/21] Replace 'start osstest-confirm-booted' code by function"):

Quoted 3 lines

Schedul start osstest-confirm-booted script via call `host_install_postboot_complete' fucntion; in order to avoid open code.

See my comments on the previous patch. The code changes look fine, when taken together.

The commit messages will need combining.

It occurs to me that it would probably be easiest for me to rewrite your commit messages when we have a later version of your series, than to try to direct you how to edit them. Would that be welcome ?

Thanks,
Ian.

author
#22 lore raw

PATCH v12 8/21 Add new script to customize nested test configuration

In reply to Robert Ho

1. In this script, make some appropriate runvars which selecthost would recognise.
2. Prepare the configurations for installing L2 guest VM.
3. Create a lv disk (and create vg inside) in L0 and hot-attach it to L1. The VG which will later be used for installing L2 guest.

Changes since last version:
1. don't store l1 IP in runvar
2. create vg in the lv disk in l0; not bother to do this inside l1.

---
 ts-nested-setup | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 77 insertions(+)
 create mode 100755 ts-nested-setup
ts-nested-setup new file +77 -0
new file mode 100755
index 0000000..207ec85
@@ -0,0 +1,77 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Intel Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+# Pass L0's ident and L1's guestname
+our ($l0_ident,$l1_gn) = @ARGV;
+our ($l0,$l1) = ts_get_host_guest($l0_ident,$l1_gn);
+
+guest_check_ip($l1);
+
+# L1 guest's ident is same as guestname
+our $l1_ident = $l1->{Guest};
+
+store_runvar($l1_ident,$l1->{Guest});
+
+host_install_postboot_complete($l1);
+
+# Just for efficiency reason to install genisoimage here
+target_install_packages_norec($l1, qw(lvm2 rsync genisoimage));
+
+# We need to attach an extra disk to the L1 guest to be used as L2
+# guest storage.
+#
+# When running in a nested HVM environment the L1 domain is acting
+# as both a guest to L0 and a host to L2 guests and therefore potentially
+# sees connections to two independent xenstore instances, one provided by
+# the L0 host and one which is provided by the L1 instance of xenstore.
+#
+# Unfortunately the kernel is not capable of dealing with this and is only
+# able to cope with a single xenstore connection. Since the L1 toolstack and
+# L2 guests absolutely require xenstore to function we therefore cannot use
+# the L0 xenstore and therefore cannot use PV devices (xvdX etc) in the L1
+# guest and must use emulated devices (sdX etc).
+#
+# However at the moment we have not yet rebooted L1 into Xen and so it does
+# have PV devices available and sdb actually appears as xvdb. We could
+# disable the Xen platform device and use emulated devices for the install
+# phase too but that would be needlessly slow.
+# 
+# To avoid the confusion on xvdb and sdb, we here according to Ian J.'s
+# advice, create vg in l0.
+my $vgname = $l1->{Vg};
+my $guest_storage_lv_name = "${l1_ident}_guest_storage";
+my $guest_storage_lv_size = guest_var($l1,'guest_storage_size',undef);
+die "guest_storage_lv_size is undefined" unless $guest_storage_lv_size;
+my $guest_storage_lvdev = "/dev/${vgname}/${guest_storage_lv_name}";
+
+die "toolstack $r{toolstack}" unless $r{toolstack} eq "xl";
+target_cmd_root($l0, <<END);
+    vgremove -f ${l1_gn}-disk ||:
+    lvremove -f $guest_storage_lvdev ||:
+    lvcreate -L ${guest_storage_lv_size}M -n $guest_storage_lv_name $vgname
+    dd if=/dev/zero of=$guest_storage_lvdev count=10
+    pvcreate $guest_storage_lvdev
+    vgcreate ${l1_gn}-disk $guest_storage_lvdev
+    xl block-attach $l1->{Name} ${guest_storage_lvdev},raw,sdb,rw
+END
Signature · 1 line
1.8.3.1
author
#23 lore raw

PATCH v12 9/21 Wrapper and use core_dump_setup() for nested host and normal host to setup coredump sysctl

In reply to Robert Ho

This patch does these 4 things:
1. wrapper coredump setup code from original ts-host-install into TestSupport.pm
2. replace ts-host-install original code with this wrapper function
3. in debian-hvm-install, create '/var/core' in hvm host post installation.
4. in ts-nested-setup, call this function for l1 host.

---
 Osstest/TestSupport.pm | 19 +++++++++++++++++++
 ts-debian-hvm-install  |  1 +
 ts-host-install        | 15 +--------------
 ts-nested-setup        |  2 ++
 4 files changed, 23 insertions(+), 14 deletions(-)
Osstest/TestSupport.pm +19 -0
index f2ec9be..1768c5e 100644
@@ -116,6 +116,7 @@ BEGIN {
                       iso_copy_content_from_image
                       guest_editconfig_nocd
                       host_install_postboot_complete
+                      core_dump_setup
                       );
     %EXPORT_TAGS = ( );
 
@@ -2379,4 +2380,22 @@ sub host_install_postboot_complete ($) {
     target_cmd_root($ho, "update-rc.d osstest-confirm-booted start 99 2 .");
 }
 
+sub core_dump_setup ($) {
+    my ($ho) = @_;
+    target_cmd_root($ho, 'mkdir -p /var/core');
+    target_editfile_root($ho, '/etc/sysctl.conf',
+        sub { target_editfile_kvp_replace(
+                  "kernel.core_pattern",
+                  # %p==pid,%e==executable name,%t==timestamp
+                  "/var/core/%t.%p.%e.core") });
+    target_cmd_root($ho, "sysctl --load /etc/sysctl.conf");
+    my $coredumps_conf = <<'END';
+#<domain>      <type>  <item>       <value>
+*               soft    core         -1
+root            soft    core         -1
+END
+    target_putfilecontents_root_stash($ho,10,$coredumps_conf,
+                                    '/etc/security/limits.d/coredumps.conf');
+}
+
 1;
ts-debian-hvm-install +1 -0
index e271af8..d569272 100755
@@ -105,6 +105,7 @@ preseed_hook_command($gho, 'late_command', '', <<END);
 set -ex
 
 in-target sed -i 's/^deb *cdrom/#&/g' /etc/apt/sources.list
+in-target mkdir /var/core
 END
 
     $preseed_file .= preseed_hook_cmds();
ts-host-install +1 -14
index 3f7f060..77b8957 100755
@@ -150,20 +150,7 @@ END
 	});
     }
 
-    target_cmd_root($ho, 'mkdir -p /var/core');
-    target_editfile_root($ho, '/etc/sysctl.conf',
-	sub { target_editfile_kvp_replace(
-		  "kernel.core_pattern",
-		  # %p==pid,%e==executable name,%t==timestamp
-		  "/var/core/%t.%p.%e.core") });
-    target_cmd_root($ho, "sysctl --load /etc/sysctl.conf");
-    my $coredumps_conf = <<'END';
-#<domain>      <type>  <item>       <value>
-*               soft    core         -1
-root            soft    core         -1
-END
-    target_putfilecontents_root_stash($ho,10,$coredumps_conf,
-				'/etc/security/limits.d/coredumps.conf');
+    core_dump_setup($ho);
 
     host_install_postboot_complete($ho);
 
ts-nested-setup +2 -0
index 207ec85..2eff4ec 100755
@@ -35,6 +35,8 @@ store_runvar($l1_ident,$l1->{Guest});
 
 host_install_postboot_complete($l1);
 
+core_dump_setup($l1);
+
 # Just for efficiency reason to install genisoimage here
 target_install_packages_norec($l1, qw(lvm2 rsync genisoimage));
 
Signature · 1 line
1.8.3.1
#24 lore raw

Re: [OSSTest Nested v12 09/21] Wrapper and use core_dump_setup() for nested host and normal host to setup coredump sysctl

Robert Ho writes ("[OSSTest Nested v12 09/21] Wrapper and use core_dump_setup() for nested host and normal host to setup coredump sysctl"):

Quoted 3 lines

This patch does these 4 things:
1. wrapper coredump setup code from original ts-host-install into TestSupport.pm
2. replace ts-host-install original code with this wrapper function

You mean
break coredump setup code into new function `core_dump_setup'

Please break this part (the refactoring - ie points 1,2) out into its own patch. You can then say `no functional change' (assuming that's true).

Quoted 2 lines

3. in debian-hvm-install, create '/var/core' in hvm host post installation.
4. in ts-nested-setup, call this function for l1 host.

Is there some reason why the mkdir isn't done in core_dump_setup ? I think it is somewhat illogical that the /var/core should be (implicitly) used in ts-nested-setup but created in debian-hvm-install.

Also it should do `mkdir -p' in case the directory already exists somehow.

Thanks,
Ian.

#25 lore raw

Re: [OSSTest Nested v12 09/21] Wrapper and use core_dump_setup() for nested host and normal host to setup coredump sysctl

Quoted 15 lines · Ian Jackson

Robert Ho writes ("[OSSTest Nested v12 09/21] Wrapper and use core_dump_setup() for nested host and normal host to setup coredump sysctl"):

Quoted 4 lines

This patch does these 4 things:
1. wrapper coredump setup code from original ts-host-install into TestSupport.pm
2. replace ts-host-install original code with this wrapper function

You mean
break coredump setup code into new function `core_dump_setup'

Please break this part (the refactoring - ie points 1,2) out into its own patch. You can then say `no functional change' (assuming that's true).

Quoted 5 lines
Quoted 3 lines

3. in debian-hvm-install, create '/var/core' in hvm host post installation.
4. in ts-nested-setup, call this function for l1 host.

Is there some reason why the mkdir isn't done in core_dump_setup ?

It is, so it's unclear why it is also done in debian-hvm-install, I suspect that is left over from a previous version of the series which didn't do the refactoring. Probably it can just be dropped.

Quoted 2 lines

Also it should do `mkdir -p' in case the directory already exists somehow.

This is what the code which is refactored into core_dump_setup does already.

Ian.

#26 lore raw

Re: [OSSTest Nested v12 09/21] Wrapper and use core_dump_setup() for nested host and normal host to setup coredump sysctl

Ian Campbell writes ("Re: [OSSTest Nested v12 09/21] Wrapper and use core_dump_setup() for nested host and normal host to setup coredump sysctl"):

Quoted 6 lines · Ian Jackson
Quoted 2 lines

Also it should do `mkdir -p' in case the directory already exists somehow.

This is what the code which is refactored into core_dump_setup does already.

So it does!

Ian.

author
#27 lore raw

PATCH v12 10/21 Add test job for nest test case

In reply to Robert Ho

1. This patch adds creation of the nested test job, when job creation procedure is invoked.
2. Set nested L1's vif model, nestedhvm feature, set specific disk size and memory size for nested test by make-flight.

Changes since last version:
1. '\' right aligned
2. remove some unnecessary ''
3. no need to set l2 disk size

---
 make-flight | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)
make-flight +29 -0
index 00f797e..aa6c9b5 100755
@@ -220,6 +220,34 @@ do_hvm_win7_x64_tests () {
             all_hostflags=$most_hostflags,hvm
 }
 
+do_hvm_debian_nested_tests () {
+  bios=$1
+
+  if [ $xenarch != amd64 -o $dom0arch != amd64 \
+      -o "x$qemuu_suffix" != "x-qemuu" ]; then
+    return
+  fi
+
+  case $xenbranch in
+    xen-3.*-testing)      return;;
+    xen-4.0-testing)      return;;
+    xen-4.1-testing)      return;;
+    xen-4.2-testing)      return;;
+    xen-4.3-testing)      return;;
+  esac
+
+  job_create_test test-$xenarch$kern-$dom0arch$qemuu_suffix-nested \
+            test-nested xl $xenarch $dom0arch $qemuu_runvar        \
+            l1_image=debian-7.2.0-amd64-CD-1.iso                   \
+            l1_vifmodel='e1000'                                    \
+            l1_memsize='3072'                                      \
+            l1_enable_nestedhvm='true'                             \
+            l1_guest_storage_size='20000'                          \
+            l2_image=debian-7.2.0-amd64-CD-1.iso                   \
+            bios=$bios                                             \
+            all_hostflags=$most_hostflags,hvm
+}
+
 do_hvm_debian_test_one () {
   testname=$1
   toolstack=$2
@@ -522,6 +550,7 @@ test_matrix_do_one () {
     do_hvm_rhel6_tests
 
     do_hvm_debian_tests
+    do_hvm_debian_nested_tests seabios
 
   done # qemuu_suffix
 
Signature · 1 line
1.8.3.1
author
#28 lore raw

PATCH v12 11/21 Tcl: Provide lunappend

In reply to Robert Ho

From: Ian Jackson <[email protected]>

---
 tcl/osstestlib.tcl | 7 +++++++
 1 file changed, 7 insertions(+)
tcl/osstestlib.tcl +7 -0
index 61a6a09..b5a52d3 100644
@@ -75,6 +75,13 @@ proc lshift {listvar} {
     return $head
 }
 
+proc lunappend {listvar} {
+    upvar 1 $listvar list
+    set tail [lindex $list end]
+    set list [lrange $list 0 end-1]
+    return $tail
+}
+
 proc var-or-default {varname {default {}}} {
     upvar 1 $varname var
     if {[info exists var]} { return $var }
Signature · 1 line
1.8.3.1
author
#29 lore raw

PATCH v12 12/21 sg-run-job: Declare Tcl (for the benefit of Emacs)

In reply to Robert Ho

From: Ian Jackson <[email protected]>

Resolve conflicts:
sg-run-job

---
 sg-run-job | 1 +
 1 file changed, 1 insertion(+)
sg-run-job +1 -0
index e4ebc22..ec03cce 100755
@@ -1,4 +1,5 @@
 #!/usr/bin/tclsh
+# -*- Tcl -*-
 
 # This is part of "osstest", an automated testing framework for Xen.
 # Copyright (C) 2009-2013 Citrix Inc.
Signature · 1 line
1.8.3.1
author
#30 lore raw

PATCH v12 13/21 sg-run-job: Break out per-host-prep and per-host-finish

In reply to Robert Ho

From: Ian Jackson <[email protected]>

No functional change.

We now call the per-host-ts finish steps unconditionally, rather than only if !$need_build_host, per-host-ts is (complicated) no-op if $need_build_host, since in that case $need_xen_hosts is {}.

Tested by: Robert Ho <[email protected]>

---
 sg-run-job | 31 +++++++++++++++++++------------
 1 file changed, 19 insertions(+), 12 deletions(-)
sg-run-job +19 -12
index ec03cce..0614d20 100755
@@ -24,6 +24,20 @@ source ./tcl/osstestlib.tcl
 readconfig
 source-method JobDB
 
+proc per-host-prep {} {
+    per-host-ts .       host-ping-check-native/@ ts-host-ping-check
+    per-host-ts .       xen-install/@     ts-xen-install
+    per-host-ts .       xen-boot/@        ts-host-reboot
+
+    per-host-ts .       host-ping-check-xen/@ ts-host-ping-check
+    per-host-ts .       =(*)             {ts-leak-check basis}
+}
+
+proc per-host-finish {} {
+    per-host-ts .       =                {ts-leak-check check}
+    per-host-ts !broken capture-logs/@(*) ts-logs-capture
+}
+
 proc run-job {job} {
     global jobinfo builds flight ok need_xen_hosts anyfailed
 
@@ -52,22 +66,15 @@ proc run-job {job} {
     if {$ok} { setstatus running                                          }
 
     per-host-ts broken  host-install/@(*) ts-host-install-twice
-    per-host-ts .       host-ping-check-native/@ ts-host-ping-check
-    per-host-ts .       xen-install/@     ts-xen-install
-    per-host-ts .       xen-boot/@        ts-host-reboot
-    per-host-ts .       host-ping-check-xen/@ ts-host-ping-check
 
-    per-host-ts .       =(*)             {ts-leak-check basis}
+    per-hosts-prep
 
     if {$ok} { catching-otherwise fail      run-job/$jobinfo(recipe)      }
-    per-host-ts .       =                {ts-leak-check check}
 
-    if {!$need_build_host} {
-        per-host-ts !broken capture-logs/@(*) ts-logs-capture
-    } else {
-        if {$anyfailed} {
-            run-ts  !broken capture-logs      ts-logs-capture + host
-        }
+    per-host-finish
+
+    if {$need_build_host && $anyfailed} {
+	run-ts  !broken capture-logs      ts-logs-capture + host
     }
 
     if {$ok} { setstatus pass                                             }
Signature · 1 line
1.8.3.1
author
#31 lore raw

PATCH v12 14/21 sg-run-job: Provide infrastructure for layers of nesting

In reply to Robert Ho

From: Ian Jackson <[email protected]>

Provides nested-layer-descend, which can be called in an individual test job at the appropriate point (after the L1 has been set up).

The inner host is a guest of the outer host; powering it off means destroying it. Putting the poweroff at this point in the loop, rather than in per-host-finish, avoids powering off physical servers. The use of .' rather than !.' for iffail means we do not power off after failures (as we might want to preserve the state for debugging etc).

---
 sg-run-job | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
sg-run-job +20 -1
index 0614d20..7cb6cac 100755
@@ -40,6 +40,7 @@ proc per-host-finish {} {
 
 proc run-job {job} {
     global jobinfo builds flight ok need_xen_hosts anyfailed
+    global nested_layers_hosts
 
     set ok 1
     set anyfailed 0
@@ -53,6 +54,7 @@ proc run-job {job} {
         set need_xen_hosts $nh
         set need_build_host 0
     }
+    set nested_layers_hosts {}
 
     catching-otherwise blocked            check-not-blocked
     if {!$ok} return
@@ -71,7 +73,15 @@ proc run-job {job} {
 
     if {$ok} { catching-otherwise fail      run-job/$jobinfo(recipe)      }
 
-    per-host-finish
+    while 1 {
+	per-host-finish
+
+	if {![llength nested_layers_hosts]} break
+
+	per-host-ts . = final-poweroff {ts-host-powercycle --power=0}
+
+        set need_xen_hosts [lunappend nested_layers_hosts]
+    }
 
     if {$need_build_host && $anyfailed} {
 	run-ts  !broken capture-logs      ts-logs-capture + host
@@ -246,6 +256,15 @@ proc per-host-ts {iffail ident script args} {
     }
 }
 
+proc nested-layer-descend {nested_hosts} {
+    # We save need_xen_hosts on a stack in nested_layers_hosts
+    # It gets popped again during the cleanup part of run-job
+    global nested_layers_hosts need_xen_hosts
+    lappend nested_layers_hosts $need_xen_hosts
+    set need_xen_hosts $nested_hosts
+    per-host-prep
+}
+
 #---------- test recipes ----------
 
 proc need-hosts/test-debian-nomigr {} { return host }
Signature · 1 line
1.8.3.1
author
#32 lore raw

PATCH v12 15/21 Integrate Ian J. code and small corrections.

In reply to Robert Ho
---
 sg-run-job | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
sg-run-job +3 -3
index 7cb6cac..4ae651d 100755
@@ -69,14 +69,14 @@ proc run-job {job} {
 
     per-host-ts broken  host-install/@(*) ts-host-install-twice
 
-    per-hosts-prep
+    per-host-prep
 
     if {$ok} { catching-otherwise fail      run-job/$jobinfo(recipe)      }
 
     while 1 {
 	per-host-finish
-
-	if {![llength nested_layers_hosts]} break
+	
+	if {![llength $nested_layers_hosts]} break
 
 	per-host-ts . = final-poweroff {ts-host-powercycle --power=0}
 
Signature · 1 line
1.8.3.1
#33 lore raw

Re: [OSSTest Nested v12 15/21] Integrate Ian J. code and small corrections.

Robert Ho writes ("[OSSTest Nested v12 15/21] Integrate Ian J. code and small corrections."):

Quoted 1 line

Thanks for fixing up my code.

But can you please squash these fixes into their corresponding patches ? You should add your own S-o-b to each of them, next to mine.

Thanks,
Ian.

author
#34 lore raw

PATCH v12 16/21 Add PDU power method for nested L1 and L2 guest

In reply to Robert Ho

For nested host/guest, its power on/off method shall be its host invoke $(toolstack)->create/destroy method.

---
 Osstest/PDU/guest.pm   | 63 ++++++++++++++++++++++++++++++++++++++++++++++++++
 Osstest/TestSupport.pm |  3 +++
 2 files changed, 66 insertions(+)
 create mode 100755 Osstest/PDU/guest.pm
Osstest/PDU/guest.pm new file +63 -0
new file mode 100755
index 0000000..8b4b9f5
@@ -0,0 +1,63 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Intel.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+package Osstest::PDU::guest;
+
+use strict;
+use warnings;
+use Switch;
+
+use Osstest;
+use Osstest::TestSupport;
+use IO::File;
+
+BEGIN {
+    use Exporter ();
+    our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+    $VERSION     = 1.00;
+    @ISA         = qw(Exporter);
+    @EXPORT      = qw();
+    %EXPORT_TAGS = ( );
+
+    @EXPORT_OK   = qw();
+}
+
+sub new {
+    my ($class, $ho) = @_;
+    return bless { Host => $ho }, $class;
+}
+
+sub pdu_power_state {
+    my ($mo, $on) = @_;
+    my $onoff= $on ? "on" : "off";
+    my $ho;
+    my $gho;
+
+    switch ($mo->{Host}{Ident}){
+    	case /l1/	{($ho,$gho)=ts_get_host_guest('host',$mo->{Host}{Name})}
+    	case /l2/	{($ho,$gho)=ts_get_host_guest('l1',$mo->{Host}{Name})}
+    	else		{logm("Unkown Host Ident $mo->{Host}{Ident}");die;}
+    }
+
+    logm("$ho->{Name} going to power $onoff $gho->{Name}");
+    switch ($onoff){
+    	case /on/	{toolstack($ho)->create($gho)}
+    	case /off/	{toolstack($ho)->destroy($gho)}
+    }
+}
+
+1;
Osstest/TestSupport.pm +3 -0
index 1768c5e..06a0c57 100644
@@ -878,6 +878,9 @@ sub selecthost ($) {
     $ho->{Ether}= get_host_property($ho,'ether');
     $ho->{DiskDevice}= get_host_property($ho,'disk-device');
     $ho->{Power}= get_host_property($ho,'power-method');
+    if ($ho->{Ident} =~ m/l1|l2/) {
+    	$ho->{Power} = 'guest';
+    }
 
     $mhostdb->default_methods($ho);
 
Signature · 1 line
1.8.3.1
#35 lore raw

Re: [OSSTest Nested v12 16/21] Add PDU power method for nested L1 and L2 guest

Robert Ho writes ("[OSSTest Nested v12 16/21] Add PDU power method for nested L1 and L2 guest"):

Quoted 2 lines

For nested host/guest, its power on/off method shall be its host invoke $(toolstack)->create/destroy method.

Thanks for this patch, which I have substantially edited for my v14.

However, I notice that it was missing a signed-off-by. Can you please confirm that I should add your s-o-b ?

Thanks,
Ian.

author
#36 lore raw

Re: [OSSTest Nested v12 16/21] Add PDU power method for nested L1 and L2 guest

Quoted 11 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 12:36 AM
To: Hu, Robert <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Jin, Gordon <[email protected]>; Zheng, Di <[email protected]>
Subject: Re: [OSSTest Nested v12 16/21] Add PDU power method for nested L1 and L2 guest

Robert Ho writes ("[OSSTest Nested v12 16/21] Add PDU power method for nested L1 and L2 guest"):

Quoted 7 lines
Quoted 2 lines

For nested host/guest, its power on/off method shall be its host invoke $(toolstack)->create/destroy method.

Thanks for this patch, which I have substantially edited for my v14.

However, I notice that it was missing a signed-off-by. Can you please confirm that I should add your s-o-b ?

Sorry I forgot to add my s-o-b. Yes, please add mine.

Quoted 3 lines

Thanks,
Ian.

#37 lore raw

Re: [OSSTest Nested v12 16/21] Add PDU power method for nested L1 and L2 guest

Quoted 12 lines · Robert
Quoted 12 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 12:36 AM
To: Hu, Robert <[email protected]>
Cc: [email protected]; [email protected]; [email protected];
Jin, Gordon <[email protected]>; Zheng, Di <[email protected]>
Subject: Re: [OSSTest Nested v12 16/21] Add PDU power method for nested L1 and L2 guest

Robert Ho writes ("[OSSTest Nested v12 16/21] Add PDU power method for nested L1 and L2 guest"):

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

For nested host/guest, its power on/off method shall be its host invoke $(toolstack)->create/destroy method.

Thanks for this patch, which I have substantially edited for my v14.

However, I notice that it was missing a signed-off-by. Can you please confirm that I should add your s-o-b ?

Sorry I forgot to add my s-o-b. Yes, please add mine.

It's useful in cases like these to explicitly spell out the full Signed-off -by (or Ack-, or Reviewed-by), with your email address etc. This removes any possible ambiguity etc.

Ian.

Quoted 3 lines
Quoted 3 lines

Thanks,
Ian.

author
#38 lore raw

PATCH v12 17/21 Compose the main recipe of nested test job

In reply to Robert Ho

After integrated Ian J.'s code of nested infrasture changes, re- write the nested job's recipe.

---
 sg-run-job | 10 ++++++++++
 1 file changed, 10 insertions(+)
sg-run-job +10 -0
index 4ae651d..3f24fd6 100755
@@ -347,6 +347,16 @@ proc run-job/test-pair-oneway {} {
     run-ts . =              ts-guest-stop          dst_host          + debian
 }
 
+proc need-hosts/test-nested {} {return host}
+proc run-job/test-nested {} {
+    run-ts . = ts-debian-hvm-install + host l1
+    run-ts . = ts-nested-setup + host l1
+    nested-layer-descend l1
+    run-ts . = ts-debian-hvm-install l1 l2
+    run-ts . = ts-guest-stop l1 l2
+    run-ts . = ts-guest-destroy + host l1
+}
+
 proc test-guest-migr {g} {
     set to_reap [spawn-ts . = ts-migrate-support-check + host $g 1]
     set can_migrate [reap-ts $to_reap]
Signature · 1 line
1.8.3.1
author
#39 lore raw

PATCH v12 18/21 After Xen install, configure its xenbr0 interface as dhcp.

In reply to Robert Ho

This is more idiomatic because, even if host is fixed IP in DHCP server side, client side shall still be dhcp mode, rather than fixed self. And if it is not configured fixed in DHCP server side, it shall more necessarily use dhcp mode.

This applies for both nested test and other test cases.

---
 ts-xen-install | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
ts-xen-install +1 -4
index e039173..6d04982 100755
@@ -303,10 +303,7 @@ END
             if (m/^\s* iface \s+ (?: $physif | xenbr0 ) \s+ inet \s /x) {
                 $suppress= 1;
                 print EO <<END;
-iface $iface inet static
-    address $ho->{Ip}
-    netmask $netmask
-    gateway $gateway
+iface $iface inet dhcp
 $bridgex
 END
             }
Signature · 1 line
1.8.3.1
author
#40 lore raw

PATCH v12 19/21 Selecthost uses dynamic IP address if the host is not configured static IP.

In reply to Robert Ho

In this patch
1. in check_ip(), we change $lstash to use {Name} key-value, rather than {Guest}, because {Name} is both usable by $ho and $gho hash.
2. $ho->{Ether} assignment: if configured in host property, good, use it; otherwise, try to see if runvar has the assignment (this is the case of nested test).

---
 Osstest/DhcpWatch/leases.pm |  2 +-
 Osstest/TestSupport.pm      | 23 +++++++++++++----------
 2 files changed, 14 insertions(+), 11 deletions(-)
Osstest/DhcpWatch/leases.pm +1 -1
index 9a74c40..16e67d3 100644
@@ -61,7 +61,7 @@ sub check_ip ($$) {
 	if (!defined $leases) { return "connect to $leasesfn: $!"; }
     }
 
-    my $lstash= "dhcpleases-$gho->{Guest}";
+    my $lstash= "dhcpleases-$gho->{Name}";
     my $inlease;
     my $props;
     my $best;
Osstest/TestSupport.pm +13 -10
index 06a0c57..1d1d6b1 100644
@@ -875,7 +875,7 @@ sub selecthost ($) {
     $ho->{Fqdn} = get_host_property($ho,'fqdn',$defaultfqdn);
 
 
-    $ho->{Ether}= get_host_property($ho,'ether');
+    $ho->{Ether}= get_host_property($ho,'ether') || $r{"$ho->{Ident}_ether"};
     $ho->{DiskDevice}= get_host_property($ho,'disk-device');
     $ho->{Power}= get_host_property($ho,'power-method');
     if ($ho->{Ident} =~ m/l1|l2/) {
@@ -889,17 +889,20 @@ sub selecthost ($) {
     serial_host_setup($ho);
 
     $ho->{IpStatic} = get_host_property($ho,'ip-addr');
-    if (defined $r{"${ident}_ip"}) {
-        $ho->{Ip} = $r{"${ident}_ip"};
-    } else {
-        if (!defined $ho->{IpStatic}) {
-	    my $ip_packed= gethostbyname($ho->{Fqdn});
-	    die "$ho->{Fqdn} ?" unless $ip_packed;
+    if (!defined $ho->{IpStatic}) {
+	my $ip_packed= gethostbyname($ho->{Fqdn});
+	if (!$ip_packed) {
+	    logm("Going to look for $ho->{Name}'s ether: $ho->{Ether}");
+	    $ho->{DhcpWatch}->check_ip($ho);
+	    $ho->{IpStatic}=$ho->{Ip};
+	}
+#	die "$ho->{Fqdn} ?" unless $ip_packed;
+	else {
 	    $ho->{IpStatic}= inet_ntoa($ip_packed);
 	    die "$ho->{Fqdn} ?" unless defined $ho->{IpStatic};
-        }
-        $ho->{Ip}= $ho->{IpStatic};
-    }
+	    $ho->{Ip}= $ho->{IpStatic};
+	}
+    }    
 
     #----- tftp -----
 
Signature · 1 line
1.8.3.1
#41 lore raw

Re: [OSSTest Nested v12 19/21] Selecthost uses dynamic IP address if the host is not configured static IP.

Robert Ho writes ("[OSSTest Nested v12 19/21] Selecthost uses dynamic IP address if the host is not configured static IP."):

Quoted 6 lines

In this patch
1. in check_ip(), we change $lstash to use {Name} key-value, rather than {Guest}, because {Name} is both usable by $ho and $gho hash.
2. $ho->{Ether} assignment: if configured in host property, good, use it; otherwise, try to see if runvar has the assignment (this is the case of nested test).

I am going to drop this patch from my v14 of the nested HVM tests series.

This is not because this change is valueless. But, in the context of my other changes, it is no longer needed for supporting L1 guests: L1 guests do not any longer run through the code path that this patch modifies.

This patch might still be valuable in the future to support physical hosts without static IP addresses. But AFAIAA none of our environments have such things and dropping this patch will avoid me having to review this code.

It will also avoid anyone having to test it.

Ian.

author
#42 lore raw

Re: [OSSTest Nested v12 19/21] Selecthost uses dynamic IP address if the host is not configured static IP.

Quoted 26 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 12:59 AM
To: Hu, Robert <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Jin, Gordon <[email protected]>; Zheng, Di <[email protected]>
Subject: Re: [OSSTest Nested v12 19/21] Selecthost uses dynamic IP address if the host is not configured static IP.

Robert Ho writes ("[OSSTest Nested v12 19/21] Selecthost uses dynamic IP address if the host is not configured static IP."):

Quoted 6 lines

In this patch
1. in check_ip(), we change $lstash to use {Name} key-value, rather than {Guest}, because {Name} is both usable by $ho and $gho hash.
2. $ho->{Ether} assignment: if configured in host property, good, use it; otherwise, try to see if runvar has the assignment (this is the case of nested test).

I am going to drop this patch from my v14 of the nested HVM tests series.

This is not because this change is valueless. But, in the context of my other changes, it is no longer needed for supporting L1 guests: L1 guests do not any longer run through the code path that this patch modifies.

Quoted 6 lines

This patch might still be valuable in the future to support physical hosts without static IP addresses. But AFAIAA none of our environments have such things and dropping this patch will avoid me having to review this code.

It will also avoid anyone having to test it.

OK. I have no objection if current L1 running around this code path.

Quoted 2 lines

Ian.

author
#43 lore raw

PATCH v12 20/21 Don't lvextend if actually no more space to extend

In reply to Robert Ho

Though passes if judgement, the

	overall_limit_pe(\$vg_more_free_pe);

may final judge no more free_pe to extend.
So, check if $vg_more_free_pe is 0, if so, we don't lvextend, otherwise lvextend will report error on nonsense operation.

---
 ts-xen-build-prep | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
ts-xen-build-prep +2 -1
index 03ad35c..d3f9d01 100755
@@ -155,7 +155,8 @@ sub lvextend1 ($$$) {
         logm("$what: unstriped $vg_more_free_pe PEs");
         overall_limit_pe(\$vg_more_free_pe);
         $more_pe += $vg_more_free_pe;
-        target_cmd_root($ho, "lvextend -i1 -l +$vg_more_free_pe $lv");
+        target_cmd_root($ho, "lvextend -i1 -l +$vg_more_free_pe $lv")
+		if $vg_more_free_pe != 0;
     }
 
     my $timeout = 2000 + int($pe_size * 0.000003 * $more_pe);
Signature · 1 line
1.8.3.1
#44 lore raw

Re: [OSSTest Nested v12 20/21] Don't lvextend if actually no more space to extend

Robert Ho writes ("[OSSTest Nested v12 20/21] Don't lvextend if actually no more space to extend"):

Quoted 5 lines

Though passes if judgement, the

	overall_limit_pe(\$vg_more_free_pe);

may final judge no more free_pe to extend.
So, check if $vg_more_free_pe is 0, if so, we don't lvextend, otherwise lvextend will report error on nonsense operation.

I think this should be done by moving the start of the if block to after overall_limit_pe. That would avoid a tested if (and various other slight anomalies).

Thanks,
Ian.

#45 lore raw

Re: [OSSTest Nested v12 20/21] Don't lvextend if actually no more space to extend

Quoted 11 lines · Ian Jackson

Robert Ho writes ("[OSSTest Nested v12 20/21] Don't lvextend if actually no more space to extend"):

Quoted 5 lines

Though passes if judgement, the

	overall_limit_pe(\$vg_more_free_pe);

may final judge no more free_pe to extend.
So, check if $vg_more_free_pe is 0, if so, we don't lvextend, otherwise lvextend will report error on nonsense operation.

I think this should be done by moving the start of the if block to after overall_limit_pe. That would avoid a tested if (and various other slight anomalies).

IOW move the overall_limit_pe outside the if?

Like so:

ts-xen-build-prep +1 -1
index 03ad35c..b35e91b 100755
@@ -151,9 +151,9 @@ sub lvextend1 ($$$) {
 
     $do_limit_pe->(\$vg_more_free_pe, 'unstriped');
 
+    overall_limit_pe(\$vg_more_free_pe);
     if ($vg_more_free_pe) {
         logm("$what: unstriped $vg_more_free_pe PEs");
-        overall_limit_pe(\$vg_more_free_pe);
         $more_pe += $vg_more_free_pe;
         target_cmd_root($ho, "lvextend -i1 -l +$vg_more_free_pe $lv");
     }

(untested, but I've just tripped over this in a standalone run myself).

Ian.

Acked-by
#46 lore raw

Re: [OSSTest Nested v12 20/21] Don't lvextend if actually no more space to extend

Ian Campbell writes ("Re: [OSSTest Nested v12 20/21] Don't lvextend if actually no more space to extend"):

Quoted 25 lines · Ian Jackson
Quoted 3 lines

I think this should be done by moving the start of the if block to after overall_limit_pe. That would avoid a tested if (and various other slight anomalies).

IOW move the overall_limit_pe outside the if?

Like so:

ts-xen-build-prep +1 -1
index 03ad35c..b35e91b 100755
@@ -151,9 +151,9 @@ sub lvextend1 ($$$) {
 
     $do_limit_pe->(\$vg_more_free_pe, 'unstriped');
 
+    overall_limit_pe(\$vg_more_free_pe);
     if ($vg_more_free_pe) {
         logm("$what: unstriped $vg_more_free_pe PEs");
-        overall_limit_pe(\$vg_more_free_pe);
         $more_pe += $vg_more_free_pe;
         target_cmd_root($ho, "lvextend -i1 -l +$vg_more_free_pe $lv");
     }
Quoted 1 line

(untested, but I've just tripped over this in a standalone run myself).

Yes. With a suitable commit message:

author
#47 lore raw

PATCH v12 21/21 await_tcp(): check_ip for each loop iteration

In reply to Robert Ho

await_tcp usually invoked after a reboot, $ho IP may change, especially when $ho 'client name' of DHCP request changes. Therefore, await_tcp() check will fail if we don't update $ho->{IP} accordingly.
But, if $ho has static IP, this won't apply.
Also, this patch add $ho->{Ip} in its $what message, so that we can observe clearly if $ho->{Ip} changes.

To use $ho->{IpStatic} to indicate what its name means, we have to correct its assignment: we assign value to it only from configuration or DNS query. We already have $ho->{Ip}, which shall indicate $ho IP address in dhcp cases.

---
 Osstest/TestSupport.pm | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
Osstest/TestSupport.pm +5 -2
index 1d1d6b1..298fca9 100644
@@ -894,7 +894,6 @@ sub selecthost ($) {
 	if (!$ip_packed) {
 	    logm("Going to look for $ho->{Name}'s ether: $ho->{Ether}");
 	    $ho->{DhcpWatch}->check_ip($ho);
-	    $ho->{IpStatic}=$ho->{Ip};
 	}
 #	die "$ho->{Fqdn} ?" unless $ip_packed;
 	else {
@@ -1945,8 +1944,12 @@ sub target_tcp_check ($$) {
 sub await_tcp ($$$) {
     my ($maxwait,$interval,$ho) = @_;
     poll_loop($maxwait,$interval,
-              "await tcp $ho->{Name} $ho->{TcpCheckPort}",
+              "await tcp $ho->{Name} $ho->{Ip} $ho->{TcpCheckPort}",
               sub {
+    	#await_tcp() usually called after a reboot, where $ho IP may change.
+    	#unless it has static IP
+    	$ho->{DhcpWatch}->check_ip($ho) 
+    		unless $ho->{IpStatic};
         return target_tcp_check($ho,$interval);
     });
 }
Signature · 1 line
1.8.3.1
#48 lore raw

Re: [OSSTest Nested v12 21/21] await_tcp(): check_ip for each loop iteration

Robert Ho writes ("[OSSTest Nested v12 21/21] await_tcp(): check_ip for each loop iteration"):

Quoted 8 lines

await_tcp usually invoked after a reboot, $ho IP may change, especially when $ho 'client name' of DHCP request changes. Therefore, await_tcp() check will fail if we don't update $ho->{IP} accordingly.
But, if $ho has static IP, this won't apply.
Also, this patch add $ho->{Ip} in its $what message, so that we can observe clearly if $ho->{Ip} changes.

Quoted 4 lines

To use $ho->{IpStatic} to indicate what its name means, we have to correct its assignment: we assign value to it only from configuration or DNS query. We already have $ho->{Ip}, which shall indicate $ho IP address in dhcp cases.

I don't think I'm quite convinced by this reasoning.

This patch is effectively turning await_tcp into most of guest_await_dhcp_tcp.

I think perhaps instead target_reboot and target_reboot hard should unset $ho->{Ip} and await_tcp should call guest_check_ip only if $ho->{Ip} is not set.

The unsetting of $ho->{Ip} should probably actually involve also a call to the part of selecthost which sets it from IpStatic and gethostbyname etc. That would have to be made conditional on $ho->{Fqdn} existing because L1 nested hosts don't have a useful fqdn.

Ian.

#49 lore raw
In reply to Robert Ho

Robert Ho writes ("[OSSTest Nested v12 00/21] Introduction of netsted HVM test job"):

Quoted 1 line

This patch set adds nested HVM test case for osstest.

Sorry for the delay in reviewing the last few patches in this series. I've finished with it now. I hope my previous review comments etc. have been helpful.

Do you intend to resend a v13 soon ? If you are finding my comments and suggestions difficult to implement, perhaps it would be better for you to give me a git branch for me to work on.

I would be happy to rework the bits of your series which need the most reorganisation, and then pass it back on to you for testing. Would that be a good workflow for you ?

Regards,
Ian.

author
#50 lore raw
Quoted 9 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Wednesday, September 16, 2015 10:37 PM
To: Hu, Robert
Cc: [email protected]; [email protected]; [email protected]; Jin, Gordon; Zheng, Di
Subject: Re: [OSSTest Nested v12 00/21] Introduction of netsted HVM test job

Quoted 7 lines

Robert Ho writes ("[OSSTest Nested v12 00/21] Introduction of netsted HVM test job"):

Quoted 1 line

This patch set adds nested HVM test case for osstest.

Sorry for the delay in reviewing the last few patches in this series. I've finished with it now. I hope my previous review comments etc. have been helpful.

Though just took a quick glance at a few, I believe they are helpful.

Quoted 4 lines

Do you intend to resend a v13 soon ? If you are finding my comments and suggestions difficult to implement, perhaps it would be better for you to give me a git branch for me to work on.

Yes.
That would be a great favor, thank you very very much!
Would you give me a branch I can push my code to? for I'm afraid that company policy doesn't allow me to set up git repo for external access.

Quoted 4 lines

I would be happy to rework the bits of your series which need the most reorganisation, and then pass it back on to you for testing. Would that be a good workflow for you ?

Sure, I'd love to.

Quoted 3 lines

Regards,
Ian.

#51 lore raw

Hu, Robert writes ("RE: [OSSTest Nested v12 00/21] Introduction of netsted HVM test job"):

Quoted 6 lines

[Ian Jackson:]

Quoted 3 lines

Sorry for the delay in reviewing the last few patches in this series. I've finished with it now. I hope my previous review comments etc. have been helpful.

Though just took a quick glance at a few, I believe they are helpful.

OK, good, thanks.

Quoted 5 lines
Quoted 3 lines

Do you intend to resend a v13 soon ? If you are finding my comments and suggestions difficult to implement, perhaps it would be better for you to give me a git branch for me to work on.

Yes.

Great.

Quoted 3 lines

That would be a great favor, thank you very very much! Would you give me a branch I can push my code to? for I'm afraid that company policy doesn't allow me to set up git repo for external access.

How irritating. We will set you up with an account on xenbits. (Lars, I trust this is OK.)

Robert, could you please send me your ssh public key ?

Quoted 5 lines
Quoted 3 lines

I would be happy to rework the bits of your series which need the most reorganisation, and then pass it back on to you for testing. Would that be a good workflow for you ?

Sure, I'd love to.

Great.

Thanks,
Ian.

#52 lore raw

v14 cover Nested HVM preparation patches

This is the first part of Robert Hu's osstest patch series to support nested HVM tests. (v13 was passed to me as a git branch `under the table' by Robert.) I have rewritten the commit messages, refactored a few patches, and reordered the series slightly.

I have made only very minimal code changes to Robert's v13, and these initial patches are straightforward and have been well-reviewed. In IRL discussion with Ian Campbell we concluded that rereview by him would not be worthwhile. Personally I am fairly confident that these changes won't break anything. Accordingly I am about to push this part of the series into osstest pretest, along with a number of other miscellaneous cleanup and fixup patches from various other contributors.

The remaining parts of the series need more substantial work by me and then retesting by Robert. I will send a `v14 PART 2' when I have done that.

Thanks,
Ian.

#53 lore raw

PATCH v14 1/9 Tcl: Provide lunappend

---
 tcl/osstestlib.tcl |    7 +++++++
 1 file changed, 7 insertions(+)
tcl/osstestlib.tcl +7 -0
index 61a6a09..b5a52d3 100644
@@ -75,6 +75,13 @@ proc lshift {listvar} {
     return $head
 }
 
+proc lunappend {listvar} {
+    upvar 1 $listvar list
+    set tail [lindex $list end]
+    set list [lrange $list 0 end-1]
+    return $tail
+}
+
 proc var-or-default {varname {default {}}} {
     upvar 1 $varname var
     if {[info exists var]} { return $var }
Signature · 1 line
1.7.10.4
#54 lore raw

PATCH v14 2/9 Debian grub2: Optimize and re-format submenu parsing

In reply to Ian Jackson

From: Robert Ho <[email protected]>

* space between ')' and '{'; and after '='
* omit unnecessary 'define' and '!defined' usage
* break long '{}' into several lines

---
v14: Drop removal of MenuEntryPath setting in grub2 submenu parse

---
 Osstest/Debian.pm |    8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
Osstest/Debian.pm +5 -3
index a8b62ca..2b8d9f3 100644
@@ -421,10 +421,10 @@ sub setupboot_grub2 ($$$$) {
             next if m/^\s*\#/ || !m/\S/;
             if (m/^\s*\}\s*$/) {
                 die unless $entry || $submenu;
-                if (!defined $entry && defined $submenu) {
+                if (!$entry && $submenu) {
                     logm("Met end of a submenu $submenu->{StartLine}..$.. ".
                         "Our want kern is $want_kernver");
-                    $submenu=undef;
+                    $submenu= undef;
                     pop @offsets;
                     $offsets[$#offsets]++;
                     next;
@@ -465,7 +465,9 @@ sub setupboot_grub2 ($$$$) {
             }
             if (m/^\s*menuentry\s+[\'\"](.*)[\'\"].*\{\s*$/) {
                 die $entry->{StartLine} if $entry;
-                $entry= { Title => $1, StartLine => $., MenuEntryPath => join ">", @offsets };
+                $entry= { Title => $1,
+			  StartLine => $.,
+			  MenuEntryPath => join ">", @offsets };
                 $offsets[$#offsets]++;
             }
             if (m/^\s*submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {
Signature · 1 line
1.7.10.4
#55 lore raw

PATCH v14 3/9 Debian grub2: Correct a mistake in Xen entry parsing pattern

In reply to Ian Jackson

From: Robert Ho <[email protected]>

In b77a6a2522d8 "Changes to support '/boot' leading paths of kernel, xen, in grub", this pattern was erroneously changed.

Revert that aspect of that commit, so that we once again require Xen image versions to start with a digit.

---
 Osstest/Debian.pm |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
index 2b8d9f3..7677fa3 100644
@@ -479,7 +479,7 @@ sub setupboot_grub2 ($$$$) {
                 $entry->{Hv}= $1;
                 $entry->{Chainload} = 1;
             }
-            if (m/^\s*multiboot\s*(?:\/boot)?\/(xen\S+)/) {
+            if (m/^\s*multiboot\s*(?:\/boot)?\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
                 die unless $entry;
                 $entry->{Hv}= $1;
                 $entry->{Chainload} = 0;
Signature · 1 line
1.7.10.4
#56 lore raw

PATCH v14 4/9 Debian HVM guests: Comment out CDROM entry in HVM guest VM

In reply to Ian Jackson

From: Robert Ho <[email protected]>

Comment out the CDROM entry in the installed system's sources.list. The installation ISO is not generally present in the virtual CDROM while the guest is running, so this entry is nonfunctional.

Removing it causes the L1 HVM guest VM to use the http entry instead. The result is that now `apt-get' works in the guest.

---
 ts-debian-hvm-install |    7 +++++++
 1 file changed, 7 insertions(+)
ts-debian-hvm-install +7 -0
index 0f3996a..5fb6f4c 100755
@@ -106,6 +106,13 @@ if [ -e \$b/debian/grubx64.efi ] ; then
 fi
 END
 
+preseed_hook_command($gho, 'late_command', '', <<END);
+#!/bin/sh
+set -ex
+
+in-target sed -i 's/^deb *cdrom/#&/g' /etc/apt/sources.list
+END
+
     $preseed_file .= preseed_hook_cmds();
 
     return $preseed_file;
Signature · 1 line
1.7.10.4
#57 lore raw

PATCH v14 5/9 Debian HVM guests: Honour guest disk and ram size runvars

In reply to Ian Jackson

From: Robert Ho <[email protected]>

Allow runvars to specify guest disk and ram size, turning previous values into defaults:

The default disk size for the guest is `10000M' which is not going to be sufficient for nested HVM tests. We are going to want to use a larger disk size for the nested L1. The appropriate disk_size will be defined in make-flight, in forthcoming changes.

Also, also allow ram size to be defined by runvar. The runvar takes precedence over the default (which is calculated based on host RAM etc.)

No functional change with current sets of runvars.

---
 Osstest/TestSupport.pm |    3 ++-
 ts-debian-hvm-install  |   14 ++++++++++----
 2 files changed, 12 insertions(+), 5 deletions(-)
Osstest/TestSupport.pm +2 -1
index 7ba51ec..d966fe1 100644
@@ -1745,7 +1745,8 @@ sub target_put_guest_image ($$;$) {
 
 sub more_prepareguest_hvm ($$$$;@) {
     my ($ho, $gho, $ram_mb, $disk_mb, %xopts) = @_;
-    
+    # $ram_mb and $disk_mb are defaults, used if runvars don't say 
+
     my $passwd= 'xenvnc';
 
     prepareguest_part_lvmdisk($ho, $gho, $disk_mb);
ts-debian-hvm-install +10 -4
index 5fb6f4c..f2fc31b 100755
@@ -220,13 +220,19 @@ sub prep () {
     my $host_freemem_mb = host_get_free_memory($ho);
     my $ram_minslop = 100;
     my $ram_lots = 5000;
-    if ($host_freemem_mb > $ram_lots * 2 + $ram_minslop) {
-        $ram_mb = $ram_lots;
-    } else {
-        $ram_mb = 768;
+
+    $ram_mb = guest_var($gho,'memsize',undef);
+    if (!$ram_mb) {
+         if ($host_freemem_mb > $ram_lots * 2 + $ram_minslop) {
+              $ram_mb = $ram_lots;
+         } else {
+              $ram_mb = 768;
+         }
     }
     logm("Host has $host_freemem_mb MB free memory, setting guest memory size to $ram_mb MB");
 
+    $disk_mb= guest_var($gho,'disksize',$disk_mb);
+
     more_prepareguest_hvm($ho,$gho, $ram_mb, $disk_mb,
                           OnReboot => 'preserve',
                           Bios => $r{bios},
Signature · 1 line
1.7.10.4
#58 lore raw

PATCH v14 6/9 Debian HVM guests: Honour enable_nestedhvm guest runvar

In reply to Ian Jackson

From: Robert Ho <[email protected]>

There are not yet any jobs which set this.

---
v14: Use guest_var_boolean

---
 ts-debian-hvm-install |    5 +++++
 1 file changed, 5 insertions(+)
ts-debian-hvm-install +5 -0
index f2fc31b..2c7580c 100755
@@ -215,6 +215,10 @@ sub prep () {
 
     iso_create_empty($ho, $emptyiso, $emptydir);
 
+    my $extra_config = '';
+    $extra_config .= "nestedhvm=1\n"
+	if guest_var_boolean($gho,"enable_nestedhvm");
+
     # If host has >8G free memory, create a guest with 4G memory to catch
     # any error that triggers cross 4G boundary
     my $host_freemem_mb = host_get_free_memory($ho);
@@ -236,6 +240,7 @@ sub prep () {
     more_prepareguest_hvm($ho,$gho, $ram_mb, $disk_mb,
                           OnReboot => 'preserve',
                           Bios => $r{bios},
+                          ExtraConfig => $extra_config,
                           PostImageHook => sub {
         my $cmds = iso_copy_content_from_image($gho, $newiso);
         target_cmd_root($ho, $cmds, $isotimeout);
Signature · 1 line
1.7.10.4
#59 lore raw

PATCH v14 7/9 Host install: Break out host_install_postboot_complete

In reply to Ian Jackson

From: Robert Ho <[email protected]>

We are going to want to do this for nested HVM L1s too.

No functional change.

---
v14: Squashed two patches into this one.

---
 Osstest/TestSupport.pm |    6 ++++++
 ts-host-install        |    2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)
Osstest/TestSupport.pm +6 -0
index d966fe1..a3a9982 100644
@@ -115,6 +115,7 @@ BEGIN {
                       iso_gen_flags_basic
                       iso_copy_content_from_image
                       guest_editconfig_nocd
+                      host_install_postboot_complete
                       );
     %EXPORT_TAGS = ( );
 
@@ -2395,4 +2396,9 @@ sub guest_editconfig_nocd ($$) {
     });
 }
 
+sub host_install_postboot_complete ($) {
+    my ($ho) = @_;
+    target_cmd_root($ho, "update-rc.d osstest-confirm-booted start 99 2 .");
+}
+
 1;
ts-host-install +1 -1
index f734a9c..3f7f060 100755
@@ -165,7 +165,7 @@ END
     target_putfilecontents_root_stash($ho,10,$coredumps_conf,
 				'/etc/security/limits.d/coredumps.conf');
 
-    target_cmd_root($ho, "update-rc.d osstest-confirm-booted start 99 2 .");
+    host_install_postboot_complete($ho);
 
     logm('OK: install completed');
 }   
Signature · 1 line
1.7.10.4
#60 lore raw

PATCH v14 8/9 Host install: Break out target_core_dump_setup

In reply to Ian Jackson

We are going to want to do this for nested HVM L1s too.

No functional change.

---
v14: Split this nfc patch out from other changes
Drop changes to whitespace usage
Rename new function from `core_dump_setup'
Drop introduction of an unnecessary mkdir -p

---
 Osstest/TestSupport.pm |   19 +++++++++++++++++++
 ts-host-install        |   15 +--------------
 2 files changed, 20 insertions(+), 14 deletions(-)
Osstest/TestSupport.pm +19 -0
index a3a9982..fb24a1e 100644
@@ -116,6 +116,7 @@ BEGIN {
                       iso_copy_content_from_image
                       guest_editconfig_nocd
                       host_install_postboot_complete
+                      target_core_dump_setup
                       );
     %EXPORT_TAGS = ( );
 
@@ -2401,4 +2402,22 @@ sub host_install_postboot_complete ($) {
     target_cmd_root($ho, "update-rc.d osstest-confirm-booted start 99 2 .");
 }
 
+sub target_core_dump_setup ($) {
+    my ($ho) = @_;
+    target_cmd_root($ho, 'mkdir -p /var/core');
+    target_editfile_root($ho, '/etc/sysctl.conf',
+	sub { target_editfile_kvp_replace(
+		  "kernel.core_pattern",
+		  # %p==pid,%e==executable name,%t==timestamp
+		  "/var/core/%t.%p.%e.core") });
+    target_cmd_root($ho, "sysctl --load /etc/sysctl.conf");
+    my $coredumps_conf = <<'END';
+#<domain>      <type>  <item>       <value>
+*               soft    core         -1
+root            soft    core         -1
+END
+    target_putfilecontents_root_stash($ho,10,$coredumps_conf,
+				'/etc/security/limits.d/coredumps.conf');
+}
+
 1;
ts-host-install +1 -14
index 3f7f060..5bbc0a7 100755
@@ -150,20 +150,7 @@ END
 	});
     }
 
-    target_cmd_root($ho, 'mkdir -p /var/core');
-    target_editfile_root($ho, '/etc/sysctl.conf',
-	sub { target_editfile_kvp_replace(
-		  "kernel.core_pattern",
-		  # %p==pid,%e==executable name,%t==timestamp
-		  "/var/core/%t.%p.%e.core") });
-    target_cmd_root($ho, "sysctl --load /etc/sysctl.conf");
-    my $coredumps_conf = <<'END';
-#<domain>      <type>  <item>       <value>
-*               soft    core         -1
-root            soft    core         -1
-END
-    target_putfilecontents_root_stash($ho,10,$coredumps_conf,
-				'/etc/security/limits.d/coredumps.conf');
+    target_core_dump_setup($ho);
 
     host_install_postboot_complete($ho);
 
Signature · 1 line
1.7.10.4
#61 lore raw

PATCH v14 9/9 Host install: Move target_core_dump_setup call

In reply to Ian Jackson

Move this into host_install_postboot_complete. All hosts subject to host_install_postboot_complete should have the coredump settings too.

No functional change.

---
v14: New patch.
Dropped patch which adds core_dump_setup call to ts-nested-setup

---
 Osstest/TestSupport.pm |    1 +
 ts-host-install        |    2 --
 2 files changed, 1 insertion(+), 2 deletions(-)
Osstest/TestSupport.pm +1 -0
index fb24a1e..168aa9c 100644
@@ -2399,6 +2399,7 @@ sub guest_editconfig_nocd ($$) {
 
 sub host_install_postboot_complete ($) {
     my ($ho) = @_;
+    target_core_dump_setup($ho);
     target_cmd_root($ho, "update-rc.d osstest-confirm-booted start 99 2 .");
 }
 
ts-host-install +0 -2
index 5bbc0a7..9a46454 100755
@@ -150,8 +150,6 @@ END
 	});
     }
 
-    target_core_dump_setup($ho);
-
     host_install_postboot_complete($ho);
 
     logm('OK: install completed');
Signature · 1 line
1.7.10.4
#62 lore raw

PATCH v14 1/1 Nested HVM testing

In reply to Ian Jackson

This is the second part of v14 Robert Ho's osstest patch series to support nested HVM tests.

It is also available here:
git://xenbits.xen.org/people/iwj/xen.git
http://xenbits.xen.org/git-http/people/iwj/xen.git
in wip.nested-hvm.v14.part1..wip.nested-hvm.v14

Compared to Robert's v13, which was passed to me by private email,
* I have rebased onto current osstest pretest;
* I have changed how selecthost() is told it's dealing with a nested host (in practice, L1 guest);
* There are a large number of minor cleanups;
* There are some new preparatory cleanup and admin patches;
* I have rewritten almost all of the commit messages.

However, I have done only VERY LIMITED testing. Much of the code here is UNTESTED since my changes. My testing was confined to:
* Verifying that my changes to cs-adjust-flight worked
* Checking that ad-hoc runs of ts-host-reboot and ts-host-powercycle seemed to work when a guest was specified on the command line.

Robert, you kindly volunteered to test a revised version of this series. I would appreciate if you would check that all of this still works as you expect. I expect there will be some bugs, perhaps even very silly bugs, introduced by me.

I noticed that this series lacks guest serial debug keys and log collection for the L1 guest, because there is no Osstest/Serial/guest.pm. I would appreciate it if you would provide one. I don't think it needs to actually collect any logs, because the L1 serial output log will be collected as part of the L0 log collection. But it ought to support sending debug keys to the L1 guest. When you have provided it you can (in the same patch) fix the corresponding todo' in selecthost, changing noop' to `guest'.

Workflow:

Robert: I'm handing this (what I have called `part 2') over to you now.

When you make changes, feel free to either rebase, or to make fixup commits (perhaps in git-rebase -i --autosquash' format) on top. If you do the latter then you'll probably want to pass that to me as a git branch (via git push to xenbits or emailing me a git bundle), since squash!' and `fixup!' commits don't look good in email :-).

If you rebase, please put changes
v15: <description of what you did>
in the commit messages, as I have done myself in v14. Leave my v14 notes in place.

Of course if you have any comments or queries about how I have done things, they would be very welcome.

Please do not rebase any of the commits in wip.nested-hvm.v14.part1. If you discover bugs in `part 1' please let us know as I have fed that into the osstest self-test mill with the expectation that it will go into production.

I do not expect you to test the changes to cs-adjust-flight. I have done that. Indeed they are not really related to the Nested HVM work and Ian C and I may pick them up in another series.

Ian Campbell: You probably want to defer re-reviewing this until Robert reports back.

#63 lore raw

PATCH 10/26 cs-adjust-flight: Add some missing doc comment info

---
v14: New patch

---
 cs-adjust-flight |    8 ++++++++
 1 file changed, 8 insertions(+)
cs-adjust-flight +8 -0
index 834e2c8..a7b50f5 100755
@@ -22,6 +22,14 @@
 #   .                 means all jobs
 #   ^<pcre>           means $foo =~ m/^<pcre>/
 #   /<pcre>           means $foo =~ m/<pcre>/
+#
+# <dst-flight>:
+#   <flight>
+#   new:<intended-blessing>
+#
+# options:
+#   -v              verbose - list changes to stderr
+#   -D              debugging output to stderr
 
 # This is part of "osstest", an automated testing framework for Xen.
 # Copyright (C) 2009-2013 Citrix Inc.
Signature · 1 line
1.7.10.4
Acked-by on v1 10/26
#64 lore raw

Re: [OSSTEST PATCH 10/26] cs-adjust-flight: Add some missing doc comment info

Quoted 1 line · Ian Jackson

cs-adjust-flight line 28

Quoted 3 lines
cs-adjust-flight +3 -0
+# <dst-flight>:
+#   <flight>
+#   new:<intended-blessing>

I had no idea this latter syntax existed, how useful!

#65 lore raw

PATCH 11/26 cs-adjust-flight: Allow adjusting "this" flight

In reply to Ian Jackson

This allows cs-adjust-flight to be run by hand to adjust runvars, in a flight being used with hand-invocation of ./ts-* scripts.

---
v14: New patch

---
 cs-adjust-flight |    8 ++++++++
 1 file changed, 8 insertions(+)
cs-adjust-flight +8 -0
index a7b50f5..972d1d5 100755
@@ -26,6 +26,7 @@
 # <dst-flight>:
 #   <flight>
 #   new:<intended-blessing>
+#   running:                 uses OSSTEST_FLIGHT (may be `constructing' too)
 #
 # options:
 #   -v              verbose - list changes to stderr
@@ -375,6 +376,13 @@ sub main () {
             verbose_discard();
             changes();
         });
+    } elsif ($dstflightspec =~ m/^running:$/) {
+        $dstflight = $ENV{OSSTEST_FLIGHT} // die;
+        db_retry($dstflight,[qw(constructing running)],
+                 $dbh_tests, [qw(flights)], sub {
+            verbose_discard();
+            changes();
+        });
     } elsif ($dstflightspec =~ m/^new:/) {
         my $intended = $'; #';
         db_retry($dbh_tests, [qw(flights)], sub {
Signature · 1 line
1.7.10.4
#67 lore raw

PATCH 12/26 selecthost: Minor cleanups

In reply to Ian Jackson

Document the syntax for $ident.

Log the ident as well as the selected hostname.

---
v14: New patch

---
 Osstest/TestSupport.pm |   16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
Osstest/TestSupport.pm +15 -1
index 168aa9c..8f0122c 100644
@@ -804,6 +804,20 @@ sub power_state ($$) {
 sub selecthost ($) {
     my ($ident) = @_;
     # must be run outside transaction
+
+    # $ident is <identspec>
+    #
+    # <identspec> can be <ident>, typically "host" or "xxx_host"
+    # which means look up the runvar $r{$ident} which
+    # contains <hostspec>
+    # OR
+    # <identspec> can be <ident>=<hostspec>
+    # which means ignore <ident> except for logging purposes etc.
+    # and use <hostspec>
+    #
+    # <hostspec> uis <hostname> which means use that host (and all
+    # its flags and properties from the configuration and database)
+
     my $name;
     if ($ident =~ m/=/) {
         $ident= $`;
@@ -911,7 +925,7 @@ sub selecthost ($) {
 
     $mjobdb->host_check_allocated($ho);
 
-    logm("host: selected $ho->{Name} ".
+    logm("host $ho->{Ident}: selected $ho->{Name} ".
 	 (defined $ho->{Ether} ? $ho->{Ether} : '<unknown-ether>').
 	 " $ho->{Ip}".
          (!$ho->{Shared} ? '' :
Signature · 1 line
1.7.10.4
author Tested-by
#69 lore raw

Re: [OSSTEST PATCH 12/26] selecthost: Minor cleanups

Quoted 8 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Monday, September 28, 2015 6:02 PM
To: Ian Jackson <[email protected]>; [email protected]
Cc: Hu, Robert <[email protected]>
Subject: Re: [OSSTEST PATCH 12/26] selecthost: Minor cleanups

Quoted 8 lines
Quoted 5 lines · Ian Jackson

Document the syntax for $ident.

Log the ident as well as the selected hostname.

[Hu, Robert]

#70 lore raw

PATCH 13/26 selecthost: Support nested hosts (guests which are also hosts)

In reply to Ian Jackson

We introduce a new syntax: instead of a hostname (which might appear in a command line argument to a ts-* script and hence be passed to selecthost, or which might be in a runvar), we now support <hostspec>:<domname>.

Such `hosts' (let us refer to such a thing as an L1, although in principle further nesting may be possible) are expected to be dynamically created. So they do not have flags and properties in the configuration (or in an Executive instance's database).

The IP address is determined dynamically from the leases file. If the L1 is not running, then no IP address may be found. This is not an error. Users of this facility will need to make sure that ts-* scripts which are unaware of the L1's special status are only invoked when it is known that the L1 is up and has obtained its IP address.

`Power cycling' the L1 will be done by VM control operations in the L0; this will come in a subsequent patch.

`Serial access' to the L1 guest will likewise need to be done via the console arrangements in L0.

---
v14: New patch

---
 Osstest/TestSupport.pm |   37 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)
Osstest/TestSupport.pm +36 -1
index 8f0122c..9b10602 100644
@@ -801,6 +801,7 @@ sub power_state ($$) {
 
 #---------- host selection and properties ----------
 
+sub selecthost ($);
 sub selecthost ($) {
     my ($ident) = @_;
     # must be run outside transaction
@@ -815,8 +816,12 @@ sub selecthost ($) {
     # which means ignore <ident> except for logging purposes etc.
     # and use <hostspec>
     #
-    # <hostspec> uis <hostname> which means use that host (and all
+    # <hostspec> can be <hostname> which means use that host (and all
     # its flags and properties from the configuration and database)
+    # OR
+    # <hostspec> can be <parent_identspec>:<domname> meaning use the
+    # Xen domain name <domname> on the host specified by
+    # <parent_identspec>, which is an <identspec> as above.
 
     my $name;
     if ($ident =~ m/=/) {
@@ -832,6 +837,7 @@ sub selecthost ($) {
         Ident => $ident,
         Name => $name,
         TcpCheckPort => 22,
+        NestingLevel => 0,
         Info => [],
     };
     if (defined $job) {
@@ -839,6 +845,35 @@ sub selecthost ($) {
 					  $c{DebianSuite});
     }
 
+    #----- handle hosts which are themselves guests (nested) -----
+
+    if ($name =~ s/^(.*)://) {
+	my $parentname = $1;
+	my $parent = selecthost($parentname);
+	my $child = selectguest($name,$parent);
+	$child->{Ident} = $ho->{Ident};
+	$child->{Info} = [ "in", $parent->{Name}, @{ $parent->{Info} } ];
+	$child->{NestingLevel} = $parent->{NestingLevel}+1;
+
+	# $child->{Power} = 'guest';   todo
+	power_cycle_host_setup($child);
+
+	$child->{Properties}{Serial} = 'noop'; # todo
+	serial_host_setup($child);
+
+	my $msg = "L$child->{NestingLevel} host $child->{Ident}:";
+	$msg .= " guest $child->{Guest} (@{ $child->{Info} })";
+	$msg .= " $child->{Ether}";
+
+	my $err = guest_check_ip($child);
+	$msg .= " ".(defined $err ? "<no-ip> $err" : $child->{Ip});
+
+	logm($msg);
+
+	# all the rest of selecthost is wrong for this case
+	return $child;
+    }
+
     #----- calculation of the host's properties -----
 
     $ho->{Properties} = { };
Signature · 1 line
1.7.10.4
Acked-by on v1 13/26
#71 lore raw

Re: [OSSTEST PATCH 13/26] selecthost: Support nested hosts (guests which are also hosts)

Quoted 16 lines · Ian Jackson

We introduce a new syntax: instead of a hostname (which might appear in a command line argument to a ts-* script and hence be passed to selecthost, or which might be in a runvar), we now support <hostspec>:<domname>.

Such `hosts' (let us refer to such a thing as an L1, although in principle further nesting may be possible) are expected to be dynamically created. So they do not have flags and properties in the configuration (or in an Executive instance's database).

The IP address is determined dynamically from the leases file. If the L1 is not running, then no IP address may be found. This is not an error. Users of this facility will need to make sure that ts-* scripts which are unaware of the L1's special status are only invoked when it is known that the L1 is up and has obtained its IP address.

Quoted 7 lines

`Power cycling' the L1 will be done by VM control operations in the L0; this will come in a subsequent patch.

`Serial access' to the L1 guest will likewise need to be done via the console arrangements in L0.

#72 lore raw

PATCH 14/26 Nested hosts: Provide PDU power method

In reply to Ian Jackson

From: Robert Ho <[email protected]>

This `guest' power method uses VM create/destroy. It is automatically used for nested hosts. It would not make much sense to configure it manually.

For nested host/guest, its power on/off method shall be its host invoke $(toolstack)->create/destroy method.

XXX Missing Signed-off-by from Robert Hu

---
v14: Mostly rewritten by iwj

---
 Osstest/PDU/guest.pm   |   59 ++++++++++++++++++++++++++++++++++++++++++++++++
 Osstest/TestSupport.pm |    2 +-
 2 files changed, 60 insertions(+), 1 deletion(-)
 create mode 100755 Osstest/PDU/guest.pm
Osstest/PDU/guest.pm new file +59 -0
new file mode 100755
index 0000000..b6bf9a1
@@ -0,0 +1,59 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Intel.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+package Osstest::PDU::guest;
+
+use strict;
+use warnings;
+use Switch;
+
+use Osstest;
+use Osstest::TestSupport;
+use IO::File;
+
+BEGIN {
+    use Exporter ();
+    our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+    $VERSION     = 1.00;
+    @ISA         = qw(Exporter);
+    @EXPORT      = qw();
+    %EXPORT_TAGS = ( );
+
+    @EXPORT_OK   = qw();
+}
+
+sub new {
+    my ($class, $ho) = @_;
+    return bless { Target => $ho }, $class;
+}
+
+sub pdu_power_state {
+    my ($mo, $on) = @_;
+
+    my $child = $mo->{Target};
+    my $parent = $child->{Host};
+    die "$child->{Name} ?" unless $parent;
+
+    logm("power $child->{Name} nested on $parent->{Name} ".($on+0));
+    if ($on) {
+	toolstack($parent)->create($child);
+    } else {
+	toolstack($parent)->destroy($child);
+    }
+}
+
+1;
Osstest/TestSupport.pm +1 -1
index 9b10602..3025c9f 100644
@@ -855,7 +855,7 @@ sub selecthost ($) {
 	$child->{Info} = [ "in", $parent->{Name}, @{ $parent->{Info} } ];
 	$child->{NestingLevel} = $parent->{NestingLevel}+1;
 
-	# $child->{Power} = 'guest';   todo
+	$child->{Power} = 'guest';
 	power_cycle_host_setup($child);
 
 	$child->{Properties}{Serial} = 'noop'; # todo
Signature · 1 line
1.7.10.4
Acked-by on v1 14/26
#73 lore raw

Re: [OSSTEST PATCH 14/26] Nested hosts: Provide PDU power method

Quoted 11 lines · Ian Jackson

From: Robert Ho <[email protected]>

This `guest' power method uses VM create/destroy. It is automatically used for nested hosts. It would not make much sense to configure it manually.

For nested host/guest, its power on/off method shall be its host invoke $(toolstack)->create/destroy method.

XXX Missing Signed-off-by from Robert Hu

Quoted 90 lines

---
v14: Mostly rewritten by iwj

---
 Osstest/PDU/guest.pm   |   59
++++++++++++++++++++++++++++++++++++++++++++++++
 Osstest/TestSupport.pm |    2 +-
 2 files changed, 60 insertions(+), 1 deletion(-)
 create mode 100755 Osstest/PDU/guest.pm
Osstest/PDU/guest.pm new file +5 -0
new file mode 100755
index 0000000..b6bf9a1
@@ -0,0 +1,59 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Intel.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as

published by

Osstest/PDU/guest.pm new file +9 -0
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public

License

Osstest/PDU/guest.pm new file +45 -0
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+package Osstest::PDU::guest;
+
+use strict;
+use warnings;
+use Switch;
+
+use Osstest;
+use Osstest::TestSupport;
+use IO::File;
+
+BEGIN {
+    use Exporter ();
+    our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+    $VERSION     = 1.00;
+    @ISA         = qw(Exporter);
+    @EXPORT      = qw();
+    %EXPORT_TAGS = ( );
+
+    @EXPORT_OK   = qw();
+}
+
+sub new {
+    my ($class, $ho) = @_;
+    return bless { Target => $ho }, $class;
+}
+
+sub pdu_power_state {
+    my ($mo, $on) = @_;
+
+    my $child = $mo->{Target};
+    my $parent = $child->{Host};
+    die "$child->{Name} ?" unless $parent;
+
+    logm("power $child->{Name} nested on $parent->{Name} ".($on+0));
+    if ($on) {
+	toolstack($parent)->create($child);
+    } else {
+	toolstack($parent)->destroy($child);
+    }
+}
+
+1;
Osstest/TestSupport.pm +1 -1
index 9b10602..3025c9f 100644
@@ -855,7 +855,7 @@ sub selecthost ($) {
 	$child->{Info} = [ "in", $parent->{Name}, @{ $parent->{Info} }
 ];
 	$child->{NestingLevel} = $parent->{NestingLevel}+1;
 
-	# $child->{Power} = 'guest';   todo
+	$child->{Power} = 'guest';
 	power_cycle_host_setup($child);
 

$child->{Properties}{Serial} = 'noop'; # todo

author Tested-by
#74 lore raw

Re: [OSSTEST PATCH 14/26] Nested hosts: Provide PDU power method

Quoted 16 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Monday, September 28, 2015 6:11 PM
To: Ian Jackson <[email protected]>; [email protected]
Cc: Hu, Robert <[email protected]>
Subject: Re: [OSSTEST PATCH 14/26] Nested hosts: Provide PDU power method

Quoted 6 lines · Ian Jackson

From: Robert Ho <[email protected]>

This `guest' power method uses VM create/destroy. It is automatically used for nested hosts. It would not make much sense to configure it manually.

Quoted 7 lines
Quoted 5 lines

For nested host/guest, its power on/off method shall be its host invoke $(toolstack)->create/destroy method.

XXX Missing Signed-off-by from Robert Hu

[Hu, Robert]

Quoted 92 lines
Quoted 59 lines

---
v14: Mostly rewritten by iwj

---
 Osstest/PDU/guest.pm   |   59
++++++++++++++++++++++++++++++++++++++++++++++++
 Osstest/TestSupport.pm |    2 +-
 2 files changed, 60 insertions(+), 1 deletion(-)
 create mode 100755 Osstest/PDU/guest.pm
Osstest/PDU/guest.pm new file +5 -0
new file mode 100755
index 0000000..b6bf9a1
@@ -0,0 +1,59 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Intel.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as

published by

Osstest/PDU/guest.pm new file +9 -0
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public

License

Osstest/PDU/guest.pm new file +27 -0
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+package Osstest::PDU::guest;
+
+use strict;
+use warnings;
+use Switch;
+
+use Osstest;
+use Osstest::TestSupport;
+use IO::File;
+
+BEGIN {
+    use Exporter ();
+    our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+    $VERSION     = 1.00;
+    @ISA         = qw(Exporter);
+    @EXPORT      = qw();
+    %EXPORT_TAGS = ( );
+
+    @EXPORT_OK   = qw();
+}
+
+sub new {
+    my ($class, $ho) = @_;
+    return bless { Target => $ho }, $class;

Osstest/PDU/guest.pm

Quoted 10 lines
Osstest/PDU/guest.pm new file +10 -0
+}
+
+sub pdu_power_state {
+    my ($mo, $on) = @_;
+
+    my $child = $mo->{Target};
+    my $parent = $child->{Host};
+    die "$child->{Name} ?" unless $parent;
+
+    logm("power $child->{Name} nested on $parent->{Name}

".($on+0));

Quoted 22 lines
Osstest/PDU/guest.pm new file +8 -0
+    if ($on) {
+	toolstack($parent)->create($child);
+    } else {
+	toolstack($parent)->destroy($child);
+    }
+}
+
+1;
Osstest/TestSupport.pm +1 -1
index 9b10602..3025c9f 100644
@@ -855,7 +855,7 @@ sub selecthost ($) {
 	$child->{Info} = [ "in", $parent->{Name}, @{ $parent->{Info} }
 ];
 	$child->{NestingLevel} = $parent->{NestingLevel}+1;
 
-	# $child->{Power} = 'guest';   todo
+	$child->{Power} = 'guest';
 	power_cycle_host_setup($child);
 

$child->{Properties}{Serial} = 'noop'; # todo

#75 lore raw

PATCH 15/26 DhcpWatch::leases: Fix a reporting message

In reply to Ian Jackson

This talks about `guest_check_ip', but this code is now factored out into a method. Use the correct method name in reporting.

---
v14: New patch

---
 Osstest/DhcpWatch/leases.pm |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/DhcpWatch/leases.pm +1 -1
index 9a74c40..b74ebf0 100644
@@ -171,7 +171,7 @@ sub check_ip ($$) {
     }
     $gho->{Ip}= $best->{' addr'};
 
-    report_once($gho, 'guest_check_ip', 
+    report_once($gho, 'leases::check_ip', 
 		"guest $gho->{Name}: $gho->{Ether} $gho->{Ip}");
     return undef;
 }
Signature · 1 line
1.7.10.4
author
#77 lore raw

Re: [OSSTEST PATCH 15/26] DhcpWatch::leases: Fix a reporting message

In reply to Ian Jackson
Quoted 24 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>; Ian Jackson <[email protected]>
Subject: [OSSTEST PATCH 15/26] DhcpWatch::leases: Fix a reporting message

This talks about `guest_check_ip', but this code is now factored out into a method. Use the correct method name in reporting.

---
v14: New patch

---
 Osstest/DhcpWatch/leases.pm |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/DhcpWatch/leases.pm +0 -0
index 9a74c40..b74ebf0 100644

Osstest/DhcpWatch/leases.pm sub check_ip ($$) { · line 177

Quoted 8 lines
Osstest/DhcpWatch/leases.pm +1 -1
@@ -171,7 +171,7 @@ sub check_ip ($$) {
     }
     $gho->{Ip}= $best->{' addr'};
 
-    report_once($gho, 'guest_check_ip',
+    report_once($gho, 'leases::check_ip',
 		"guest $gho->{Name}: $gho->{Ether} $gho->{Ip}");
     return undef;
 }

Ack.

Quoted 2 lines

--
1.7.10.4

author Tested-by
#78 lore raw

Re: [OSSTEST PATCH 15/26] DhcpWatch::leases: Fix a reporting message

Quoted 38 lines

-----Original Message-----
From: Hu, Robert
Sent: Monday, October 12, 2015 11:08 AM
To: Ian Jackson <[email protected]>; [email protected]
Cc: Ian Campbell <[email protected]>
Subject: RE: [OSSTEST PATCH 15/26] DhcpWatch::leases: Fix a reporting message

Quoted 8 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>; Ian Jackson <[email protected]>
Subject: [OSSTEST PATCH 15/26] DhcpWatch::leases: Fix a reporting

message

Quoted 4 lines

This talks about `guest_check_ip', but this code is now factored out into a method. Use the correct method name in reporting.

Review

Quoted 8 lines

---
v14: New patch

---
 Osstest/DhcpWatch/leases.pm |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0

b/Osstest/DhcpWatch/leases.pm

Quoted 7 lines

index 9a74c40..b74ebf0 100644

Osstest/DhcpWatch/leases.pm +0 -0
@@ -171,7 +171,7 @@ sub check_ip ($$) {
     }
     $gho->{Ip}= $best->{' addr'};
 
Quoted 6 lines

Osstest/DhcpWatch/leases.pm sub check_ip ($$) { · line 177

Quoted 5 lines
Osstest/DhcpWatch/leases.pm +1 -1
-    report_once($gho, 'guest_check_ip',
+    report_once($gho, 'leases::check_ip',
 		"guest $gho->{Name}: $gho->{Ether} $gho->{Ip}");
     return undef;
 }

Ack.

[Hu, Robert]

Quoted 2 lines
Quoted 2 lines

--
1.7.10.4

#79 lore raw

PATCH 16/26 target_check_ip: Rename and improve from guest_check_ip

In reply to Ian Jackson

Make this function suitable for running on targets with static IP addresses. (Ie, on physical hosts.) Accordingly, rename it and adjust all call sites.

---
v14: New patch

---
 Osstest/TestSupport.pm |   11 ++++++-----
 ts-guest-localmigrate  |    2 +-
 ts-guest-migrate       |    2 +-
 ts-guest-saverestore   |    2 +-
 4 files changed, 9 insertions(+), 8 deletions(-)
Osstest/TestSupport.pm +6 -5
index 3025c9f..95b6fda 100644
@@ -94,7 +94,7 @@ BEGIN {
                       prepareguest_part_lvmdisk prepareguest_part_diskimg
                       prepareguest_part_xencfg
                       guest_umount_lv guest_await guest_await_dhcp_tcp
-                      guest_checkrunning guest_check_ip guest_find_ether
+                      guest_checkrunning target_check_ip guest_find_ether
                       guest_find_domid guest_check_up guest_check_up_quick
                       guest_get_state guest_await_reboot
                       guest_await_shutdown guest_await_destroy guest_destroy
@@ -724,8 +724,9 @@ sub dhcp_watch_setup ($$) {
     $gho->{DhcpWatch} = get_host_method_object($ho, 'DhcpWatch', $meth);
 }
 
-sub guest_check_ip ($) {
-    my ($gho) = @_;
+sub target_check_ip ($) {
+    my ($gho) = @_; # returns error message or undef
+    return undef if $gho->{IpStatic};
     guest_find_ether($gho);
     $gho->{DhcpWatch}->check_ip($gho);
 }
@@ -865,7 +866,7 @@ sub selecthost ($) {
 	$msg .= " guest $child->{Guest} (@{ $child->{Info} })";
 	$msg .= " $child->{Ether}";
 
-	my $err = guest_check_ip($child);
+	my $err = target_check_ip($child);
 	$msg .= " ".(defined $err ? "<no-ip> $err" : $child->{Ip});
 
 	logm($msg);
@@ -1937,7 +1938,7 @@ sub guest_await_dhcp_tcp ($$) {
 	      " $gho->{TcpCheckPort}".
               " link/ip/tcp",
               sub {
-        my $err= guest_check_ip($gho);
+        my $err= target_check_ip($gho);
         return $err if defined $err;
 
         return
ts-guest-localmigrate +1 -1
index 8fe986d..85a0887 100755
@@ -40,7 +40,7 @@ sub migrate () {
 guest_await_dhcp_tcp($gho, 5);
 guest_check_up($gho);
 
-my $err= guest_check_ip($gho);  die "$err $gho->{Name}" if defined $err;
+my $err= target_check_ip($gho);  die "$err $gho->{Name}" if defined $err;
 
 for (my $rep=1; $rep<=$reps; $rep++) {
     logm("========== rep $rep ==========");
ts-guest-migrate +1 -1
index b77d0de..505fab2 100755
@@ -30,7 +30,7 @@ our $gho = selectguest($ARGV[2],$sho);
 
 sub migrate () {
     guest_checkrunning($sho,$gho) or die $gho->{Name};
-    my $err= guest_check_ip($gho);  die "$err $gho->{Name}" if defined $err;
+    my $err= target_check_ip($gho);  die "$err $gho->{Name}" if defined $err;
     toolstack($sho)->migrate($gho, $dho, $timeout{Migrate});
 }
 
ts-guest-saverestore +1 -1
index 7e13d90..73883ef 100755
@@ -26,7 +26,7 @@ our ($ho,$gho) = ts_get_host_guest(@ARGV);
 
 sub save () {
     guest_checkrunning($ho,$gho) or die $gho->{Name};
-    my $err= guest_check_ip($gho);  die "$err $gho->{Name}" if defined $err;
+    my $err= target_check_ip($gho);  die "$err $gho->{Name}" if defined $err;
     toolstack($ho)->save($gho,"image",200);
     target_ping_check_down($gho);
 }
Signature · 1 line
1.7.10.4
author Tested-by
#81 lore raw

Re: [OSSTEST PATCH 16/26] target_check_ip: Rename and improve from guest_check_ip

Quoted 9 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Monday, September 28, 2015 6:15 PM
To: Ian Jackson <[email protected]>; [email protected]
Cc: Hu, Robert <[email protected]>
Subject: Re: [OSSTEST PATCH 16/26] target_check_ip: Rename and improve from guest_check_ip

Quoted 8 lines
Quoted 5 lines · Ian Jackson

Make this function suitable for running on targets with static IP addresses. (Ie, on physical hosts.) Accordingly, rename it and adjust all call sites.

[Hu, Robert]

#82 lore raw

PATCH 17/26 await_tcp(): Run check_ip on each loop iteration

In reply to Ian Jackson

From: Robert Ho <[email protected]>

await_tcp is often invoked after a reboot.

In this situation the target's IP address may change. If this happens while await_tcp is running, we would continue to poll the old IP address. Fix this by running target_check_ip on each iteration.

---
v14: Dropped change to selecthost, which was in code which is no
longer present in this version of the series.
Rewritten to use target_check_ip.
Dropped IMO-unnecessary comment.

---
 Osstest/TestSupport.pm |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
Osstest/TestSupport.pm +3 -2
index 95b6fda..ad017a4 100644
@@ -2010,9 +2010,10 @@ sub await_tcp ($$$) {
     my ($maxwait,$interval,$ho) = @_;
     target_adjust_timeout($ho,\$maxwait);
     poll_loop($maxwait,$interval,
-              "await tcp $ho->{Name} $ho->{TcpCheckPort}",
+              "await tcp $ho->{Name} $ho->{Ip} $ho->{TcpCheckPort}",
               sub {
-        return target_tcp_check($ho,$interval);
+	return target_check_ip($ho) //
+               target_tcp_check($ho,$interval);
     });
 }
 
Signature · 1 line
1.7.10.4
Acked-by on v1 17/26
#83 lore raw

Re: [OSSTEST PATCH 17/26] await_tcp(): Run check_ip on each loop iteration

Quoted 10 lines · Ian Jackson

From: Robert Ho <[email protected]>

await_tcp is often invoked after a reboot.

In this situation the target's IP address may change. If this happens while await_tcp is running, we would continue to poll the old IP address. Fix this by running target_check_ip on each iteration.

author Tested-by
#84 lore raw

Re: [OSSTEST PATCH 17/26] await_tcp(): Run check_ip on each loop iteration

Quoted 14 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Monday, September 28, 2015 6:16 PM
To: Ian Jackson <[email protected]>; [email protected]
Cc: Hu, Robert <[email protected]>
Subject: Re: [OSSTEST PATCH 17/26] await_tcp(): Run check_ip on each loop iteration

Quoted 4 lines · Ian Jackson

From: Robert Ho <[email protected]>

await_tcp is often invoked after a reboot.

Quoted 8 lines
Quoted 6 lines

In this situation the target's IP address may change. If this happens while await_tcp is running, we would continue to poll the old IP address. Fix this by running target_check_ip on each iteration.

[Hu, Robert]

#85 lore raw

PATCH 18/26 LVM: Break out lv_create

In reply to Ian Jackson

We are going to want to reuse this.

---
v14: New patch

---
 Osstest/TestSupport.pm |   15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
Osstest/TestSupport.pm +11 -4
index ad017a4..2d1db5d 100644
@@ -62,7 +62,7 @@ BEGIN {
                       target_install_packages target_install_packages_norec
                       target_jobdir target_extract_jobdistpath_subdir
                       target_extract_jobdistpath
-                      lv_dev_mapper target_guest_lv_name
+                      lv_create lv_dev_mapper target_guest_lv_name
 
                       poll_loop tcpconnect await_tcp
                       contents_make_cpio file_simple_write_contents
@@ -702,6 +702,15 @@ sub poll_loop ($$$&) {
     logm("$what: ok. (${waited}s)");
 }
 
+sub lv_create ($$$) {
+    my ($ho, $vg, $lv, $mb) = @_;
+    my $lvdev = "/dev/$lv/$vg";
+    target_cmd_root($ho, "lvremove -f $lvdev ||:");
+    target_cmd_root($ho, "lvcreate -L ${mb}M -n $lv $vg");
+    target_cmd_root($ho, "dd if=/dev/zero of=$lvdev count=10");
+    return $lvdev;
+}
+
 sub lv_dev_mapper ($$) {
     my ($vg,$lv) = @_;
     $vg =~ s/-/--/g;
@@ -1685,9 +1694,7 @@ sub prepareguest ($$$$$$) {
 
 sub prepareguest_part_lvmdisk ($$$) {
     my ($ho, $gho, $disk_mb) = @_;
-    target_cmd_root($ho, "lvremove -f $gho->{Lvdev} ||:");
-    target_cmd_root($ho, "lvcreate -L ${disk_mb}M -n $gho->{Lv} $gho->{Vg}");
-    target_cmd_root($ho, "dd if=/dev/zero of=$gho->{Lvdev} count=10");
+    lvm_lv_create($ho, $gho->{Vg}, $gho->{Lv}, $disk_mb);
 }
 
 sub make_vhd ($$$) {
Signature · 1 line
1.7.10.4
author
#87 lore raw

Re: [OSSTEST PATCH 18/26] LVM: Break out lv_create

In reply to Ian Jackson
Quoted 27 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>; Ian Jackson <[email protected]>
Subject: [OSSTEST PATCH 18/26] LVM: Break out lv_create

We are going to want to reuse this.

---
v14: New patch

---
 Osstest/TestSupport.pm |   15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
Osstest/TestSupport.pm +0 -0
index ad017a4..2d1db5d 100644
@@ -62,7 +62,7 @@ BEGIN {
                       target_install_packages

target_install_packages_norec
target_jobdir target_extract_jobdistpath_subdir target_extract_jobdistpath

Osstest/TestSupport.pm BEGIN { · line 65

Quoted 2 lines
Osstest/TestSupport.pm +1 -1
-                      lv_dev_mapper target_guest_lv_name
+                      lv_create lv_dev_mapper target_guest_lv_name

'target_guest_lv_name' seems lacking in my work directory. Would you double check if it is in production tree? I checked, seems not.

Quoted 34 lines

poll_loop tcpconnect await_tcp contents_make_cpio
file_simple_write_contents

Osstest/TestSupport.pm +10 -3
@@ -702,6 +702,15 @@ sub poll_loop ($$$&) {
     logm("$what: ok. (${waited}s)");
 }
 
+sub lv_create ($$$) {
+    my ($ho, $vg, $lv, $mb) = @_;
+    my $lvdev = "/dev/$lv/$vg";
+    target_cmd_root($ho, "lvremove -f $lvdev ||:");
+    target_cmd_root($ho, "lvcreate -L ${mb}M -n $lv $vg");
+    target_cmd_root($ho, "dd if=/dev/zero of=$lvdev count=10");
+    return $lvdev;
+}
+
 sub lv_dev_mapper ($$) {
     my ($vg,$lv) = @_;
     $vg =~ s/-/--/g;
@@ -1685,9 +1694,7 @@ sub prepareguest ($$$$$$) {
 
 sub prepareguest_part_lvmdisk ($$$) {
     my ($ho, $gho, $disk_mb) = @_;
-    target_cmd_root($ho, "lvremove -f $gho->{Lvdev} ||:");
-    target_cmd_root($ho, "lvcreate -L ${disk_mb}M -n $gho->{Lv}
 $gho->{Vg}");
-    target_cmd_root($ho, "dd if=/dev/zero of=$gho->{Lvdev} count=10");
+    lvm_lv_create($ho, $gho->{Vg}, $gho->{Lv}, $disk_mb);
 }
 

sub make_vhd ($$$) {
--
1.7.10.4

#88 lore raw

Re: [OSSTEST PATCH 18/26] LVM: Break out lv_create

Quoted 18 lines · Robert
Quoted 18 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>; Ian Jackson <[email protected]>
Subject: [OSSTEST PATCH 18/26] LVM: Break out lv_create

We are going to want to reuse this.

---
v14: New patch

---
 Osstest/TestSupport.pm |   15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
Quoted 13 lines

Osstest/TestSupport.pm BEGIN { · line 65

Quoted 11 lines
Osstest/TestSupport.pm +0 -0
index ad017a4..2d1db5d 100644
@@ -62,7 +62,7 @@ BEGIN {
                       target_install_packages

target_install_packages_norec
target_jobdir target_extract_jobdistpath_subdir target_extract_jobdistpath

Osstest/TestSupport.pm +1 -1
-                      lv_dev_mapper target_guest_lv_name
+                      lv_create lv_dev_mapper target_guest_lv_name

'target_guest_lv_name' seems lacking in my work directory. Would you double check if it is in production tree? I checked, seems not.

It was dropped from the production version over the weekend due to my patch from [0] landing.

The previous production was 24d79ce9cbde907c7907b3d24bd96d9642953d40, so if you are unable to resolve the conflict yourself this might be a better baseline until Ian can rebase.

Ian.

[0]
http://lists.xen.org/archives/html/xen-devel/2015-10/msg00596.html

Quoted 35 lines
Quoted 34 lines

poll_loop tcpconnect await_tcp contents_make_cpio
file_simple_write_contents

Osstest/TestSupport.pm +10 -3
@@ -702,6 +702,15 @@ sub poll_loop ($$$&) {
     logm("$what: ok. (${waited}s)");
 }
 
+sub lv_create ($$$) {
+    my ($ho, $vg, $lv, $mb) = @_;
+    my $lvdev = "/dev/$lv/$vg";
+    target_cmd_root($ho, "lvremove -f $lvdev ||:");
+    target_cmd_root($ho, "lvcreate -L ${mb}M -n $lv $vg");
+    target_cmd_root($ho, "dd if=/dev/zero of=$lvdev count=10");
+    return $lvdev;
+}
+
 sub lv_dev_mapper ($$) {
     my ($vg,$lv) = @_;
     $vg =~ s/-/--/g;
@@ -1685,9 +1694,7 @@ sub prepareguest ($$$$$$) {
 
 sub prepareguest_part_lvmdisk ($$$) {
     my ($ho, $gho, $disk_mb) = @_;
-    target_cmd_root($ho, "lvremove -f $gho->{Lvdev} ||:");
-    target_cmd_root($ho, "lvcreate -L ${disk_mb}M -n $gho->{Lv}
 $gho->{Vg}");
-    target_cmd_root($ho, "dd if=/dev/zero of=$gho->{Lvdev} count=10");
+    lvm_lv_create($ho, $gho->{Vg}, $gho->{Lv}, $disk_mb);
 }
 

sub make_vhd ($$$) {
--
1.7.10.4

author
#89 lore raw

Re: [OSSTEST PATCH 18/26] LVM: Break out lv_create

In reply to Ian Jackson
Quoted 50 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>; Ian Jackson <[email protected]>
Subject: [OSSTEST PATCH 18/26] LVM: Break out lv_create

We are going to want to reuse this.

---
v14: New patch

---
 Osstest/TestSupport.pm |   15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
Osstest/TestSupport.pm +0 -0
index ad017a4..2d1db5d 100644
@@ -62,7 +62,7 @@ BEGIN {
                       target_install_packages

target_install_packages_norec
target_jobdir target_extract_jobdistpath_subdir target_extract_jobdistpath

Osstest/TestSupport.pm +1 -1
-                      lv_dev_mapper target_guest_lv_name
+                      lv_create lv_dev_mapper target_guest_lv_name
 
                       poll_loop tcpconnect await_tcp
                       contents_make_cpio

file_simple_write_contents

Osstest/TestSupport.pm +9 -0
@@ -702,6 +702,15 @@ sub poll_loop ($$$&) {
     logm("$what: ok. (${waited}s)");
 }
 
+sub lv_create ($$$) {
+    my ($ho, $vg, $lv, $mb) = @_;
+    my $lvdev = "/dev/$lv/$vg";
+    target_cmd_root($ho, "lvremove -f $lvdev ||:");
+    target_cmd_root($ho, "lvcreate -L ${mb}M -n $lv $vg");
+    target_cmd_root($ho, "dd if=/dev/zero of=$lvdev count=10");
+    return $lvdev;
+}
+
 sub lv_dev_mapper ($$) {
     my ($vg,$lv) = @_;
     $vg =~ s/-/--/g;

Osstest/TestSupport.pm sub prepareguest ($$$$$$) { · line 1698

Quoted 8 lines
Osstest/TestSupport.pm +1 -3
@@ -1685,9 +1694,7 @@ sub prepareguest ($$$$$$) {
 
 sub prepareguest_part_lvmdisk ($$$) {
     my ($ho, $gho, $disk_mb) = @_;
-    target_cmd_root($ho, "lvremove -f $gho->{Lvdev} ||:");
-    target_cmd_root($ho, "lvcreate -L ${disk_mb}M -n $gho->{Lv}
 $gho->{Vg}");
-    target_cmd_root($ho, "dd if=/dev/zero of=$gho->{Lvdev} count=10");
+    lvm_lv_create($ho, $gho->{Vg}, $gho->{Lv}, $disk_mb);

And here I guess shall be lv_create($ho, $gho->{Vg}, $gho->{Lv}, $disk_mb)?

Quoted 5 lines
Osstest/TestSupport.pm +0 -0
 }
 

sub make_vhd ($$$) {
--
1.7.10.4

#90 lore raw

PATCH 19/26 Toolstack::xl: Provide block_attach method

In reply to Ian Jackson

It is possible that this may work some of the time with xm, so I have taken no measures to prevent it running then.

----
v14: New patch

---
 Osstest/Toolstack/xl.pm |    8 ++++++++
 1 file changed, 8 insertions(+)
Osstest/Toolstack/xl.pm +8 -0
index 0f8abed..cc26d61 100644
@@ -109,4 +109,12 @@ sub restore ($$$$) {
 		    ." $f", $timeout);
 }
 
+sub block_attach ($$$$) {
+    my ($self,$gho,$xldiskspec) = @_;
+    die "quotes in $xldiskspec ?" if $xldiskspec =~ m/'/;
+    my $gn = $gho->{Name};
+    my $cmd = $self->{_VerboseCommand}." block-attach $gn '$xldiskspec'";
+    target_cmd_root($ho, $cmd, 100);
+}
+
 1;
Signature · 1 line
1.7.10.4
author
#92 lore raw

Re: [OSSTEST PATCH 19/26] Toolstack::xl: Provide block_attach method

In reply to Ian Jackson
Quoted 24 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>; Ian Jackson <[email protected]>
Subject: [OSSTEST PATCH 19/26] Toolstack::xl: Provide block_attach method

It is possible that this may work some of the time with xm, so I have taken no measures to prevent it running then.

----
v14: New patch

---
 Osstest/Toolstack/xl.pm |    8 ++++++++
 1 file changed, 8 insertions(+)
Osstest/Toolstack/xl.pm +0 -0
index 0f8abed..cc26d61 100644

Osstest/Toolstack/xl.pm sub restore ($$$$) { · line 117

Quoted 9 lines
Osstest/Toolstack/xl.pm +5 -0
@@ -109,4 +109,12 @@ sub restore ($$$$) {
 		    ." $f", $timeout);
 }
 
+sub block_attach ($$$$) {
+    my ($self,$gho,$xldiskspec) = @_;
+    die "quotes in $xldiskspec ?" if $xldiskspec =~ m/'/;
+    my $gn = $gho->{Name};
+    my $cmd = $self->{_VerboseCommand}." block-attach $gn
 '$xldiskspec'";

I guess here lacks of a

	my $ho = $self->{Host};
Quoted 6 lines
Osstest/Toolstack/xl.pm +3 -0
+    target_cmd_root($ho, $cmd, 100);
+}
+

1;
--
1.7.10.4

#93 lore raw

Re: [OSSTEST PATCH 19/26] Toolstack::xl: Provide block_attach method

Hu, Robert writes ("RE: [OSSTEST PATCH 19/26] Toolstack::xl: Provide block_attach method"):

Quoted 11 lines
Quoted 8 lines

From: Ian Jackson [mailto:[email protected]]

Osstest/Toolstack/xl.pm +5 -0
+sub block_attach ($$$$) {
+    my ($self,$gho,$xldiskspec) = @_;
+    die "quotes in $xldiskspec ?" if $xldiskspec =~ m/'/;
+    my $gn = $gho->{Name};
+    my $cmd = $self->{_VerboseCommand}." block-attach $gn

'$xldiskspec'";

I guess here lacks of a

	my $ho = $self->{Host};

Yes, it does seem so! Thanks.

Ian.

#94 lore raw

PATCH 20/26 sg-run-job: Break out per-host-prep and per-host-finish

In reply to Ian Jackson

No functional change.

We now call the per-host-ts finish steps unconditionally, rather than only if !$need_build_host, per-host-ts is (complicated) no-op if $need_build_host, since in that case $need_xen_hosts is {}.

Tested by: Robert Ho <[email protected]>
---
v14: Squash typo fix from Robert into this patch

---
 sg-run-job |   31 +++++++++++++++++++------------
 1 file changed, 19 insertions(+), 12 deletions(-)
sg-run-job +19 -12
index 66145b8..884a21d 100755
@@ -23,6 +23,20 @@ source ./tcl/osstestlib.tcl
 readconfig
 source-method JobDB
 
+proc per-host-prep {} {
+    per-host-ts .       host-ping-check-native/@ ts-host-ping-check
+    per-host-ts .       xen-install/@     ts-xen-install
+    per-host-ts .       xen-boot/@        ts-host-reboot
+
+    per-host-ts .       host-ping-check-xen/@ ts-host-ping-check
+    per-host-ts .       =(*)             {ts-leak-check basis}
+}
+
+proc per-host-finish {} {
+    per-host-ts .       =                {ts-leak-check check}
+    per-host-ts !broken capture-logs/@(*) ts-logs-capture
+}
+
 proc run-job {job} {
     global jobinfo builds flight ok need_xen_hosts anyfailed
 
@@ -51,22 +65,15 @@ proc run-job {job} {
     if {$ok} { setstatus running                                          }
 
     per-host-ts broken  host-install/@(*) ts-host-install-twice
-    per-host-ts .       host-ping-check-native/@ ts-host-ping-check
-    per-host-ts .       xen-install/@     ts-xen-install
-    per-host-ts .       xen-boot/@        ts-host-reboot
-    per-host-ts .       host-ping-check-xen/@ ts-host-ping-check
 
-    per-host-ts .       =(*)             {ts-leak-check basis}
+    per-host-prep
 
     if {$ok} { catching-otherwise fail      run-job/$jobinfo(recipe)      }
-    per-host-ts .       =                {ts-leak-check check}
 
-    if {!$need_build_host} {
-        per-host-ts !broken capture-logs/@(*) ts-logs-capture
-    } else {
-        if {$anyfailed} {
-            run-ts  !broken capture-logs      ts-logs-capture + host
-        }
+    per-host-finish
+
+    if {$need_build_host && $anyfailed} {
+	run-ts  !broken capture-logs      ts-logs-capture + host
     }
 
     if {$ok} { setstatus pass                                             }
Signature · 1 line
1.7.10.4
Acked-by on v1 20/26
#95 lore raw

Re: [OSSTEST PATCH 20/26] sg-run-job: Break out per-host-prep and per-host-finish

Quoted 9 lines · Ian Jackson

No functional change.

We now call the per-host-ts finish steps unconditionally, rather than only if !$need_build_host, per-host-ts is (complicated) no-op if $need_build_host, since in that case $need_xen_hosts is {}.

Tested by: Robert Ho <[email protected]>

author Tested-by
#96 lore raw

Re: [OSSTEST PATCH 20/26] sg-run-job: Break out per-host-prep and per-host-finish

Quoted 16 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Monday, September 28, 2015 6:19 PM
To: Ian Jackson <[email protected]>; [email protected]
Cc: Hu, Robert <[email protected]>
Subject: Re: [OSSTEST PATCH 20/26] sg-run-job: Break out per-host-prep and per-host-finish

Quoted 6 lines · Ian Jackson

No functional change.

We now call the per-host-ts finish steps unconditionally, rather than only if !$need_build_host, per-host-ts is (complicated) no-op if $need_build_host, since in that case $need_xen_hosts is {}.

Quoted 5 lines
Quoted 3 lines

Tested by: Robert Ho <[email protected]>

[Hu, Robert]

#97 lore raw

PATCH 21/26 sg-run-job: Provide infrastructure for layers of nesting

In reply to Ian Jackson

Provides nested-layer-descend, which can be called in an individual test job at the appropriate point (after the L1 has been set up).

The inner host is a guest of the outer host; powering it off means destroying it. Putting the poweroff at this point in the loop, rather than in per-host-finish, avoids powering off physical servers. The use of .' rather than !.' for iffail means we do not power off after failures (as we might want to preserve the state for debugging etc).

---
v14: Squash syntax fix from Robert Ho into this patch

---
 sg-run-job |   21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
sg-run-job +20 -1
index 884a21d..8174ef7 100755
@@ -39,6 +39,7 @@ proc per-host-finish {} {
 
 proc run-job {job} {
     global jobinfo builds flight ok need_xen_hosts anyfailed
+    global nested_layers_hosts
 
     set ok 1
     set anyfailed 0
@@ -52,6 +53,7 @@ proc run-job {job} {
         set need_xen_hosts $nh
         set need_build_host 0
     }
+    set nested_layers_hosts {}
 
     catching-otherwise blocked            check-not-blocked
     if {!$ok} return
@@ -70,7 +72,15 @@ proc run-job {job} {
 
     if {$ok} { catching-otherwise fail      run-job/$jobinfo(recipe)      }
 
-    per-host-finish
+    while 1 {
+	per-host-finish
+	
+	if {![llength $nested_layers_hosts]} break
+
+	per-host-ts . = final-poweroff {ts-host-powercycle --power=0}
+
+        set need_xen_hosts [lunappend nested_layers_hosts]
+    }
 
     if {$need_build_host && $anyfailed} {
 	run-ts  !broken capture-logs      ts-logs-capture + host
@@ -247,6 +257,15 @@ proc per-host-ts {iffail ident script args} {
     }
 }
 
+proc nested-layer-descend {nested_hosts} {
+    # We save need_xen_hosts on a stack in nested_layers_hosts
+    # It gets popped again during the cleanup part of run-job
+    global nested_layers_hosts need_xen_hosts
+    lappend nested_layers_hosts $need_xen_hosts
+    set need_xen_hosts $nested_hosts
+    per-host-prep
+}
+
 #---------- test recipes ----------
 
 proc need-hosts/test-debian-nomigr {} { return host }
Signature · 1 line
1.7.10.4
Acked-by on v1 21/26
#98 lore raw

Re: [OSSTEST PATCH 21/26] sg-run-job: Provide infrastructure for layers of nesting

Quoted 10 lines · Ian Jackson

Provides nested-layer-descend, which can be called in an individual test job at the appropriate point (after the L1 has been set up).

The inner host is a guest of the outer host; powering it off means destroying it. Putting the poweroff at this point in the loop, rather than in per-host-finish, avoids powering off physical servers. The use of .' rather than !.' for iffail means we do not power off after failures (as we might want to preserve the state for debugging etc).

Quoted 3 lines
author
#99 lore raw

Re: [OSSTEST PATCH 21/26] sg-run-job: Provide infrastructure for layers of nesting

In reply to Ian Jackson
Quoted 51 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>; Ian Jackson <[email protected]>
Subject: [OSSTEST PATCH 21/26] sg-run-job: Provide infrastructure for layers of nesting

Provides nested-layer-descend, which can be called in an individual test job at the appropriate point (after the L1 has been set up).

The inner host is a guest of the outer host; powering it off means destroying it. Putting the poweroff at this point in the loop, rather than in per-host-finish, avoids powering off physical servers. The use of .' rather than !.' for iffail means we do not power off after failures (as we might want to preserve the state for debugging etc).

---
v14: Squash syntax fix from Robert Ho into this patch

---
 sg-run-job |   21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
sg-run-job +2 -0
index 884a21d..8174ef7 100755
@@ -39,6 +39,7 @@ proc per-host-finish {} {
 
 proc run-job {job} {
     global jobinfo builds flight ok need_xen_hosts anyfailed
+    global nested_layers_hosts
 
     set ok 1
     set anyfailed 0
@@ -52,6 +53,7 @@ proc run-job {job} {
         set need_xen_hosts $nh
         set need_build_host 0
     }
+    set nested_layers_hosts {}
 
     catching-otherwise blocked            check-not-blocked
     if {!$ok} return
@@ -70,7 +72,15 @@ proc run-job {job} {
 

sg-run-job proc run-job {job} { · line 81

Quoted 10 lines
sg-run-job +6 -1
     if {$ok} { catching-otherwise fail
 run-job/$jobinfo(recipe)      }
 
-    per-host-finish
+    while 1 {
+	per-host-finish
+
+	if {![llength $nested_layers_hosts]} break
+
+	per-host-ts . = final-poweroff {ts-host-powercycle --power=0}

[Hu, Robert]

I guess here is a unnecessary '='. Just guess, I don't manage to comprehend the Tcl part.

Quoted 24 lines
sg-run-job +3 -0
+
+        set need_xen_hosts [lunappend nested_layers_hosts]
+    }
 
     if {$need_build_host && $anyfailed} {
sg-run-job +9 -1
 	run-ts  !broken capture-logs      ts-logs-capture + host
@@ -247,6 +257,15 @@ proc per-host-ts {iffail ident script args} {
     }
 }
 
+proc nested-layer-descend {nested_hosts} {
+    # We save need_xen_hosts on a stack in nested_layers_hosts
+    # It gets popped again during the cleanup part of run-job
+    global nested_layers_hosts need_xen_hosts
+    lappend nested_layers_hosts $need_xen_hosts
+    set need_xen_hosts $nested_hosts
+    per-host-prep
+}
+
 #---------- test recipes ----------
 
 proc need-hosts/test-debian-nomigr {} { return host }
--

1.7.10.4

#100 lore raw

PATCH 22/26 Nested HVM: Provide ts-nested-setup to help make L1 usable as a host

In reply to Ian Jackson

From: Robert Ho <[email protected]>

* Provide the L1 with some storage for its own guests' disks
* Install some packages in the L1
* Optionally, set a runvar defining the L1 for the rest of the job

The recipe is going to run ts-xen-install etc.

---
v14: Use target_check_ip (renamed, earlier, in a new patch)
ts-nested-setup now has a default gueststorage_size of 20G
and this is implicitly used by make-flight.
Adjusted for new selecthost nested host syntax and correspondingly
completely changed invocation syntax.
Only optionally sets the runvar, if you pass --define. (This
will make it easier to play around with interactively.)
Broken the pieces of work out into subroutines for clarity. Comment about guest storage slightly edited and rewrapped. Guest storage runvars and perl variable names etc. renamed to
gueststorage' rather than guest_storage'.
LVM and VG names and perl variable names changed to be clearer. Install `ed' too.
Use lv_create and toolstack()->block_attach.
Dropped ack from Ian Campbell.

---
 ts-nested-setup |   99 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 99 insertions(+)
 create mode 100755 ts-nested-setup
ts-nested-setup new file +99 -0
new file mode 100755
index 0000000..f7e0ebd
@@ -0,0 +1,99 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Intel Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our $dodefine;
+if (@ARGV && $ARGV[0] eq '--define') { $dodefine=1; shift @ARGV; }
+
+our ($l1_identspec) = @ARGV;
+
+my $l1 = selecthost($l1_identspec);
+my $l0 = $l1->{Host};
+
+die unless $l0;
+
+our $def_gueststorage_size = 20000; # Mby
+
+sub packages () {
+    # Faster to install packages while L1 is still running PVHVM
+    target_install_packages_norec($l1, qw(lvm2 rsync ed genisoimage));
+}
+
+sub guest_storage () {
+    # We need to attach an extra disk to the L1 guest to be used as L2
+    # guest storage:
+    #
+    # When running in a nested HVM environment the L1 domain is acting
+    # as both a guest to L0 and a host to L2 guests and therefore
+    # potentially sees connections to two independent xenstore
+    # instances, one provided by the L0 host and one which is provided
+    # by the L1 instance of xenstore.
+    #
+    # Unfortunately the kernel is not capable of dealing with this and
+    # is only able to cope with a single xenstore connection. Since
+    # the L1 toolstack and L2 guests absolutely require xenstore to
+    # function we therefore cannot use the L0 xenstore and therefore
+    # cannot use PV devices (xvdX etc) in the L1 guest and must use
+    # emulated devices (sdX etc).
+    #
+    # However at the moment we have not yet rebooted L1 into Xen and
+    # so it does have PV devices available and sdb actually appears as
+    # xvdb.  We could disable the Xen platform device and use emulated
+    # devices for the install phase too but that would be needlessly
+    # slow.
+    # 
+    # Instead, to avoid needing to even mention the name of xvdb or
+    # sdb, we create the vg in l0.  When the l1 reboots it will
+    # automatically find the empty vg we have created for it, and
+    # target_choose_vg on l1 (which is used by all the guest creation
+    # ts-* scripts) will use it since it has plenty of space.
+
+    my $size = guest_var($l1,'gueststorage_size',$def_gueststorage_size);
+    die "gueststorage_size is undefined" unless $size;
+
+    my $outer_vg = target_choose_vg($l0, $size);
+    my $outer_lv = "$l1->{Ident}_gueststorage_outer_lv";
+    my $inner_vg = "$l1->{Ident}_gueststorage_vg";
+
+    target_cmd_root($l0, "vgremove -f $inner_vg ||:");
+    my $outer_lvdev = lv_create($l0, $outer_vg, $outer_lv, $size);
+
+    target_cmd_root($l0, <<END);
+        pvcreate $outer_lvdev
+	vgcreate $inner_vg $outer_lvdev
+END
+
+    toolstack($l0)->block_attach($l1, "$outer_lvdev,raw,sdb,rw");
+    # NB this does not update the l1 guest config so if the l1 is shut
+    # down and recreated in the l0, this will vanish.
+}
+
+packages();
+guest_storage();
+host_install_postboot_complete($l1);
+
+if ($dodefine) {
+    $l1_identspec =~ m/^(.*)=/ or die "$l1_identspec ?";
+    store_runvar($1, $'); #';
+}
Signature · 1 line
1.7.10.4
Acked-by on v1 22/26
#101 lore raw

Re: [OSSTEST PATCH 22/26] Nested HVM: Provide ts-nested-setup to help make L1 usable as a host

Quoted 11 lines · Ian Jackson

From: Robert Ho <[email protected]>

* Provide the L1 with some storage for its own guests' disks
* Install some packages in the L1
* Optionally, set a runvar defining the L1 for the rest of the job

The recipe is going to run ts-xen-install etc.

#102 lore raw

PATCH 23/26 Nested HVM: Provide test-nested recipe

In reply to Ian Jackson

From: Robert Ho <[email protected]>

---
v14: ts-nested-setup command line syntax updated.

---
 sg-run-job |   10 ++++++++++
 1 file changed, 10 insertions(+)
sg-run-job +10 -0
index 8174ef7..6b59ab3 100755
@@ -348,6 +348,16 @@ proc run-job/test-pair-oneway {} {
     run-ts . =              ts-guest-stop          dst_host          + debian
 }
 
+proc need-hosts/test-nested {} {return host}
+proc run-job/test-nested {} {
+    run-ts . = ts-debian-hvm-install + host l1
+    run-ts . = ts-nested-setup + --define l1=host:l1
+    nested-layer-descend l1
+    run-ts . = ts-debian-hvm-install l1 l2
+    run-ts . = ts-guest-stop l1 l2
+    run-ts . = ts-guest-destroy + host l1
+}
+
 proc test-guest-migr {g} {
     set to_reap [spawn-ts . = ts-migrate-support-check + host $g 1]
     set can_migrate [reap-ts $to_reap]
Signature · 1 line
1.7.10.4
author
#104 lore raw

Re: [OSSTEST PATCH 23/26] Nested HVM: Provide test-nested recipe

In reply to Ian Jackson
Quoted 27 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>
Subject: [OSSTEST PATCH 23/26] Nested HVM: Provide test-nested recipe

From: Robert Ho <[email protected]>

---
v14: ts-nested-setup command line syntax updated.

---
 sg-run-job |   10 ++++++++++
 1 file changed, 10 insertions(+)
sg-run-job +1 -0
index 8174ef7..6b59ab3 100755
@@ -348,6 +348,16 @@ proc run-job/test-pair-oneway {} {
     run-ts . =              ts-guest-stop          dst_host
+ debian
 }
 

sg-run-job proc run-job/test-pair-oneway {} { · line 361

Quoted 10 lines
sg-run-job +10 -0
+proc need-hosts/test-nested {} {return host}
+proc run-job/test-nested {} {
+    run-ts . = ts-debian-hvm-install + host l1
+    run-ts . = ts-nested-setup + --define l1=host:l1
+    nested-layer-descend l1
+    run-ts . = ts-debian-hvm-install l1 l2
+    run-ts . = ts-guest-stop l1 l2
+    run-ts . = ts-guest-destroy + host l1
+}
+

[Hu, Robert]

Not sure if rooted to this patch or patch 21 or 22, current test steps will be

[root@robert-ivt osstest]# ./standalone run-job --dry-run -h dummy test-amd64-amd64-qemuu-nested |grep testid Could not open a connection to your authentication agent.
WARNING: Unable to access ssh-agent. Some tests may fail

2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 1 testid build-check(1) ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 2 testid hosts-allocate ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 3 testid host-install(3) ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 4 testid host-ping-check-native ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 5 testid xen-install ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 6 testid xen-boot ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 7 testid host-ping-check-xen ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 8 testid leak-check/basis(8) ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 9 testid debian-hvm-install ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 10 testid nested-setup ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 11 testid host-ping-check-native/l1 ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 12 testid xen-install/l1 ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 13 testid xen-boot/l1 ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 14 testid host-ping-check-xen/l1 ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 15 testid leak-check/basis/l1(15) ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 16 testid debian-hvm-install/l1/l2 ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 17 testid guest-stop/l1/l2 ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 18 testid guest-destroy ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 19 testid leak-check/check/l1 ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 20 testid capture-logs/l1(20) ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 21 testid final-poweroff/l1 ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 22 testid leak-check/check ==========
2015-10-29 02:16:25 Z standalone.test-amd64-amd64-qemuu-nested ========== 23 testid capture-logs(23) ==========

standalone.test-amd64-amd64-qemuu-nested status = pass

Note step 18 destroies l1, while step 19 and step 20 will try to leak check and capture log in l1. I'm not quite understand these tcl part, any fix suggestions?

Quoted 5 lines
sg-run-job +0 -1
 proc test-guest-migr {g} {
     set to_reap [spawn-ts . = ts-migrate-support-check + host $g 1]
     set can_migrate [reap-ts $to_reap]
--

1.7.10.4

#105 lore raw

PATCH 24/26 Nested HVM: Add test job to appropriate flights

In reply to Ian Jackson

From: Robert Ho <[email protected]>

---
v14: Use default gueststorage_size, rather than setting runvar.
Dropped acked from Ian Campbell.

---
 make-flight |   28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
make-flight +28 -0
index 70c909d..4986dd6 100755
@@ -248,6 +248,33 @@ do_hvm_win7_x64_tests () {
             all_hostflags=$most_hostflags,hvm
 }
 
+do_hvm_debian_nested_tests () {
+  bios=$1
+
+  if [ $xenarch != amd64 -o $dom0arch != amd64 \
+      -o "x$qemuu_suffix" != "x-qemuu" ]; then
+    return
+  fi
+
+  case $xenbranch in
+    xen-3.*-testing)      return;;
+    xen-4.0-testing)      return;;
+    xen-4.1-testing)      return;;
+    xen-4.2-testing)      return;;
+    xen-4.3-testing)      return;;
+  esac
+
+  job_create_test test-$xenarch$kern-$dom0arch$qemuu_suffix-nested \
+            test-nested xl $xenarch $dom0arch $qemuu_runvar        \
+            l1_image=debian-7.2.0-amd64-CD-1.iso                   \
+            l1_vifmodel='e1000'                                    \
+            l1_memsize='3072'                                      \
+            l1_enable_nestedhvm='true'                             \
+            l2_image=debian-7.2.0-amd64-CD-1.iso                   \
+            bios=$bios                                             \
+            all_hostflags=$most_hostflags,hvm
+}
+
 branch_debianhvm_arch () {
   case $branch in
     xen-unstable-smoke) echo i386;;
@@ -570,6 +597,7 @@ test_matrix_do_one () {
     do_hvm_rhel6_tests
 
     do_hvm_debian_tests
+    do_hvm_debian_nested_tests seabios
 
   done # qemuu_suffix
 
Signature · 1 line
1.7.10.4
Acked-by on v1 24/26
#106 lore raw

Re: [OSSTEST PATCH 24/26] Nested HVM: Add test job to appropriate flights

Quoted 5 lines · Ian Jackson

From: Robert Ho <[email protected]>

[...]

make-flight do_hvm_win7_x64_tests () { · line 274

Quoted 8 lines
make-flight +8 -0
+  job_create_test test-$xenarch$kern-$dom0arch$qemuu_suffix-nested \
+            test-nested xl $xenarch $dom0arch $qemuu_runvar        \
+            l1_image=debian-7.2.0-amd64-CD-1.iso                   \
+            l1_vifmodel='e1000'                                    \
+            l1_memsize='3072'                                      \
+            l1_enable_nestedhvm='true'                             \
+            l2_image=debian-7.2.0-amd64-CD-1.iso                   \
+            bios=$bios                                             \

This is a preexisting quirk, but I suppose this really ought to be <gident>_bios. As it is it affects both l1 and l2, which is what we want.

Ian.

#107 lore raw

PATCH 25/26 ts-xen-install: Properly handle hosts without a static IP address

In reply to Ian Jackson

From: Robert Ho <[email protected]>

Check IpStatic, and if it is not set, provide a dhcp stanza in /etc/network/interfaces, rather than an `inet static' one.

This is necessary for L1 nested hosts, because they don't have a static IP address.

In principle this makes matters more correct for physical hosts without static IP addresses, but these are currently not supported by selecthost().

---
v14: Only use `inet dhcp' if !$ho->{IpStatic}.

---
 ts-xen-install |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
ts-xen-install +4 -1
index b511e2b..d9aa694 100755
@@ -320,11 +320,14 @@ END
                 /auto $iface\n/x;
             if (m/^\s* iface \s+ (?: $physif | xenbr0 ) \s+ inet \s /x) {
                 $suppress= 1;
-                print EO <<END;
+                print EO ($ho->{IpStatic} ? <<END : <<END), <<END;
 iface $iface inet static
     address $ho->{Ip}
     netmask $netmask
     gateway $gateway
+END
+iface $iface inet dhcp
+END
 $bridgex
 END
             }
Signature · 1 line
1.7.10.4
Acked-by on v1 25/26
#108 lore raw

Re: [OSSTEST PATCH 25/26] ts-xen-install: Properly handle hosts without a static IP address

Quoted 8 lines · Ian Jackson

From: Robert Ho <[email protected]>

Check IpStatic, and if it is not set, provide a dhcp stanza in /etc/network/interfaces, rather than an `inet static' one.

This is necessary for L1 nested hosts, because they don't have a static IP address.

Quoted 6 lines

In principle this makes matters more correct for physical hosts without static IP addresses, but these are currently not supported by selecthost().

author Tested-by
#109 lore raw

Re: [OSSTEST PATCH 25/26] ts-xen-install: Properly handle hosts without a static IP address

Quoted 18 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Monday, September 28, 2015 6:33 PM
To: Ian Jackson <[email protected]>; [email protected]
Cc: Hu, Robert <[email protected]>
Subject: Re: [OSSTEST PATCH 25/26] ts-xen-install: Properly handle hosts without a static IP address

Quoted 8 lines · Ian Jackson

From: Robert Ho <[email protected]>

Check IpStatic, and if it is not set, provide a dhcp stanza in /etc/network/interfaces, rather than an `inet static' one.

This is necessary for L1 nested hosts, because they don't have a static IP address.

Quoted 8 lines
Quoted 6 lines

In principle this makes matters more correct for physical hosts without static IP addresses, but these are currently not supported by selecthost().

[Hu, Robert]

#110 lore raw

PATCH 26/26 ts-xen-install: networking: Rename `nodhcp' to `ensurebridge'

In reply to Ian Jackson

This function does not (now) always undo the DHCP configuration. Sometimes it leaves it. Its main function is to ensure that we have a bridge for use by guests.

So rename the function.

---
v14: This patch was previously 4/4 of a miniature series containing
a different way of dealing with the Nested HVM L1 DHCP problem.

---
 ts-xen-install |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
ts-xen-install +2 -2
index d9aa694..3d0f394 100755
@@ -247,7 +247,7 @@ sub hosts () {
     });
 }
 
-sub nodhcp () {
+sub ensurebridge () {
     target_editfile_root($ho, "/etc/network/interfaces",
                          "etc-network-interfaces", sub {
         my $physif= get_host_property($ho,'interface force',undef);
@@ -370,6 +370,6 @@ if ($checkmode) {
     adjustconfig();
     setupboot();
     setupinitd();
-    nodhcp();
+    ensurebridge();
     hosts();
 }
Signature · 1 line
1.7.10.4
Acked-by on v1 26/26
#111 lore raw

Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename `nodhcp' to `ensurebridge'

Quoted 7 lines · Ian Jackson

This function does not (now) always undo the DHCP configuration. Sometimes it leaves it. Its main function is to ensure that we have a bridge for use by guests.

So rename the function.

author
#112 lore raw

Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename `nodhcp' to `ensurebridge'

In reply to Ian Jackson
Quoted 38 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>; Ian Jackson <[email protected]>
Subject: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'

This function does not (now) always undo the DHCP configuration. Sometimes it leaves it. Its main function is to ensure that we have a bridge for use by guests.

So rename the function.

---
v14: This patch was previously 4/4 of a miniature series containing
a different way of dealing with the Nested HVM L1 DHCP problem.

---
 ts-xen-install |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
ts-xen-install +1 -1
index d9aa694..3d0f394 100755
@@ -247,7 +247,7 @@ sub hosts () {
     });
 }
 
-sub nodhcp () {
+sub ensurebridge () {
     target_editfile_root($ho, "/etc/network/interfaces",
                          "etc-network-interfaces", sub {
         my $physif= get_host_property($ho,'interface force',undef);

ts-xen-install if ($checkmode) { · line 375

Quoted 7 lines
ts-xen-install +1 -1
@@ -370,6 +370,6 @@ if ($checkmode) {
     adjustconfig();
     setupboot();
     setupinitd();
-    nodhcp();
+    ensurebridge();
     hosts();
 }

[Hu, Robert]
Not sure if it is caused by this patch, but after Xen installed in L1, after L1 reboot, it boots fail.

Here is log dumped in L0:
(d53) drive 0x000f6330: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=20480000 (d53) drive 0x000f6360: PCHS=0/0/0 translation=lba LCHS=1024/255/63 s=40960000 (d53) Space available for UMB: ca800-ee800, f5d80-f62d0
(d53) Returned 258048 bytes of ZoneHigh
(d53) e820 map has 6 items:

(d53)   0: 0000000000000000 - 000000000009fc00 = 1 RAM
(d53)   1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
(d53)   2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
(d53)   3: 0000000000100000 - 00000000bf7ff000 = 1 RAM
(d53)   4: 00000000bf7ff000 - 00000000bf800000 = 2 RESERVED
(d53)   5: 00000000fc000000 - 0000000100000000 = 2 RESERVED

(d53) enter handle_19:
(d53) NULL
(d53) Booting from DVD/CD...
(d53) Boot failed: Could not read from CDROM (code 0004) (d53) enter handle_18:
(d53) NULL
(d53) Booting from Hard Disk...
(d53) Booting from 0000:7c00
(XEN) irq.c:275: Dom53 PCI link 0 changed 5 -> 0
(XEN) irq.c:275: Dom53 PCI link 1 changed 10 -> 0
(XEN) irq.c:275: Dom53 PCI link 2 changed 11 -> 0
(XEN) irq.c:275: Dom53 PCI link 3 changed 5 -> 0
(XEN) traps.c:3287: GPF (0000): ffff82d0801ea213 -> ffff82d080244f4b (XEN) traps.c:3287: GPF (0000): ffff82d0801ea213 -> ffff82d080244f4b

Quoted 2 lines

--
1.7.10.4

author
#113 lore raw

Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename `nodhcp' to `ensurebridge'

In reply to Ian Jackson
Quoted 74 lines

-----Original Message-----
From: Hu, Robert
Sent: Thursday, October 15, 2015 5:39 PM
To: 'Ian Jackson' <[email protected]>; [email protected]
Cc: Ian Campbell <[email protected]>
Subject: RE: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'

Quoted 8 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>; Ian Jackson <[email protected]>
Subject: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename

`nodhcp'

Quoted 30 lines

to `ensurebridge'

This function does not (now) always undo the DHCP configuration. Sometimes it leaves it. Its main function is to ensure that we have a bridge for use by guests.

So rename the function.

---
v14: This patch was previously 4/4 of a miniature series containing
a different way of dealing with the Nested HVM L1 DHCP problem.

---
 ts-xen-install |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
ts-xen-install +1 -1
index d9aa694..3d0f394 100755
@@ -247,7 +247,7 @@ sub hosts () {
     });
 }
 
-sub nodhcp () {
+sub ensurebridge () {
     target_editfile_root($ho, "/etc/network/interfaces",
                          "etc-network-interfaces", sub {
         my $physif= get_host_property($ho,'interface force',undef);

ts-xen-install if ($checkmode) { · line 375

Quoted 7 lines
ts-xen-install +1 -1
@@ -370,6 +370,6 @@ if ($checkmode) {
     adjustconfig();
     setupboot();
     setupinitd();
-    nodhcp();
+    ensurebridge();
     hosts();
 }

[Hu, Robert]
Not sure if it is caused by this patch, but after Xen installed in L1, after L1 reboot,
it boots fail.

Here is log dumped in L0:
(d53) drive 0x000f6330: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=20480000
(d53) drive 0x000f6360: PCHS=0/0/0 translation=lba LCHS=1024/255/63 s=40960000
(d53) Space available for UMB: ca800-ee800, f5d80-f62d0 (d53) Returned 258048 bytes of ZoneHigh
(d53) e820 map has 6 items:

(d53)   0: 0000000000000000 - 000000000009fc00 = 1 RAM
(d53)   1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
(d53)   2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
(d53)   3: 0000000000100000 - 00000000bf7ff000 = 1 RAM
(d53)   4: 00000000bf7ff000 - 00000000bf800000 = 2 RESERVED
(d53)   5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
Quoted 15 lines

(d53) enter handle_19:
(d53) NULL
(d53) Booting from DVD/CD...
(d53) Boot failed: Could not read from CDROM (code 0004) (d53) enter handle_18:
(d53) NULL
(d53) Booting from Hard Disk...
(d53) Booting from 0000:7c00
(XEN) irq.c:275: Dom53 PCI link 0 changed 5 -> 0
(XEN) irq.c:275: Dom53 PCI link 1 changed 10 -> 0
(XEN) irq.c:275: Dom53 PCI link 2 changed 11 -> 0
(XEN) irq.c:275: Dom53 PCI link 3 changed 5 -> 0
(XEN) traps.c:3287: GPF (0000): ffff82d0801ea213 -> ffff82d080244f4b (XEN) traps.c:3287: GPF (0000): ffff82d0801ea213 -> ffff82d080244f4b

I'm using latest xen.git master. Will try again with Xen 4.6 release which we have tested.

Quoted 3 lines
Quoted 3 lines

--
1.7.10.4

author
#114 lore raw

Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename `nodhcp' to `ensurebridge'

Quoted 101 lines

-----Original Message-----
From: Hu, Robert
Sent: Thursday, October 15, 2015 5:58 PM
To: Ian Jackson <[email protected]>; [email protected]
Cc: Ian Campbell <[email protected]>
Subject: RE: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'

Quoted 15 lines

-----Original Message-----
From: Hu, Robert
Sent: Thursday, October 15, 2015 5:39 PM
To: 'Ian Jackson' <[email protected]>; [email protected]
Cc: Ian Campbell <[email protected]>
Subject: RE: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'

Quoted 6 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>;

Ian

Quoted 15 lines
Quoted 2 lines

Jackson <[email protected]>
Subject: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename

`nodhcp'

Quoted 12 lines

to `ensurebridge'

This function does not (now) always undo the DHCP configuration. Sometimes it leaves it. Its main function is to ensure that we have a bridge for use by guests.

So rename the function.

---
v14: This patch was previously 4/4 of a miniature series containing
a different way of dealing with the Nested HVM L1 DHCP

problem.

Quoted 44 lines
Quoted 18 lines
---
 ts-xen-install |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
ts-xen-install +1 -1
index d9aa694..3d0f394 100755
@@ -247,7 +247,7 @@ sub hosts () {
     });
 }
 
-sub nodhcp () {
+sub ensurebridge () {
     target_editfile_root($ho, "/etc/network/interfaces",
                          "etc-network-interfaces", sub {
         my $physif= get_host_property($ho,'interface force',undef);

ts-xen-install if ($checkmode) { · line 375

Quoted 7 lines
ts-xen-install +1 -1
@@ -370,6 +370,6 @@ if ($checkmode) {
     adjustconfig();
     setupboot();
     setupinitd();
-    nodhcp();
+    ensurebridge();
     hosts();
 }

[Hu, Robert]
Not sure if it is caused by this patch, but after Xen installed in L1, after L1 reboot,
it boots fail.

Here is log dumped in L0:
(d53) drive 0x000f6330: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=20480000
(d53) drive 0x000f6360: PCHS=0/0/0 translation=lba LCHS=1024/255/63 s=40960000
(d53) Space available for UMB: ca800-ee800, f5d80-f62d0 (d53) Returned 258048 bytes of ZoneHigh
(d53) e820 map has 6 items:

(d53)   0: 0000000000000000 - 000000000009fc00 = 1 RAM
(d53)   1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
(d53)   2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
(d53)   3: 0000000000100000 - 00000000bf7ff000 = 1 RAM
(d53)   4: 00000000bf7ff000 - 00000000bf800000 = 2 RESERVED
(d53)   5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
Quoted 15 lines

(d53) enter handle_19:
(d53) NULL
(d53) Booting from DVD/CD...
(d53) Boot failed: Could not read from CDROM (code 0004) (d53) enter handle_18:
(d53) NULL
(d53) Booting from Hard Disk...
(d53) Booting from 0000:7c00
(XEN) irq.c:275: Dom53 PCI link 0 changed 5 -> 0
(XEN) irq.c:275: Dom53 PCI link 1 changed 10 -> 0
(XEN) irq.c:275: Dom53 PCI link 2 changed 11 -> 0
(XEN) irq.c:275: Dom53 PCI link 3 changed 5 -> 0
(XEN) traps.c:3287: GPF (0000): ffff82d0801ea213 -> ffff82d080244f4b (XEN) traps.c:3287: GPF (0000): ffff82d0801ea213 -> ffff82d080244f4b

Quoted 3 lines

I'm using latest xen.git master. Will try again with Xen 4.6 release which we have
tested.

[Hu, Robert]
After fixed several environment setting issues, I just run OSSTest on Xen 4.6.0 release code. This issue still exist there, so this is probably the new patch which introduced this.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
1.7.10.4

author
#115 lore raw

Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename `nodhcp' to `ensurebridge'

Quoted 111 lines

-----Original Message-----
From: Hu, Robert
Sent: Thursday, October 22, 2015 10:33 AM
To: 'Ian Jackson' <[email protected]>; '[email protected]' <[email protected]>
Cc: 'Ian Campbell' <[email protected]>
Subject: RE: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'

Quoted 53 lines

-----Original Message-----
From: Hu, Robert
Sent: Thursday, October 15, 2015 5:58 PM
To: Ian Jackson <[email protected]>; [email protected]
Cc: Ian Campbell <[email protected]>
Subject: RE: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'

Quoted 15 lines

-----Original Message-----
From: Hu, Robert
Sent: Thursday, October 15, 2015 5:39 PM
To: 'Ian Jackson' <[email protected]>; [email protected]
Cc: Ian Campbell <[email protected]>
Subject: RE: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'

Quoted 6 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>;

Ian

Quoted 15 lines
Quoted 2 lines

Jackson <[email protected]>
Subject: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename

`nodhcp'

Quoted 12 lines

to `ensurebridge'

This function does not (now) always undo the DHCP configuration. Sometimes it leaves it. Its main function is to ensure that we have a bridge for use by guests.

So rename the function.

---
v14: This patch was previously 4/4 of a miniature series containing
a different way of dealing with the Nested HVM L1 DHCP

problem.

Quoted 12 lines
Quoted 12 lines
---
 ts-xen-install |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
ts-xen-install +0 -0
index d9aa694..3d0f394 100755
@@ -247,7 +247,7 @@ sub hosts () {
     });
 }
 
Quoted 5 lines
Quoted 5 lines

ts-xen-install sub hosts () {

Quoted 5 lines
ts-xen-install +1 -1
-sub nodhcp () {
+sub ensurebridge () {
     target_editfile_root($ho, "/etc/network/interfaces",
                          "etc-network-interfaces", sub {
         my $physif= get_host_property($ho,'interface

force',undef);

Quoted 43 lines
Quoted 42 lines

ts-xen-install if ($checkmode) { · line 375

Quoted 8 lines
ts-xen-install +1 -1
@@ -370,6 +370,6 @@ if ($checkmode) {
     adjustconfig();
     setupboot();
     setupinitd();
-    nodhcp();
+    ensurebridge();
     hosts();
 }

[Hu, Robert]
Not sure if it is caused by this patch, but after Xen installed in L1, after L1 reboot,
it boots fail.

Here is log dumped in L0:
(d53) drive 0x000f6330: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=20480000
(d53) drive 0x000f6360: PCHS=0/0/0 translation=lba LCHS=1024/255/63 s=40960000
(d53) Space available for UMB: ca800-ee800, f5d80-f62d0 (d53) Returned 258048 bytes of ZoneHigh
(d53) e820 map has 6 items:

(d53)   0: 0000000000000000 - 000000000009fc00 = 1 RAM
(d53)   1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
(d53)   2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
(d53)   3: 0000000000100000 - 00000000bf7ff000 = 1 RAM
(d53)   4: 00000000bf7ff000 - 00000000bf800000 = 2 RESERVED
(d53)   5: 00000000fc000000 - 0000000100000000 = 2 RESERVED

(d53) enter handle_19:
(d53) NULL
(d53) Booting from DVD/CD...
(d53) Boot failed: Could not read from CDROM (code 0004) (d53) enter handle_18:
(d53) NULL
(d53) Booting from Hard Disk...
(d53) Booting from 0000:7c00
(XEN) irq.c:275: Dom53 PCI link 0 changed 5 -> 0
(XEN) irq.c:275: Dom53 PCI link 1 changed 10 -> 0
(XEN) irq.c:275: Dom53 PCI link 2 changed 11 -> 0
(XEN) irq.c:275: Dom53 PCI link 3 changed 5 -> 0
(XEN) traps.c:3287: GPF (0000): ffff82d0801ea213 -> ffff82d080244f4b (XEN) traps.c:3287: GPF (0000): ffff82d0801ea213 -> ffff82d080244f4b

Quoted 8 lines
Quoted 3 lines

I'm using latest xen.git master. Will try again with Xen 4.6 release which we have
tested.

[Hu, Robert]
After fixed several environment setting issues, I just run OSSTest on Xen 4.6.0 release code. This issue still exist there, so this is probably the new patch which
introduced this.

[Hu, Robert]
Seems the failure log shall be this, any idea? I've spent days on debugging this:( (XEN) traps.c:3290: GPF (0000): ffff82d080195082 -> ffff82d080243d9d (XEN) PCI add device 0000:00:00.0
(XEN) PCI add device 0000:00:01.0
(XEN) PCI add device 0000:00:01.1
(XEN) PCI add device 0000:00:01.2
(XEN) PCI add device 0000:00:01.3
(XEN) PCI add device 0000:00:02.0
(XEN) PCI add device 0000:00:03.0
(XEN) d0: Forcing read-only access to MFN fed00
(XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds.
Issued domain 3 reboot
qemu: terminating on signal 1 from pid 4322

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

--
1.7.10.4

#116 lore raw

Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename `nodhcp' to `ensurebridge'

Quoted 7 lines · Robert

[Hu, Robert]
Seems the failure log shall be this, any idea? I've spent days on debugging this:(
(XEN) traps.c:3290: GPF (0000): ffff82d080195082 -> ffff82d080243d9d (XEN) PCI add device 0000:00:00.0
(XEN) PCI add device 0000:00:01.0
(XEN) PCI add device 0000:00:01.1

Quoted 8 lines

(XEN) PCI add device 0000:00:01.2
(XEN) PCI add device 0000:00:01.3
(XEN) PCI add device 0000:00:02.0
(XEN) PCI add device 0000:00:03.0
(XEN) d0: Forcing read-only access to MFN fed00 (XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds. Issued domain 3 reboot
qemu: terminating on signal 1 from pid 4322

Please can you report this as a regular bug in a fresh thread, that way the relevant Xen maintainers are likely to see and react to it, rather than just us osstest folks.

Please include whatever information you can, e.g. the guest config file used, details about the type of guest, which level of nesting this is happening at, the contents of any logs under /var/log/xen in L0 and L1 host etc.

If this crash is in an L1 host then you might also want to CC the nested hvm maintainers at Intel.

See http://wiki.xenproject.org/wiki/Reporting_Bugs_against_Xen_Project
for general advice on reporting a bug and other things to consider including.

Thanks,
Ian.

author
#117 lore raw

Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename `nodhcp' to `ensurebridge'

Quoted 27 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Friday, October 23, 2015 4:15 PM
To: Hu, Robert <[email protected]>; 'Ian Jackson' <[email protected]>; '[email protected]' <[email protected]>
Subject: Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'

Quoted 7 lines · Robert

[Hu, Robert]
Seems the failure log shall be this, any idea? I've spent days on debugging this:(
(XEN) traps.c:3290: GPF (0000): ffff82d080195082 -> ffff82d080243d9d (XEN) PCI add device 0000:00:00.0
(XEN) PCI add device 0000:00:01.0
(XEN) PCI add device 0000:00:01.1

Quoted 8 lines

(XEN) PCI add device 0000:00:01.2
(XEN) PCI add device 0000:00:01.3
(XEN) PCI add device 0000:00:02.0
(XEN) PCI add device 0000:00:03.0
(XEN) d0: Forcing read-only access to MFN fed00 (XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds. Issued domain 3 reboot
qemu: terminating on signal 1 from pid 4322

Quoted 3 lines

Please can you report this as a regular bug in a fresh thread, that way the relevant Xen maintainers are likely to see and react to it, rather than just us osstest folks.

[Hu, Robert]

It shall be in that way after I confirm it is a bug.
Currently I'm just still not certain it is a nested bug or because of the latest osstest code change.
I was just asking for if you can recall some hint on what changes (of osstest) might causing this.
I'm to restore to my v12 code, with current Xen and qemu selection to try again. I think by this way, I can confirm it is an actual nested bug or not. Then I would report this in a fresh thread.

Quoted 5 lines

Please include whatever information you can, e.g. the guest config file used, details about the type of guest, which level of nesting this is happening at, the contents of any logs under /var/log/xen in L0 and L1 host etc.

[Hu, Robert]

Yes, sure. I will include these in bug reporting.

Quoted 3 lines

If this crash is in an L1 host then you might also want to CC the nested hvm maintainers at Intel.

[Hu, Robert]
Yes, thanks for remind.

Quoted 7 lines

See http://wiki.xenproject.org/wiki/Reporting_Bugs_against_Xen_Project
for general advice on reporting a bug and other things to consider including.

Thanks,
Ian.

#118 lore raw

Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename `nodhcp' to `ensurebridge'

Quoted 32 lines · Robert
Quoted 27 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Friday, October 23, 2015 4:15 PM
To: Hu, Robert <[email protected]>; 'Ian Jackson' <[email protected]>; '[email protected]' <[email protected]>
Subject: Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'

Quoted 7 lines · Robert

[Hu, Robert]
Seems the failure log shall be this, any idea? I've spent days on debugging this:(
(XEN) traps.c:3290: GPF (0000): ffff82d080195082 -> ffff82d080243d9d (XEN) PCI add device 0000:00:00.0
(XEN) PCI add device 0000:00:01.0
(XEN) PCI add device 0000:00:01.1

Quoted 8 lines

(XEN) PCI add device 0000:00:01.2
(XEN) PCI add device 0000:00:01.3
(XEN) PCI add device 0000:00:02.0
(XEN) PCI add device 0000:00:03.0
(XEN) d0: Forcing read-only access to MFN fed00 (XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds. Issued domain 3 reboot
qemu: terminating on signal 1 from pid 4322

Quoted 3 lines

Please can you report this as a regular bug in a fresh thread, that way the relevant Xen maintainers are likely to see and react to it, rather than just us osstest folks.

[Hu, Robert]

Quoted 11 lines

It shall be in that way after I confirm it is a bug. Currently I'm just still not certain it is a nested bug or because of the latest
osstest code change.
I was just asking for if you can recall some hint on what changes (of osstest)
might causing this.
I'm to restore to my v12 code, with current Xen and qemu selection to try again. I think by this way, I can confirm it is an actual nested bug or not.
Then I would report this in a fresh thread.

A dom0 crash of this sort is pretty certainly a bug somewhere in Xen, whether it is exposed by a new osstest case or not. The people who are best placed to figure out where the bug is are certainly not reading this osstest thread.

So please just report it as a bug as it stands, with the relevant information.

Ian.

author
#119 lore raw

Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename `nodhcp' to `ensurebridge'

Quoted 9 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Friday, October 23, 2015 9:38 PM
To: Hu, Robert <[email protected]>; 'Ian Jackson' <[email protected]>; '[email protected]' <[email protected]>
Subject: Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'

Quoted 45 lines
Quoted 43 lines · Robert
Quoted 27 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Friday, October 23, 2015 4:15 PM
To: Hu, Robert <[email protected]>; 'Ian Jackson' <[email protected]>; '[email protected]' <[email protected]>
Subject: Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'

Quoted 7 lines · Robert

[Hu, Robert]
Seems the failure log shall be this, any idea? I've spent days on debugging this:(
(XEN) traps.c:3290: GPF (0000): ffff82d080195082 -> ffff82d080243d9d (XEN) PCI add device 0000:00:00.0
(XEN) PCI add device 0000:00:01.0
(XEN) PCI add device 0000:00:01.1

Quoted 8 lines

(XEN) PCI add device 0000:00:01.2
(XEN) PCI add device 0000:00:01.3
(XEN) PCI add device 0000:00:02.0
(XEN) PCI add device 0000:00:03.0
(XEN) d0: Forcing read-only access to MFN fed00 (XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds. Issued domain 3 reboot
qemu: terminating on signal 1 from pid 4322

Quoted 3 lines

Please can you report this as a regular bug in a fresh thread, that way the relevant Xen maintainers are likely to see and react to it, rather than just us osstest folks.

[Hu, Robert]

It shall be in that way after I confirm it is a bug. Currently I'm just still not certain it is a nested bug or because of the latest
osstest code change.
I was just asking for if you can recall some hint on what changes (of osstest)
might causing this.
I'm to restore to my v12 code, with current Xen and qemu selection to try again. I think by this way, I can confirm it is an actual nested bug or not.
Then I would report this in a fresh thread.

[Hu, Robert]

With v12 code, on same L1 Dom0 kernel, L1 Xen and Qemu selection, nested test passes.
I've saved l1 guest configuration, l1 network configuration, and l1 Dom0 kernel config for further comparison. Anything else possibly related you can think of?

Quoted 7 lines

A dom0 crash of this sort is pretty certainly a bug somewhere in Xen, whether it is exposed by a new osstest case or not. The people who are best placed to figure out where the bug is are certainly not reading this osstest thread.

So please just report it as a bug as it stands, with the relevant information.

[Hu, Robert]

Nested Xen is in tech preview phase, not that production level matured. It is so nit-picking that any configuration change not meeting its appetite will induce its naughtiness, i.e. crash, I think. :)

Quoted 2 lines

Ian.

author
#120 lore raw

Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename `nodhcp' to `ensurebridge'

Quoted 80 lines

-----Original Message-----
From: Hu, Robert
Sent: Sunday, October 25, 2015 10:46 AM
To: Ian Campbell <[email protected]>; 'Ian Jackson' <[email protected]>; '[email protected]' <[email protected]>
Subject: RE: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'

Quoted 9 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Friday, October 23, 2015 9:38 PM
To: Hu, Robert <[email protected]>; 'Ian Jackson' <[email protected]>; '[email protected]' <[email protected]>
Subject: Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'

Quoted 16 lines
Quoted 15 lines · Robert
Quoted 15 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Friday, October 23, 2015 4:15 PM
To: Hu, Robert <[email protected]>; 'Ian Jackson' <[email protected]>; '[email protected]' <[email protected]>
Subject: Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'

Quoted 4 lines · Robert

[Hu, Robert]
Seems the failure log shall be this, any idea? I've spent days on debugging this:(
(XEN) traps.c:3290: GPF (0000): ffff82d080195082 ->

ffff82d080243d9d

Quoted 17 lines
Quoted 17 lines
Quoted 12 lines
Quoted 11 lines

(XEN) PCI add device 0000:00:00.0
(XEN) PCI add device 0000:00:01.0
(XEN) PCI add device 0000:00:01.1
(XEN) PCI add device 0000:00:01.2
(XEN) PCI add device 0000:00:01.3
(XEN) PCI add device 0000:00:02.0
(XEN) PCI add device 0000:00:03.0
(XEN) d0: Forcing read-only access to MFN fed00 (XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds. Issued domain 3 reboot
qemu: terminating on signal 1 from pid 4322

Quoted 3 lines

Please can you report this as a regular bug in a fresh thread, that way the relevant Xen maintainers are likely to see and react to it, rather than just us osstest folks.

[Hu, Robert]

Quoted 12 lines
Quoted 11 lines

It shall be in that way after I confirm it is a bug. Currently I'm just still not certain it is a nested bug or because of the latest
osstest code change.
I was just asking for if you can recall some hint on what changes (of osstest)
might causing this.
I'm to restore to my v12 code, with current Xen and qemu selection to try again. I think by this way, I can confirm it is an actual nested bug or not.
Then I would report this in a fresh thread.

[Hu, Robert]

With v12 code, on same L1 Dom0 kernel, L1 Xen and Qemu selection, nested test
passes.
I've saved l1 guest configuration, l1 network configuration, and l1 Dom0 kernel
config for further comparison. Anything else possibly related you can think of?

Quoted 2 lines

A dom0 crash of this sort is pretty certainly a bug somewhere in Xen, whether it is exposed by a new osstest case or not. The people who are

best

Quoted 3 lines

placed to figure out where the bug is are certainly not reading this osstest thread.

Quoted 7 lines
Quoted 2 lines

So please just report it as a bug as it stands, with the relevant information.

[Hu, Robert]

Nested Xen is in tech preview phase, not that production level matured. It is so nit-picking that any configuration change not meeting its appetite will induce its naughtiness, i.e. crash, I think. :)

[Hu, Robert]

Root cause found: Dom0 kernel boot cmd line: console=xvc0 matters, shall be hvc0 in nested environment.

Quoted 2 lines
Quoted 2 lines

Ian.

author
#121 lore raw

Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename `nodhcp' to `ensurebridge'

In reply to Hu, Robert
Quoted 98 lines

-----Original Message-----
From: Hu, Robert
Sent: Monday, October 26, 2015 3:05 PM
To: 'Ian Campbell' <[email protected]>; 'Ian Jackson' <[email protected]>; '[email protected]' <[email protected]>
Subject: RE: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'

Quoted 26 lines

-----Original Message-----
From: Hu, Robert
Sent: Sunday, October 25, 2015 10:46 AM
To: Ian Campbell <[email protected]>; 'Ian Jackson' <[email protected]>; '[email protected]' <[email protected]>
Subject: RE: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'

Quoted 17 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Friday, October 23, 2015 9:38 PM
To: Hu, Robert <[email protected]>; 'Ian Jackson' <[email protected]>; '[email protected]' <[email protected]>
Subject: Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'

Quoted 7 lines · Robert
Quoted 7 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Friday, October 23, 2015 4:15 PM
To: Hu, Robert <[email protected]>; 'Ian Jackson' <[email protected]>; '[email protected]' <[email protected]>
Subject: Re: [OSSTEST PATCH 26/26] ts-xen-install: networking:

Rename

Quoted 34 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

nodhcp' to ensurebridge'

Quoted 4 lines · Robert

[Hu, Robert]
Seems the failure log shall be this, any idea? I've spent days on debugging this:(
(XEN) traps.c:3290: GPF (0000): ffff82d080195082 ->

ffff82d080243d9d

Quoted 25 lines
Quoted 25 lines
Quoted 12 lines
Quoted 11 lines

(XEN) PCI add device 0000:00:00.0
(XEN) PCI add device 0000:00:01.0
(XEN) PCI add device 0000:00:01.1
(XEN) PCI add device 0000:00:01.2
(XEN) PCI add device 0000:00:01.3
(XEN) PCI add device 0000:00:02.0
(XEN) PCI add device 0000:00:03.0
(XEN) d0: Forcing read-only access to MFN fed00 (XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds. Issued domain 3 reboot
qemu: terminating on signal 1 from pid 4322

Quoted 3 lines

Please can you report this as a regular bug in a fresh thread, that way the relevant Xen maintainers are likely to see and react to it, rather than just us osstest folks.

[Hu, Robert]

It shall be in that way after I confirm it is a bug. Currently I'm just still not certain it is a nested bug or because of the latest
osstest code change.
I was just asking for if you can recall some hint on what changes (of osstest)
might causing this.
I'm to restore to my v12 code, with current Xen and qemu selection to

try

Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

again. I think by this way, I can confirm it is an actual nested bug or not.
Then I would report this in a fresh thread.

[Hu, Robert]

With v12 code, on same L1 Dom0 kernel, L1 Xen and Qemu selection,

nested

Quoted 13 lines

test
passes.
I've saved l1 guest configuration, l1 network configuration, and l1 Dom0 kernel
config for further comparison. Anything else possibly related you can think of?

Quoted 2 lines

A dom0 crash of this sort is pretty certainly a bug somewhere in Xen, whether it is exposed by a new osstest case or not. The people who are

best

Quoted 3 lines

placed to figure out where the bug is are certainly not reading this osstest thread.

Quoted 6 lines
Quoted 2 lines

So please just report it as a bug as it stands, with the relevant information.

[Hu, Robert]

Nested Xen is in tech preview phase, not that production level matured. It is so nit-picking that any configuration change not meeting its appetite

Quoted 7 lines

will

Quoted 1 line

induce its naughtiness, i.e. crash, I think. :)

[Hu, Robert]

Root cause found: Dom0 kernel boot cmd line: console=xvc0 matters, shall be
hvc0 in nested environment.

[Hu, Robert]

A patch for this: in ts-xen-install, after exact kernel and xen, check if 'kernkind' for this host exist, if not, set it with existing runvar.
The hvc0 --> xvc0 happens in debian_boot_setup() --> target_kernkind_check(). If 'kernkind' runvar for host is missing, the existing code will set it to xvc0. For our pvops kernel, it shall be hvc0.

ts-xen-install +9 -0
index 3d0f394..eb40c1e 100755
@@ -79,6 +79,15 @@ sub extract () {
                $r{"$ho->{Ident}_${part}buildjob"} // $r{"${part}buildjob"},
                \%distpath);
     }
+    
+    if (!target_var($ho, 'kernkind'))
+    {
+       my $pfx=target_var_prefix($ho);
+
+       store_runvar($pfx."kernkind", $r{'kernkind'});
+    }
+    
+
     if (target_file_exists($ho, "/usr/lib64/efi/xen.efi")) {
        target_cmd_root($ho,<<END);
            mkdir -p /boot/efi/EFI/osstest

How would you like it? it is just my proposal fix. (Tested)

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Ian.

#122 lore raw

Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename `nodhcp' to `ensurebridge'

Hu, Robert writes ("RE: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'"):

Quoted 1 line
Quoted 1 line

From: Hu, Robert

...

Quoted 4 lines
Quoted 3 lines

Root cause found: Dom0 kernel boot cmd line: console=xvc0 matters, shall be
hvc0 in nested environment.

[Hu, Robert]

Thanks for the update. I'm glad to hear you seem to be making good progress.

I think from reading this thread that this is not in fact a bug in anything except your osstest series, because the dom0 that is dying is the L1 ? I think it's just dying because it can't find its console. Is that right ?

Quoted 3 lines

A patch for this: in ts-xen-install, after exact kernel and xen, check if 'kernkind' for this host exist, if not, set it with existing runvar.

I think that it would be better to change the default for kernkind.

At the moment kernkind runvars are looked at only in target_kernkind_check, which has three possible paths:

(a) eq 'pvops'
(b) m/2618/
(c) the rest (including undef, although undef prints a warning)

I propose to change the semantics of a missing kernkind runvar from
(c) to (a).

This is safe only if no existing flights would be affected. (That is, the meaning of no existing sets of runvars would be changed.)

To check whether this would make any difference I did some database searches. Since any time target_kernkind_check is called it sets a corresponding console' runvar, I can search for console' without a corresponding `kernkind'. I ran this query:

select * from (select *, (select name from runvars r2 where r2.flight=r1.flight and r2.job=r1.job and r2.name= replace(r1.name,'console','kernkind')) kk from runvars r1 where r1.name like '%console') iq where kk is null order by flight desc;

and it found nothing since flight 7682. So I think we can change the default.

I therefore suggest something like this:

Osstest/TestSupport.pm +1 -1
index f9eba6b..48b8ffd 100644
@@ -2006,7 +2006,7 @@ sub target_var ($$) {
 sub target_kernkind_check ($) {
     my ($gho) = @_;
     my $pfx= target_var_prefix($gho);
-    my $kernkind= $r{$pfx."kernkind"};
+    my $kernkind= $r{$pfx."kernkind"} // 'pvops';
     my $isguest= exists $gho->{Guest};
     if ($kernkind eq 'pvops') {
         store_runvar($pfx."rootdev", 'xvda') if $isguest;

If you agree and this works for you please put that into your series with a proper commit message. Please quote my words about existing flights (including the database query etc.) in the commit message.

Thanks,
Ian.

author Tested-by
#123 lore raw

Re: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename `nodhcp' to `ensurebridge'

Quoted 19 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Tuesday, October 27, 2015 12:29 AM
To: Hu, Robert <[email protected]>
Cc: 'Ian Campbell' <[email protected]>; '[email protected]' <[email protected]>
Subject: RE: [OSSTEST PATCH 26/26] ts-xen-install: networking: Rename nodhcp' to ensurebridge'

Hu, Robert writes ("RE: [OSSTEST PATCH 26/26] ts-xen-install: networking:
Rename nodhcp' to ensurebridge'"):

Quoted 1 line
Quoted 1 line

From: Hu, Robert

...

Quoted 1 line
Quoted 1 line

Root cause found: Dom0 kernel boot cmd line: console=xvc0 matters,

shall

Quoted 3 lines
Quoted 2 lines

be
hvc0 in nested environment.

[Hu, Robert]

Quoted 7 lines

Thanks for the update. I'm glad to hear you seem to be making good progress.

I think from reading this thread that this is not in fact a bug in anything except your osstest series, because the dom0 that is dying is the L1 ? I think it's just dying because it can't find its console. Is that right ?

[Hu, Robert]

It must miss something in re-constructing patch from v12, which works well.
Yes, dying Dom0 is L1. It is dying because it try to use xvc0 as console while its kernkind is pvops.

Quoted 53 lines
Quoted 5 lines

A patch for this: in ts-xen-install, after exact kernel and xen, check if 'kernkind' for this host exist, if not, set it with existing runvar.

I think that it would be better to change the default for kernkind.

At the moment kernkind runvars are looked at only in target_kernkind_check, which has three possible paths:

(a) eq 'pvops'
(b) m/2618/
(c) the rest (including undef, although undef prints a warning)

I propose to change the semantics of a missing kernkind runvar from
(c) to (a).

This is safe only if no existing flights would be affected. (That is, the meaning of no existing sets of runvars would be changed.)

To check whether this would make any difference I did some database searches. Since any time target_kernkind_check is called it sets a corresponding console' runvar, I can search for console' without a corresponding `kernkind'. I ran this query:

select * from (select *, (select name from runvars r2 where r2.flight=r1.flight and r2.job=r1.job and r2.name= replace(r1.name,'console','kernkind')) kk from runvars r1 where r1.name like '%console') iq where kk is null order by flight desc;

and it found nothing since flight 7682. So I think we can change the default.

I therefore suggest something like this:

Osstest/TestSupport.pm +1 -1
index f9eba6b..48b8ffd 100644
@@ -2006,7 +2006,7 @@ sub target_var ($$) {
 sub target_kernkind_check ($) {
     my ($gho) = @_;
     my $pfx= target_var_prefix($gho);
-    my $kernkind= $r{$pfx."kernkind"};
+    my $kernkind= $r{$pfx."kernkind"} // 'pvops';
     my $isguest= exists $gho->{Guest};
     if ($kernkind eq 'pvops') {
         store_runvar($pfx."rootdev", 'xvda') if $isguest;
Quoted 3 lines

If you agree and this works for you please put that into your series with a proper commit message. Please quote my words about existing flights (including the database query etc.) in the commit message.

[Hu, Robert]

Sure.

Quoted 6 lines

Thanks,
Ian.

#124 lore raw

Re: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

In reply to Ian Jackson
Quoted 2 lines · Ian Jackson

Ian Campbell: You probably want to defer re-reviewing this until Robert reports back.

I really ought to have started by reading this intro mail. Oh well.

author
#125 lore raw

Re: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

In reply to Ian Jackson
Quoted 30 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>
Subject: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

This is the second part of v14 Robert Ho's osstest patch series to support nested HVM tests.

It is also available here:
git://xenbits.xen.org/people/iwj/xen.git
http://xenbits.xen.org/git-http/people/iwj/xen.git
in wip.nested-hvm.v14.part1..wip.nested-hvm.v14

Compared to Robert's v13, which was passed to me by private email,
* I have rebased onto current osstest pretest;
* I have changed how selecthost() is told it's dealing with a nested host (in practice, L1 guest);
* There are a large number of minor cleanups;
* There are some new preparatory cleanup and admin patches;
* I have rewritten almost all of the commit messages.

However, I have done only VERY LIMITED testing. Much of the code here is UNTESTED since my changes. My testing was confined to:
* Verifying that my changes to cs-adjust-flight worked
* Checking that ad-hoc runs of ts-host-reboot and ts-host-powercycle seemed to work when a guest was specified on the command line.

Quoted 4 lines

Robert, you kindly volunteered to test a revised version of this series. I would appreciate if you would check that all of this still works as you expect. I expect there will be some bugs, perhaps even very silly bugs, introduced by me.

Sure.
Am I supposed to test Part 1 together with this Part 2? or each individually? How's Part 1's status now? pass pretest and in production branch? or still need test?

Quoted 9 lines

I noticed that this series lacks guest serial debug keys and log collection for the L1 guest, because there is no Osstest/Serial/guest.pm. I would appreciate it if you would provide one. I don't think it needs to actually collect any logs, because the L1 serial output log will be collected as part of the L0 log collection. But it ought to support sending debug keys to the L1 guest. When you have provided it you can (in the same patch) fix the corresponding todo' in selecthost, changing noop' to `guest'.

OK, I'll try to add this.

Quoted 13 lines

Workflow:

Robert: I'm handing this (what I have called `part 2') over to you now.

When you make changes, feel free to either rebase, or to make fixup commits (perhaps in git-rebase -i --autosquash' format) on top. If you do the latter then you'll probably want to pass that to me as a git branch (via git push to xenbits or emailing me a git bundle), since squash!' and `fixup!' commits don't look good in email :-).

Quoted 4 lines

If you rebase, please put changes
v15: <description of what you did>
in the commit messages, as I have done myself in v14. Leave my v14 notes in place.

Sure. Thanks for leading the way.

Quoted 8 lines

Of course if you have any comments or queries about how I have done things, they would be very welcome.

Please do not rebase any of the commits in wip.nested-hvm.v14.part1. If you discover bugs in `part 1' please let us know as I have fed that into the osstest self-test mill with the expectation that it will go into production.

What's the result now? has it passed pretest and in production now?

Quoted 4 lines

I do not expect you to test the changes to cs-adjust-flight. I have done that. Indeed they are not really related to the Nested HVM work and Ian C and I may pick them up in another series.

OK, going to bypass it.

Quoted 7 lines

Ian Campbell: You probably want to defer re-reviewing this until Robert reports back.

author
#126 lore raw

Re: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

Quoted 38 lines

-----Original Message-----
From: Hu, Robert
Sent: Monday, October 12, 2015 11:36 AM
To: Ian Jackson <[email protected]>; [email protected]
Cc: Ian Campbell <[email protected]>
Subject: RE: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

Quoted 30 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>
Subject: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

This is the second part of v14 Robert Ho's osstest patch series to support nested HVM tests.

It is also available here:
git://xenbits.xen.org/people/iwj/xen.git
http://xenbits.xen.org/git-http/people/iwj/xen.git
in wip.nested-hvm.v14.part1..wip.nested-hvm.v14

Compared to Robert's v13, which was passed to me by private email,
* I have rebased onto current osstest pretest;
* I have changed how selecthost() is told it's dealing with a nested host (in practice, L1 guest);
* There are a large number of minor cleanups;
* There are some new preparatory cleanup and admin patches;
* I have rewritten almost all of the commit messages.

However, I have done only VERY LIMITED testing. Much of the code here is UNTESTED since my changes. My testing was confined to:
* Verifying that my changes to cs-adjust-flight worked
* Checking that ad-hoc runs of ts-host-reboot and ts-host-powercycle seemed to work when a guest was specified on the command line.

Quoted 8 lines
Quoted 4 lines

Robert, you kindly volunteered to test a revised version of this series. I would appreciate if you would check that all of this still works as you expect. I expect there will be some bugs, perhaps even very silly bugs, introduced by me.

Sure.
Am I supposed to test Part 1 together with this Part 2? or each individually? How's Part 1's status now? pass pretest and in production branch? or still need test?

Seems Part 1 already in production tree. So I directly merged Part 2 on latest production master branch.

However, when I run 'standalone-reset', I get these errors
[root@robert-ivt osstest]# ./standalone-reset
** need to generate d-i with firmware for armhf
fetching initrd.gz
http://linux-ftp.sh.intel.com/pub/mirrors/debian//dists/wheezy/main/installer-armhf/current/images/vexpress/netboot/initrd.gz => initrd.gz.new collecting firmware-bnx2
collecting backports initramfs-tools
collecting backports kernel
dpkg-deb: error: `backports.deb' is not a debian format archive

Quoted 51 lines
Quoted 9 lines

I noticed that this series lacks guest serial debug keys and log collection for the L1 guest, because there is no Osstest/Serial/guest.pm. I would appreciate it if you would provide one. I don't think it needs to actually collect any logs, because the L1 serial output log will be collected as part of the L0 log collection. But it ought to support sending debug keys to the L1 guest. When you have provided it you can (in the same patch) fix the corresponding todo' in selecthost, changing noop' to `guest'.

OK, I'll try to add this.

Quoted 13 lines

Workflow:

Robert: I'm handing this (what I have called `part 2') over to you now.

When you make changes, feel free to either rebase, or to make fixup commits (perhaps in git-rebase -i --autosquash' format) on top. If you do the latter then you'll probably want to pass that to me as a git branch (via git push to xenbits or emailing me a git bundle), since squash!' and `fixup!' commits don't look good in email :-).

Quoted 4 lines

If you rebase, please put changes
v15: <description of what you did>
in the commit messages, as I have done myself in v14. Leave my v14 notes in place.

Sure. Thanks for leading the way.

Quoted 8 lines

Of course if you have any comments or queries about how I have done things, they would be very welcome.

Please do not rebase any of the commits in wip.nested-hvm.v14.part1. If you discover bugs in `part 1' please let us know as I have fed that into the osstest self-test mill with the expectation that it will go into production.

What's the result now? has it passed pretest and in production now?

Quoted 4 lines

I do not expect you to test the changes to cs-adjust-flight. I have done that. Indeed they are not really related to the Nested HVM work and Ian C and I may pick them up in another series.

OK, going to bypass it.

Quoted 7 lines

Ian Campbell: You probably want to defer re-reviewing this until Robert reports back.

#127 lore raw

Re: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

Quoted 52 lines · Robert
Quoted 43 lines

-----Original Message-----
From: Hu, Robert
Sent: Monday, October 12, 2015 11:36 AM
To: Ian Jackson <[email protected]>; [email protected]
Cc: Ian Campbell <[email protected]>
Subject: RE: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

Quoted 31 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>
Subject: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

This is the second part of v14 Robert Ho's osstest patch series to support nested HVM tests.

It is also available here:
git://xenbits.xen.org/people/iwj/xen.git
http://xenbits.xen.org/git-http/people/iwj/xen.git
in wip.nested-hvm.v14.part1..wip.nested-hvm.v14

Compared to Robert's v13, which was passed to me by private email,
* I have rebased onto current osstest pretest;
* I have changed how selecthost() is told it's dealing with a nested host (in practice, L1 guest);
* There are a large number of minor cleanups;
* There are some new preparatory cleanup and admin patches;
* I have rewritten almost all of the commit messages.

However, I have done only VERY LIMITED testing. Much of the code here
is UNTESTED since my changes. My testing was confined to:
* Verifying that my changes to cs-adjust-flight worked
* Checking that ad-hoc runs of ts-host-reboot and ts-host-powercycle seemed to work when a guest was specified on the command line.

Quoted 4 lines

Robert, you kindly volunteered to test a revised version of this series. I would appreciate if you would check that all of this still works as you expect. I expect there will be some bugs, perhaps even very silly bugs, introduced by me.

Quoted 6 lines

Sure.
Am I supposed to test Part 1 together with this Part 2? or each individually?
How's Part 1's status now? pass pretest and in production branch? or still
need test?

Seems Part 1 already in production tree. So I directly merged Part 2 on latest production master branch.

Quoted 10 lines

However, when I run 'standalone-reset', I get these errors [root@robert-ivt osstest]# ./standalone-reset
** need to generate d-i with firmware for armhf
fetching initrd.gz
http://linux-ftp.sh.intel.com/pub/mirrors/debian//dists/wheezy/main/insta
ller-armhf/current/images/vexpress/netboot/initrd.gz => initrd.gz.new collecting firmware-bnx2
collecting backports initramfs-tools
collecting backports kernel
dpkg-deb: error: `backports.deb' is not a debian format archive

Can you look at the file backports.deb and see what it is? e.g.

    $ file backports.deb
might give a clue. For me it says "Debian binary package (format 2.0)".

My guess is that it will tern out to be some HTML explaining that this file is not available via your proxy for some reason.

author
#128 lore raw

Re: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

Quoted 74 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Monday, October 12, 2015 4:57 PM
To: Hu, Robert <[email protected]>; 'Ian Jackson' <[email protected]>; '[email protected]' <[email protected]>
Subject: Re: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

Quoted 32 lines · Robert
Quoted 32 lines

-----Original Message-----
From: Hu, Robert
Sent: Monday, October 12, 2015 11:36 AM
To: Ian Jackson <[email protected]>; [email protected]
Cc: Ian Campbell <[email protected]>
Subject: RE: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

Quoted 24 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>
Subject: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

This is the second part of v14 Robert Ho's osstest patch series to support nested HVM tests.

It is also available here:
git://xenbits.xen.org/people/iwj/xen.git
http://xenbits.xen.org/git-http/people/iwj/xen.git
in wip.nested-hvm.v14.part1..wip.nested-hvm.v14

Compared to Robert's v13, which was passed to me by private email,
* I have rebased onto current osstest pretest;
* I have changed how selecthost() is told it's dealing with a nested host (in practice, L1 guest);
* There are a large number of minor cleanups;
* There are some new preparatory cleanup and admin patches;
* I have rewritten almost all of the commit messages.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

However, I have done only VERY LIMITED testing. Much of the code here
is UNTESTED since my changes. My testing was confined to:
* Verifying that my changes to cs-adjust-flight worked
* Checking that ad-hoc runs of ts-host-reboot and

ts-host-powercycle

Quoted 12 lines
Quoted 6 lines
Quoted 6 lines

seemed to work when a guest was specified on the command line.

Robert, you kindly volunteered to test a revised version of this series. I would appreciate if you would check that all of this still works as you expect. I expect there will be some bugs, perhaps even very silly bugs, introduced by me.

Quoted 6 lines

Sure.
Am I supposed to test Part 1 together with this Part 2? or each individually?
How's Part 1's status now? pass pretest and in production branch? or still
need test?

Quoted 8 lines

Seems Part 1 already in production tree. So I directly merged Part 2 on latest production master branch.

However, when I run 'standalone-reset', I get these errors [root@robert-ivt osstest]# ./standalone-reset
** need to generate d-i with firmware for armhf fetching initrd.gz

http://linux-ftp.sh.intel.com/pub/mirrors/debian//dists/wheezy/main/insta

Quoted 5 lines

ller-armhf/current/images/vexpress/netboot/initrd.gz => initrd.gz.new collecting firmware-bnx2
collecting backports initramfs-tools
collecting backports kernel
dpkg-deb: error: `backports.deb' is not a debian format archive

Quoted 6 lines

Can you look at the file backports.deb and see what it is? e.g.

    $ file backports.deb
might give a clue. For me it says "Debian binary package (format 2.0)".

My guess is that it will tern out to be some HTML explaining that this file is not available via your proxy for some reason.

Fixed. By moving out 'armhf' in my arch list.

Quoted 1 line
#129 lore raw

Re: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

Quoted 82 lines · Robert
Quoted 82 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Monday, October 12, 2015 4:57 PM
To: Hu, Robert <[email protected]>; 'Ian Jackson' <[email protected]>; '[email protected]' <[email protected]>
Subject: Re: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

Quoted 35 lines · Robert
Quoted 35 lines

-----Original Message-----
From: Hu, Robert
Sent: Monday, October 12, 2015 11:36 AM
To: Ian Jackson <[email protected]>; [email protected]
Cc: Ian Campbell <[email protected]>
Subject: RE: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

Quoted 27 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]

Quoted 1 line

Subject: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

This is the second part of v14 Robert Ho's osstest patch series to
support nested HVM tests.

It is also available here:
git://xenbits.xen.org/people/iwj/xen.git
http://xenbits.xen.org/git-http/people/iwj/xen.git
in wip.nested-hvm.v14.part1..wip.nested-hvm.v14

Compared to Robert's v13, which was passed to me by private email,
* I have rebased onto current osstest pretest;
* I have changed how selecthost() is told it's dealing with a nested host (in practice, L1 guest);
* There are a large number of minor cleanups;
* There are some new preparatory cleanup and admin patches;
* I have rewritten almost all of the commit messages.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

However, I have done only VERY LIMITED testing. Much of the code here
is UNTESTED since my changes. My testing was confined to:
* Verifying that my changes to cs-adjust-flight worked
* Checking that ad-hoc runs of ts-host-reboot and

ts-host-powercycle

Quoted 15 lines
Quoted 8 lines
Quoted 8 lines

seemed to work when a guest was specified on the command line.

Robert, you kindly volunteered to test a revised version of this series. I would appreciate if you would check that all of this still
works as you expect. I expect there will be some bugs, perhaps even
very silly bugs, introduced by me.

Quoted 7 lines

Sure.
Am I supposed to test Part 1 together with this Part 2? or each individually?
How's Part 1's status now? pass pretest and in production branch? or
still
need test?

Quoted 9 lines

Seems Part 1 already in production tree. So I directly merged Part 2 on
latest production master branch.

However, when I run 'standalone-reset', I get these errors [root@robert-ivt osstest]# ./standalone-reset
** need to generate d-i with firmware for armhf fetching initrd.gz

http://linux-ftp.sh.intel.com/pub/mirrors/debian//dists/wheezy/main/ins
ta

Quoted 5 lines

ller-armhf/current/images/vexpress/netboot/initrd.gz => initrd.gz.new collecting firmware-bnx2
collecting backports initramfs-tools
collecting backports kernel
dpkg-deb: error: `backports.deb' is not a debian format archive

Quoted 8 lines
Quoted 7 lines

Can you look at the file backports.deb and see what it is? e.g.

    $ file backports.deb
might give a clue. For me it says "Debian binary package (format 2.0)".

My guess is that it will tern out to be some HTML explaining that this file
is not available via your proxy for some reason.

Fixed. By moving out 'armhf' in my arch list.

Yes, I suppose Intel wouldn't be mirroring that ;-)

Ian.

author
#130 lore raw

Re: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

Quoted 100 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Monday, October 12, 2015 6:03 PM
To: Hu, Robert <[email protected]>; 'Ian Jackson' <[email protected]>; '[email protected]' <[email protected]>
Subject: Re: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

Quoted 8 lines · Robert
Quoted 8 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Monday, October 12, 2015 4:57 PM
To: Hu, Robert <[email protected]>; 'Ian Jackson' <[email protected]>; '[email protected]' <[email protected]>
Subject: Re: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

Quoted 8 lines
Quoted 8 lines · Robert
Quoted 7 lines
Quoted 7 lines

-----Original Message-----
From: Hu, Robert
Sent: Monday, October 12, 2015 11:36 AM
To: Ian Jackson <[email protected]>; [email protected]
Cc: Ian Campbell <[email protected]>
Subject: RE: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM

testing

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson

<[email protected]

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 1 line

Subject: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

This is the second part of v14 Robert Ho's osstest patch series to
support nested HVM tests.

It is also available here:
git://xenbits.xen.org/people/iwj/xen.git
http://xenbits.xen.org/git-http/people/iwj/xen.git
in wip.nested-hvm.v14.part1..wip.nested-hvm.v14

Compared to Robert's v13, which was passed to me by private email,
* I have rebased onto current osstest pretest;
* I have changed how selecthost() is told it's dealing with a nested host (in practice, L1 guest);
* There are a large number of minor cleanups;
* There are some new preparatory cleanup and admin patches;
* I have rewritten almost all of the commit messages.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

However, I have done only VERY LIMITED testing. Much of the

code

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

here
is UNTESTED since my changes. My testing was confined to:
* Verifying that my changes to cs-adjust-flight worked
* Checking that ad-hoc runs of ts-host-reboot and

ts-host-powercycle

Quoted 1 line
Quoted 1 line
Quoted 1 line

seemed to work when a guest was specified on the command

line.

Quoted 35 lines
Quoted 35 lines
Quoted 14 lines
Quoted 7 lines
Quoted 7 lines

Robert, you kindly volunteered to test a revised version of this series. I would appreciate if you would check that all of this still
works as you expect. I expect there will be some bugs, perhaps even
very silly bugs, introduced by me.

Quoted 7 lines

Sure.
Am I supposed to test Part 1 together with this Part 2? or each individually?
How's Part 1's status now? pass pretest and in production branch? or
still
need test?

Quoted 9 lines

Seems Part 1 already in production tree. So I directly merged Part 2 on
latest production master branch.

However, when I run 'standalone-reset', I get these errors [root@robert-ivt osstest]# ./standalone-reset
** need to generate d-i with firmware for armhf fetching initrd.gz

http://linux-ftp.sh.intel.com/pub/mirrors/debian//dists/wheezy/main/ins
ta

Quoted 5 lines

ller-armhf/current/images/vexpress/netboot/initrd.gz => initrd.gz.new collecting firmware-bnx2
collecting backports initramfs-tools
collecting backports kernel
dpkg-deb: error: `backports.deb' is not a debian format archive

Can you look at the file backports.deb and see what it is? e.g.

    $ file backports.deb
might give a clue. For me it says "Debian binary package (format 2.0)".
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines

My guess is that it will tern out to be some HTML explaining that this file
is not available via your proxy for some reason.

Fixed. By moving out 'armhf' in my arch list.

Yes, I suppose Intel wouldn't be mirroring that ;-)

Some other issue arises:
1. pax '-M norm', this option isn't support by my RHEL-distributed pax. Shall I simply omit it? or use '-t' substitute it? I tried the latter, seems working.
2. I initially run build-amd64 job, it will firstly re-install host, that's right. But after it is installed, it restarts host and again jump into a manual off-on loop. Do you know which part of code is controlling this?

2015-10-12 09:44:54 Z Forcing interface auto
tmp/t.osstest-host2.initrd.cpio:	 93.4% -- replaced with tmp/t.osstest-host2.initrd.cpio.gz
2015-10-12 09:44:54 Z using initrds: /var/lib/tftpboot//root/osstest/debian-installer/amd64/current-wheezy/initrd.gz tmp/t.osstest-host2.initrd.cpio.gz
2015-10-12 09:44:55 Z wrote /var/lib/tftpboot/pxelinux.cfg/C0A8C746 (stashed as osstest-host2-pxelinux.cfg+6)
2015-10-12 09:44:55 Z power-cycle: waiting 5s
2015-10-12 09:45:00 Z power: setting 1 for osstest-host2
### Manual power switch request: turn host osstest-host2 on ###
2015-10-12 09:45:09 Z fetch osstest-host2_preseed: waiting 350s...
2015-10-12 09:45:09 Z fetch osstest-host2_preseed: (none) (waiting) ...
2015-10-12 09:46:38 Z fetch osstest-host2_preseed: ok. (89s)
2015-10-12 09:46:38 Z wrote /var/lib/tftpboot/pxelinux.cfg/C0A8C746 (stashed as osstest-host2-pxelinux.cfg+7)
2015-10-12 09:46:38 Z await tcp osstest-host2 192.168.199.70 22: waiting 2400s...
2015-10-12 09:46:38 Z await tcp osstest-host2 192.168.199.70 22: nc: 256 Ncat: Version 6.40 ( http://nmap.org/ncat ) | Ncat: Connection refused. |  (waiting) ...
2015-10-12 09:57:36 Z await tcp osstest-host2 192.168.199.70 22: nc: 256 Ncat: Version 6.40 ( http://nmap.org/ncat ) | Ncat: Connection timed out. |  (waiting) ...
2015-10-12 09:57:56 Z await tcp osstest-host2 192.168.199.70 22: nc: 256 Ncat: Version 6.40 ( http://nmap.org/ncat ) | Ncat: No route to host. |  (waiting) ...
2015-10-12 09:59:14 Z await tcp osstest-host2 192.168.199.70 22: ok. (756s)
2015-10-12 09:59:14 Z executing ssh ... [email protected] lvdisplay --colon

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is 87:a2:7b:8a:73:b4:b4:57:30:15:ee:00:31:df:17:63.
Please contact your system administrator.
Add correct host key in tmp/t.known_hosts_standalone.build-amd64 to get rid of this message. Offending ECDSA key in tmp/t.known_hosts_standalone.build-amd64:1 Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks. Permission denied (publickey,password).

2015-10-12 09:59:14 Z command nonzero waitstatus 65280: timeout 60 ssh -o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=100 -o ServerAliveInterval=100 -o PasswordAuthentication=no -o ChallengeResponseAuthentication=no -o UserKnownHostsFile=tmp/t.known_hosts_standalone.build-amd64 [email protected] lvdisplay --colon
status 65280 at Osstest/TestSupport.pm line 410.
+ ./ts-host-install
2015-10-12 09:59:15 Z starting standalone.build-amd64
2015-10-12 09:59:15 Z setting arch=amd64
2015-10-12 09:59:15 Z setting build_lvextend_max=50
2015-10-12 09:59:15 Z setting enable_ovmf=true
2015-10-12 09:59:15 Z setting enable_xend=false
2015-10-12 09:59:15 Z setting enable_xsm=false
2015-10-12 09:59:15 Z setting host=osstest-host2
2015-10-12 09:59:15 Z setting host_hostflags=share-build-wheezy-amd64,arch-amd64,suite-wheezy,purpose-build
2015-10-12 09:59:15 Z setting revision_ovmf=
2015-10-12 09:59:15 Z setting revision_qemu=
2015-10-12 09:59:15 Z setting revision_qemuu=
2015-10-12 09:59:15 Z setting revision_seabios=
2015-10-12 09:59:15 Z setting revision_xen=
2015-10-12 09:59:15 Z setting tree_ovmf=git://vt-sync/ovmf.git/.git
2015-10-12 09:59:15 Z setting tree_qemu=git://vt-sync/qemu-xen-unstable.git
2015-10-12 09:59:15 Z setting tree_qemuu=git://vt-sync/qemu-upstream-unstable.git/.git
2015-10-12 09:59:15 Z setting tree_seabios=git://vt-sync/seabios.git/.git
2015-10-12 09:59:15 Z setting tree_xen=git://vt-sync/xen.git/.git
2015-10-12 09:59:15 Z TftpScope is default
2015-10-12 09:59:15 Z host host: selected osstest-host2 <unknown-ether> 192.168.199.70
2015-10-12 09:59:15 Z power: setting 0 for osstest-host2

### Manual power switch request: turn host osstest-host2 off ###

Quoted 2 lines

Ian.

#131 lore raw

Re: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

On Mon, 2015-10-12 at 10:23 +0000, Hu, Robert wrote:

(please can you trim your quotes)

Quoted 3 lines
Quoted 1 line

Some other issue arises:

1. pax '-M norm', this option isn't support by my RHEL-distributed pax. Shall I simply omit it? or use '-t' substitute it? I tried the latter, seems working.

The purpose of "-M norm" is to make the resulting archive deterministic, which I don't think -t achieves.

You can omit it locally I think, but I'd prefer to keep it in git. Hopefully newer RHEL will eventually support this.

Quoted 3 lines

2. I initially run build-amd64 job, it will firstly re-install host, that's right. But after it is installed, it restarts host and again jump into a manual off-on loop. Do you know which part of code is controlling this?

The post install check appears to have failed here:
[...]

Quoted 14 lines
2015-10-12 09:59:14 Z executing ssh ... [email protected] lvdisplay --colon
(quoted diff) +0 -0
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @

(quoted diff) +0 -0
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is 87:a2:7b:8a:73:b4:b4:57:30:15:ee:00:31:df:17:63.
Please contact your system administrator.
Add correct host key in tmp/t.known_hosts_standalone.build-amd64 to get rid of this message. Offending ECDSA key in tmp/t.known_hosts_standalone.build-amd64:1 Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks. Permission denied (publickey,password).

Quoted 2 lines
2015-10-12 09:59:14 Z command nonzero waitstatus 65280: timeout 60 ssh -o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=100 -o ServerAliveInterval=100 -o PasswordAuthentication=no -o ChallengeResponseAuthentication=no -o UserKnownHostsFile=tmp/t.known_hosts_standalone.build-amd64 [email protected] lvdisplay --colon

status 65280 at Osstest/TestSupport.pm line 410.

And therefore ts-host-install-twice is trying again, which begins with a power off.

ts-host-install-twice is a workaround for some issues with preseeding LVM in Debian installer when there is an existing LVM configuration on the host, but it also means that other failures end up having a second go (which will probably also fail).

I think the stuff about known hosts is benign, the "Permission denied (publickey,password)." is the real issue, it looks like either ssh cannot find your public key or your private key did not correctly get installed on the host.

Looking at patches 10-26 here I don't see anything which I would expect to effect host installation in this way. Neither ts-host-install nor Osstest/Debian.pm are touched here.

Ian.

author
#132 lore raw

Re: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

Quoted 14 lines

-----Original Message-----
From: Ian Campbell [mailto:[email protected]]
Sent: Monday, October 12, 2015 6:48 PM
To: Hu, Robert <[email protected]>; 'Ian Jackson' <[email protected]>; '[email protected]' <[email protected]>
Subject: Re: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

On Mon, 2015-10-12 at 10:23 +0000, Hu, Robert wrote:

(please can you trim your quotes)

Quoted 2 lines
Quoted 1 line

Some other issue arises:

1. pax '-M norm', this option isn't support by my RHEL-distributed pax. Shall

Quoted 8 lines

I

Quoted 1 line

simply omit it? or use '-t' substitute it? I tried the latter, seems working.

The purpose of "-M norm" is to make the resulting archive deterministic, which I don't think -t achieves.

You can omit it locally I think, but I'd prefer to keep it in git. Hopefully newer RHEL will eventually support this.

Yes I will keep it in git. Just change that locally to align with my environment.

Quoted 57 lines
Quoted 3 lines

2. I initially run build-amd64 job, it will firstly re-install host, that's right. But

after

Quoted 2 lines

it is installed, it restarts host and again jump into a manual off-on loop. Do you know which part of code is controlling this?

The post install check appears to have failed here:
[...]

Quoted 1 line
2015-10-12 09:59:14 Z executing ssh ... [email protected] lvdisplay

--colon

Quoted 1 line
(quoted diff) +0 -0
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@
Quoted 2 lines

@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @

(quoted diff) +0 -0
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@
Quoted 2 lines

IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle

attack)!

Quoted 5 lines

It is also possible that a host key has just been changed. The fingerprint for the ECDSA key sent by the remote host is 87:a2:7b:8a:73:b4:b4:57:30:15:ee:00:31:df:17:63.
Please contact your system administrator.
Add correct host key in tmp/t.known_hosts_standalone.build-amd64 to get

rid of this message.

Quoted 2 lines

Offending ECDSA key in tmp/t.known_hosts_standalone.build-amd64:1 Keyboard-interactive authentication is disabled to avoid man-in-the-middle

attacks.

Quoted 2 lines

Permission denied (publickey,password).

2015-10-12 09:59:14 Z command nonzero waitstatus 65280: timeout 60

ssh -o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=100 -o ServerAliveInterval=100 -o PasswordAuthentication=no -o ChallengeResponseAuthentication=no -o UserKnownHostsFile=tmp/t.known_hosts_standalone.build-amd64 [email protected] lvdisplay --colon

Quoted 1 line

status 65280 at Osstest/TestSupport.pm line 410.

And therefore ts-host-install-twice is trying again, which begins with a power off.

ts-host-install-twice is a workaround for some issues with preseeding LVM in Debian installer when there is an existing LVM configuration on the host, but it also means that other failures end up having a second go (which will probably also fail).

I think the stuff about known hosts is benign, the "Permission denied (publickey,password)." is the real issue, it looks like either ssh cannot find your public key or your private key did not correctly get installed on the host.

Looking at patches 10-26 here I don't see anything which I would expect to effect host installation in this way. Neither ts-host-install nor Osstest/Debian.pm are touched here.

Ian.

#133 lore raw

Re: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

In reply to Hu, Robert
Quoted 34 lines · Robert
Quoted 30 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>
Subject: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

This is the second part of v14 Robert Ho's osstest patch series to support nested HVM tests.

It is also available here:
git://xenbits.xen.org/people/iwj/xen.git
http://xenbits.xen.org/git-http/people/iwj/xen.git
in wip.nested-hvm.v14.part1..wip.nested-hvm.v14

Compared to Robert's v13, which was passed to me by private email,
* I have rebased onto current osstest pretest;
* I have changed how selecthost() is told it's dealing with a nested host (in practice, L1 guest);
* There are a large number of minor cleanups;
* There are some new preparatory cleanup and admin patches;
* I have rewritten almost all of the commit messages.

However, I have done only VERY LIMITED testing. Much of the code here is UNTESTED since my changes. My testing was confined to:
* Verifying that my changes to cs-adjust-flight worked
* Checking that ad-hoc runs of ts-host-reboot and ts-host-powercycle seemed to work when a guest was specified on the command line.

Quoted 4 lines

Robert, you kindly volunteered to test a revised version of this series. I would appreciate if you would check that all of this still works as you expect. I expect there will be some bugs, perhaps even very silly bugs, introduced by me.

Quoted 6 lines

Sure.
Am I supposed to test Part 1 together with this Part 2? or each individually?
How's Part 1's status now? pass pretest and in production branch? or still
need test?

AFAIK Part 1 is in production already. (For a little while, I think).

Ian.

author
#134 lore raw

Re: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

In reply to Ian Jackson
Quoted 35 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>
Subject: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

This is the second part of v14 Robert Ho's osstest patch series to support nested HVM tests.

It is also available here:
git://xenbits.xen.org/people/iwj/xen.git
http://xenbits.xen.org/git-http/people/iwj/xen.git
in wip.nested-hvm.v14.part1..wip.nested-hvm.v14

Compared to Robert's v13, which was passed to me by private email,
* I have rebased onto current osstest pretest;
* I have changed how selecthost() is told it's dealing with a nested host (in practice, L1 guest);
* There are a large number of minor cleanups;
* There are some new preparatory cleanup and admin patches;
* I have rewritten almost all of the commit messages.

However, I have done only VERY LIMITED testing. Much of the code here is UNTESTED since my changes. My testing was confined to:
* Verifying that my changes to cs-adjust-flight worked
* Checking that ad-hoc runs of ts-host-reboot and ts-host-powercycle seemed to work when a guest was specified on the command line.

Robert, you kindly volunteered to test a revised version of this series. I would appreciate if you would check that all of this still works as you expect. I expect there will be some bugs, perhaps even very silly bugs, introduced by me.

Quoted 8 lines

I noticed that this series lacks guest serial debug keys and log collection for the L1 guest, because there is no Osstest/Serial/guest.pm. I would appreciate it if you would provide one. I don't think it needs to actually collect any logs, because the L1 serial output log will be collected as part of the L0 log collection. But it ought to support sending debug keys to the L1 guest. When you have provided it you can (in the same patch) fix the corresponding todo' in selecthost, changing noop' to `guest'.

[Hu, Robert]

Hi Ian,
Are you sure would like me to add this part? I took glance at the module code (noop, xenuse, etc.), didn't quite understand.
I can imitate them for the Serial::guest.pm, but afraid will not that good.

Quoted 13 lines

Workflow:

Robert: I'm handing this (what I have called `part 2') over to you now.

When you make changes, feel free to either rebase, or to make fixup commits (perhaps in git-rebase -i --autosquash' format) on top. If you do the latter then you'll probably want to pass that to me as a git branch (via git push to xenbits or emailing me a git bundle), since squash!' and `fixup!' commits don't look good in email :-).

Quoted 4 lines

If you rebase, please put changes
v15: <description of what you did>
in the commit messages, as I have done myself in v14. Leave my v14 notes in place.

[Hu, Robert]

Now I've completed this part of work. Am I going to hand over the v15 bundle to you, with the above unresolved?
Current changes based on your patch:
* Some fixed (already get your confirmation) squashed into original patches, with v15 annotation.
* 2 fixes (not get your confirmation) are separated as !fixup patch for your clear review; actually only 1 explicit fixup patch, the other was by mistake squashed in but I made the annotation clearly.
* 2 more patches added, you've already been aware of:
Osstest/Testsupport.pm: change target's default kernkind to 'pvops' Osstest/Testsupport.pm: use get_target_property() for some host setup

Quoted 19 lines

Of course if you have any comments or queries about how I have done things, they would be very welcome.

Please do not rebase any of the commits in wip.nested-hvm.v14.part1. If you discover bugs in `part 1' please let us know as I have fed that into the osstest self-test mill with the expectation that it will go into production.

I do not expect you to test the changes to cs-adjust-flight. I have done that. Indeed they are not really related to the Nested HVM work and Ian C and I may pick them up in another series.

Ian Campbell: You probably want to defer re-reviewing this until Robert reports back.

author
#135 lore raw

Re: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

In reply to Ian Jackson
Quoted 51 lines

-----Original Message-----
From: Hu, Robert
Sent: Monday, November 2, 2015 11:44 AM
To: 'Ian Jackson' <[email protected]>; [email protected]
Cc: Ian Campbell <[email protected]>
Subject: RE: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

Quoted 35 lines

-----Original Message-----
From: Ian Jackson [mailto:[email protected]]
Sent: Saturday, September 26, 2015 3:15 AM
To: [email protected]
Cc: Hu, Robert <[email protected]>; Ian Campbell <[email protected]>; Ian Jackson <[email protected]>
Subject: [OSSTEST PATCH v14 PART 2 10-26/26] Nested HVM testing

This is the second part of v14 Robert Ho's osstest patch series to support nested HVM tests.

It is also available here:
git://xenbits.xen.org/people/iwj/xen.git
http://xenbits.xen.org/git-http/people/iwj/xen.git
in wip.nested-hvm.v14.part1..wip.nested-hvm.v14

Compared to Robert's v13, which was passed to me by private email,
* I have rebased onto current osstest pretest;
* I have changed how selecthost() is told it's dealing with a nested host (in practice, L1 guest);
* There are a large number of minor cleanups;
* There are some new preparatory cleanup and admin patches;
* I have rewritten almost all of the commit messages.

However, I have done only VERY LIMITED testing. Much of the code here is UNTESTED since my changes. My testing was confined to:
* Verifying that my changes to cs-adjust-flight worked
* Checking that ad-hoc runs of ts-host-reboot and ts-host-powercycle seemed to work when a guest was specified on the command line.

Robert, you kindly volunteered to test a revised version of this series. I would appreciate if you would check that all of this still works as you expect. I expect there will be some bugs, perhaps even very silly bugs, introduced by me.

Quoted 8 lines

I noticed that this series lacks guest serial debug keys and log collection for the L1 guest, because there is no Osstest/Serial/guest.pm. I would appreciate it if you would provide one. I don't think it needs to actually collect any logs, because the L1 serial output log will be collected as part of the L0 log collection. But it ought to support sending debug keys to the L1 guest. When you have provided it you can (in the same patch) fix the corresponding todo' in selecthost, changing noop' to `guest'.

Quoted 7 lines

[Hu, Robert]

Hi Ian,
Are you sure would like me to add this part? I took glance at the module code (noop, xenuse, etc.), didn't quite understand.
I can imitate them for the Serial::guest.pm, but afraid will not that good.

[Hu, Robert]

Don't quite understand this "\x18\x18\x18" part. What's it for? What's the meaning of $conswitch? I think it is not needed by guest case.

sub serial_fetch_logs ($) {
    my ($ho) = @_;

    logm("serial: requesting debug information from $ho->{Name}");

    foreach my $mo (@{ $ho->{SerialMethobjs} }) {
	$mo->request_debug("\x18\x18\x18",
			   "0HMQacdegimnrstuvz",
			   "q") or next;

...

Quoted 55 lines
Quoted 13 lines

Workflow:

Robert: I'm handing this (what I have called `part 2') over to you now.

When you make changes, feel free to either rebase, or to make fixup commits (perhaps in git-rebase -i --autosquash' format) on top. If you do the latter then you'll probably want to pass that to me as a git branch (via git push to xenbits or emailing me a git bundle), since squash!' and `fixup!' commits don't look good in email :-).

Quoted 4 lines

If you rebase, please put changes
v15: <description of what you did>
in the commit messages, as I have done myself in v14. Leave my v14 notes in place.

[Hu, Robert]

Now I've completed this part of work. Am I going to hand over the v15 bundle
to you, with the above unresolved?
Current changes based on your patch:
* Some fixed (already get your confirmation) squashed into original patches, with
v15 annotation.
* 2 fixes (not get your confirmation) are separated as !fixup patch for your clear
review; actually only 1 explicit fixup patch, the other was by mistake squashed in
but I made the annotation clearly.
* 2 more patches added, you've already been aware of:
Osstest/Testsupport.pm: change target's default kernkind to 'pvops' Osstest/Testsupport.pm: use get_target_property() for some host setup

Quoted 19 lines

Of course if you have any comments or queries about how I have done things, they would be very welcome.

Please do not rebase any of the commits in wip.nested-hvm.v14.part1. If you discover bugs in `part 1' please let us know as I have fed that into the osstest self-test mill with the expectation that it will go into production.

I do not expect you to test the changes to cs-adjust-flight. I have done that. Indeed they are not really related to the Nested HVM work and Ian C and I may pick them up in another series.

Ian Campbell: You probably want to defer re-reviewing this until Robert reports back.

#136 lore raw

Re: [OSSTEST PATCH v15] Nested HVM

In reply to Ian Jackson

Ian Jackson writes ("[OSSTEST PATCH v14 PART 1 0/9] Nested HVM preparation patches"):

Quoted 4 lines

This is the first part of Robert Hu's osstest patch series to support nested HVM tests. (v13 was passed to me as a git branch `under the table' by Robert.) I have rewritten the commit messages, refactored a few patches, and reordered the series slightly.

Robert has passed me a tentative version of v15 via `git bundle'. I have made it available here:

http://xenbits.xen.org/gitweb/?p=people/iwj/osstest.git;a=shortlog;h=refs/heads/wip.nested-v15

I'm going to fold in the fixup!s, review it, and probably adjust it somewhat. I will call the result v16, even though I am not sending a v15 patchbomb to the list.

Ian.