Skip to contentSkip to conversation

host: introduce modify_host

PATCH In review Roger Pau Monne 2 participants 12 messages Mar 11, 2020 – Mar 12, 2020

lore
Share & tools
lore
#0 lore raw

PATCH v5 1/5 host: introduce modify_host

Abstract the set_property checks and DB call into a helper.

No functional change.

---
Requested on IRC:
17:09:30 Diziet Also if it were me I would put the modify_host refactoring in its own nfc patch,
but I won't insist on that...
---
changes since v4:
- New in this version.

---
 Osstest/HostDB/Executive.pm | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)
Osstest/HostDB/Executive.pm +11 -5
index 7ffca6c4..d402bcac 100644
@@ -51,6 +51,16 @@ END
     }
 }
 
+sub modify_host ($$$) {
+    my ($hd, $ho, $query) = @_;
+    my $blessing = intended_blessing();
+
+    die "Attempting to modify host with intended blessing $blessing != real"
+        if $blessing ne "real";
+
+    db_retry($dbh_tests, [qw(resources)], $query);
+}
+
 sub set_property($$$$) {
     my ($hd, $ho, $prop, $val) = @_;
     my $rmq = $dbh_tests->prepare(<<END);
@@ -61,12 +71,8 @@ END
         INSERT INTO resource_properties (restype,resname,name,val)
                VALUES ('host', ?,?,?)
 END
-    my $blessing = intended_blessing();
-
-    die "Attempting to modify host props with intended blessing $blessing != real"
-        if $blessing ne "real";
 
-    db_retry($dbh_tests, [qw(resources)], sub {
+    modify_host($hd, $ho, sub {
         $rmq->execute($ho->{Name}, $prop);
         if (length $val) {
             $addq->execute($ho->{Name}, $prop, $val);
Signature · 7 lines
2.25.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH v5 2/5 host: introduce a helper to modify hostflags

Add a generic function to perform database changes related to a host flag and add a wrapper to TestSupport.

---
Changes since v4:
- Move addition of hostflag_putative_record to a different patch.
- Introduce a single helper in TestSupport: modify_host_flag.

Changes since v3:
- Introduce modify_flag instead of {set/remove}_flag.
- Introduce a generic modify_host helper.
- Split from patch 1.
---
Requested on IRC:
17:08:58 Diziet royger: I think your ts-examine-hostprops-save hunk in 2/ belongs in 1/ ? (Or in
a separate 1.5/ along with hostflag_putative_record.)

---
 Osstest/HostDB/Executive.pm | 17 +++++++++++++++++
 Osstest/HostDB/Static.pm    |  7 +++++++
 Osstest/TestSupport.pm      |  8 +++++++-
 3 files changed, 31 insertions(+), 1 deletion(-)
Osstest/HostDB/Executive.pm +17 -0
index d402bcac..a6dc4462 100644
@@ -96,6 +96,23 @@ END
     return $flags;
 }
 
+sub modify_flag ($$$$) {
+    my ($hd, $ho, $flag, $set) = @_;
+    my $rmq = $dbh_tests->prepare(<<END);
+        DELETE FROM hostflags WHERE hostname=? AND hostflag=?
+END
+    my $addq = $dbh_tests->prepare(<<END);
+        INSERT INTO hostflags (hostname,hostflag) VALUES (?,?)
+END
+
+    modify_host($hd, $ho, sub {
+        $rmq->execute($ho->{Name}, $flag);
+        if ($set) {
+            $addq->execute($ho->{Name}, $flag);
+        }
+    });
+}
+
 sub get_arch_platforms ($$$) {
     my ($hd, $blessing, $arch, $suite) = @_;
 
Osstest/HostDB/Static.pm +7 -0
index 0c6be3ee..d0669fb2 100644
@@ -72,6 +72,13 @@ sub get_flags ($$) { #method
     return $flags;
 }
 
+sub modify_flag ($$$$) {
+    my ($hd, $ho, $flag, $set) = @_;
+
+    die
+    "Cannot modify flags in standalone mode for $ho->{Name} $flag set: $set\n";
+}
+
 sub get_arch_platforms ($$$) {
     my ($hd, $blessing, $arch, $suite) = @_;
 
Osstest/TestSupport.pm +7 -1
index f49ed529..ceb6bb7b 100644
@@ -84,7 +84,7 @@ BEGIN {
                       get_target_property get_host_native_linux_console
                       hostnamepath hostnamepath_list set_runtime_hostflag
                       power_state power_cycle power_reboot_attempts
-                      serial_fetch_logs set_host_property
+                      serial_fetch_logs set_host_property modify_host_flag
                       propname_massage propname_check
                       hostprop_putative_record
          
@@ -1411,6 +1411,12 @@ sub hostprop_putative_record ($$$) {
     store_runvar("hostprop/$ho->{Ident}/$prop", $val);
 }
 
+sub modify_host_flag ($$$) {
+    my ($ho, $flag, $set) = @_;
+
+    $mhostdb->modify_flag($ho, $flag, $set);
+}
+
 sub get_target_property ($$;$);
 sub get_target_property ($$;$) {
     my ($ho, $prop, $defval) = @_;
Signature · 7 lines
2.25.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v5 2/5
#2 lore raw

Re: [Xen-devel] [PATCH OSSTEST v5 2/5] host: introduce a helper to modify hostflags

Roger Pau Monne writes ("[PATCH OSSTEST v5 2/5] host: introduce a helper to modify hostflags"):

Quoted 2 lines

Add a generic function to perform database changes related to a host flag and add a wrapper to TestSupport.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw

PATCH v5 3/5 ts-examine-hostprops-save: record hostflags also

In reply to Roger Pau Monne

Commit putative hotflags into the database if present on the runvars.

---
Changes since v4:
- New in this version.
---
Requested by Ian on IRC:
17:08:58 Diziet royger: I think your ts-examine-hostprops-save hunk in 2/ belongs in 1/ ? (Or in
a separate 1.5/ along with hostflag_putative_record.)

---
 Osstest/TestSupport.pm    |  8 +++++++-
 ts-examine-hostprops-save | 23 ++++++++++++++---------
 2 files changed, 21 insertions(+), 10 deletions(-)
Osstest/TestSupport.pm +7 -1
index ceb6bb7b..1c13e2af 100644
@@ -86,7 +86,7 @@ BEGIN {
                       power_state power_cycle power_reboot_attempts
                       serial_fetch_logs set_host_property modify_host_flag
                       propname_massage propname_check
-                      hostprop_putative_record
+                      hostprop_putative_record hostflag_putative_record
          
                       get_stashed open_unique_stashfile compress_stashed
                       dir_identify_vcs
@@ -1417,6 +1417,12 @@ sub modify_host_flag ($$$) {
     $mhostdb->modify_flag($ho, $flag, $set);
 }
 
+sub hostflag_putative_record ($$$) {
+    my ($ho, $prop, $set) = @_;
+
+    store_runvar("hostflag/$ho->{Ident}/$prop", !!$set);
+}
+
 sub get_target_property ($$;$);
 sub get_target_property ($$;$) {
     my ($ho, $prop, $defval) = @_;
ts-examine-hostprops-save +14 -9
index 55d23392..e50ea7fb 100755
@@ -27,20 +27,25 @@ tsreadconfig();
 
 our $blessing = intended_blessing();
 
-logm("setting host properties");
+logm("setting host properties and flags");
 
 # NB: in order to aid debug only attempt to save the host props on flights
 # with intended real blessing, for the rest just do a dry run.
 our $dry_run = $blessing ne "real";
-logm("not saving host props with intended blessing $blessing != real")
+logm("not saving host props/flags with intended blessing $blessing != real")
     if $dry_run;
 
 foreach my $k (sort keys %r) {
-    next unless $k =~ m/^hostprop\/([^\/]*)\/([^\/]*)$/;
-    my $ho = selecthost($1);
-    my $prop = $2;
-
-    logm("recording for $ho->{Name} $prop=$r{$k}");
-
-    set_host_property($ho, $prop, $r{$k}) if !$dry_run;
+    next unless $k =~ m/^host(prop|flag)\/([^\/]*)\/([^\/]*)$/;
+    my $type = $1;
+    my $ho = selecthost($2);
+    my $prop = $3;
+
+    if ($type eq "flag") {
+        logm("recording flag $prop set: $r{$k} for $ho->{Name}");
+        modify_host_flag($ho, $prop, !!$r{$k}) if !$dry_run;
+    } else {
+        logm("recording prop for $ho->{Name} $prop=$r{$k}");
+        set_host_property($ho, $prop, $r{$k}) if !$dry_run;
+    }
 }
Signature · 7 lines
2.25.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v5 3/5
#4 lore raw

Re: [Xen-devel] [PATCH OSSTEST v5 3/5] ts-examine-hostprops-save: record hostflags also

Roger Pau Monne writes ("[PATCH OSSTEST v5 3/5] ts-examine-hostprops-save: record hostflags also"):

Quoted 3 lines

Commit putative hotflags into the database if present on the runvars.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw

PATCH v5 4/5 examine: detect IOMMU availability and add it as a hostflag

In reply to Roger Pau Monne

Introduce a new test to check for iommu availability and add it as a hostflag if found.

---
Changes since v4:
- Split out code into separate patches.

Changes since v3:
- Fail if xl info command fails.

Changes since v2:
- Allow flags to be removed.
- Fix set_flag addition to HostBD/Static.pm.

---
 sg-run-job       |  1 +
 ts-examine-iommu | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 33 insertions(+)
 create mode 100755 ts-examine-iommu
sg-run-job +1 -0
index 7c58d4ba..f6bfdfd5 100755
@@ -679,6 +679,7 @@ proc examine-host-examine {install} {
     if {$ok} {
 	run-ts -.  =           ts-examine-serial-post + host
 	run-ts .   =           ts-examine-logs-save   + host
+	run-ts .   =           ts-examine-iommu       + host
 	run-ts .   =           ts-examine-hostprops-save
     }
 }
ts-examine-iommu new file +32 -0
new file mode 100755
index 00000000..099d4be5
@@ -0,0 +1,32 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2020 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+BEGIN { unshift @INC, qw(.); }
+use Osstest;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our ($whhost) = @ARGV;
+$whhost ||= 'host';
+our $ho= selecthost($whhost);
+our $info = target_cmd_output_root($ho, 'xl info', 10);
+our $has_iommu = $info =~ /directio/;
+
+logm("$ho->{Ident} iommu: $has_iommu");
+hostflag_putative_record($ho, "iommu", $has_iommu);
Signature · 7 lines
2.25.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#6 lore raw

Re: [Xen-devel] [PATCH OSSTEST v5 4/5] examine: detect IOMMU availability and add it as a hostflag

Roger Pau Monne writes ("[PATCH OSSTEST v5 4/5] examine: detect IOMMU availability and add it as a hostflag"):

Quoted 2 lines

Introduce a new test to check for iommu availability and add it as a hostflag if found.

...

Quoted 1 line

+our $has_iommu = $info =~ /directio/;

I think this regexp is too lax. For example, if the Xen command line had something mentioning directio' it would always say yes'...

This appears in virt_caps, right ? So I think you want

  +our $has_iommu = $info =~ /^virt_caps\s*:.*\sdirectio\s/m;

or some such. (Untested.)

Sorry for only spotting this now. Otherwise this LGTM.

Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#7 lore raw

Re: [Xen-devel] [PATCH OSSTEST v5 4/5] examine: detect IOMMU availability and add it as a hostflag

In reply to Roger Pau Monne

Roger Pau Monne writes ("[PATCH OSSTEST v5 4/5] examine: detect IOMMU availability and add it as a hostflag"):

Quoted 25 lines

Introduce a new test to check for iommu availability and add it as a hostflag if found.

---
Changes since v4:
- Split out code into separate patches.

Changes since v3:
- Fail if xl info command fails.

Changes since v2:
- Allow flags to be removed.
- Fix set_flag addition to HostBD/Static.pm.

---
 sg-run-job       |  1 +
 ts-examine-iommu | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 33 insertions(+)
 create mode 100755 ts-examine-iommu
sg-run-job +0 -0
index 7c58d4ba..f6bfdfd5 100755

sg-run-job proc examine-host-examine {install} { · line 683

Quoted 5 lines
sg-run-job +1 -0
@@ -679,6 +679,7 @@ proc examine-host-examine {install} {
     if {$ok} {
 	run-ts -.  =           ts-examine-serial-post + host
 	run-ts .   =           ts-examine-logs-save   + host
+	run-ts .   =           ts-examine-iommu       + host
 	run-ts .   =           ts-examine-hostprops-save

Thanks. I looked at your test flight and that caused me to notice that this ts-examine-iommu is in the wrong place, at least logically speaking. Log capture should come last, just before -hostprops-save.

So can you swap -logs-save and -iommu please ?

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#8 lore raw

PATCH v6 4/5 examine: detect IOMMU availability and add it as a hostflag

In reply to Roger Pau Monne

Introduce a new test to check for iommu availability and add it as a hostflag if found.

---
Changes since v5:
- Use a more restrictive regex.
- Place iommu test before log collection.

Changes since v4:
- Split out code into separate patches.

Changes since v3:
- Fail if xl info command fails.

Changes since v2:
- Allow flags to be removed.
- Fix set_flag addition to HostBD/Static.pm.

---
 sg-run-job       |  1 +
 ts-examine-iommu | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 33 insertions(+)
 create mode 100755 ts-examine-iommu
sg-run-job +1 -0
index 7c58d4ba..97011843 100755
@@ -678,6 +678,7 @@ proc examine-host-examine {install} {
     run-ts !broken capture-logs ts-logs-capture       + host
     if {$ok} {
 	run-ts -.  =           ts-examine-serial-post + host
+	run-ts .   =           ts-examine-iommu       + host
 	run-ts .   =           ts-examine-logs-save   + host
 	run-ts .   =           ts-examine-hostprops-save
     }
ts-examine-iommu new file +32 -0
new file mode 100755
index 00000000..3c28d46a
@@ -0,0 +1,32 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2020 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+BEGIN { unshift @INC, qw(.); }
+use Osstest;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our ($whhost) = @ARGV;
+$whhost ||= 'host';
+our $ho= selecthost($whhost);
+our $info = target_cmd_output_root($ho, 'xl info', 10);
+our $has_iommu = $info =~ /^virt_caps\s*:.*\s(hvm|pv)_directio\s.*/m;
+
+logm("$ho->{Ident} iommu: $has_iommu");
+hostflag_putative_record($ho, "iommu", $has_iommu);
Signature · 7 lines
2.25.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v6 4/5
#9 lore raw

Re: [Xen-devel] [PATCH OSSTEST v6 4/5] examine: detect IOMMU availability and add it as a hostflag

Roger Pau Monne writes ("[PATCH OSSTEST v6 4/5] examine: detect IOMMU availability and add it as a hostflag"):

Quoted 2 lines

Introduce a new test to check for iommu availability and add it as a hostflag if found.

Thanks,

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#10 lore raw

PATCH v5 5/5 make-flight: add dom0 PVH test

In reply to Roger Pau Monne

Add a dom0 PVH test, the test to be run is launching a PV guest (test-debian). Note the PVH dom0 tests are only enabled for Xen >= 4.13.

The runvar difference is:

+test-amd64-amd64-dom0pvh-xl-amd all_host_di_version 2020-02-10
+test-amd64-amd64-dom0pvh-xl-intel all_host_di_version 2020-02-10

+test-amd64-amd64-dom0pvh-xl-amd   all_host_suite      stretch
+test-amd64-amd64-dom0pvh-xl-intel all_host_suite      stretch
+test-amd64-amd64-dom0pvh-xl-amd   all_hostflags       arch-amd64,arch-xen-amd64,suite-stretch,purpose-test,hvm-amd,iommu
+test-amd64-amd64-dom0pvh-xl-intel all_hostflags       arch-amd64,arch-xen-amd64,suite-stretch,purpose-test,hvm-intel,iommu
+test-amd64-amd64-dom0pvh-xl-amd   arch                amd64
+test-amd64-amd64-dom0pvh-xl-intel arch                amd64
+test-amd64-amd64-dom0pvh-xl-amd   buildjob            build-amd64
+test-amd64-amd64-dom0pvh-xl-intel buildjob            build-amd64
+test-amd64-amd64-dom0pvh-xl-amd   debian_arch         amd64
+test-amd64-amd64-dom0pvh-xl-intel debian_arch         amd64
+test-amd64-amd64-dom0pvh-xl-amd   debian_kernkind     pvops
+test-amd64-amd64-dom0pvh-xl-intel debian_kernkind     pvops
+test-amd64-amd64-dom0pvh-xl-amd   debian_suite        stretch
+test-amd64-amd64-dom0pvh-xl-intel debian_suite        stretch
+test-amd64-amd64-dom0pvh-xl-amd   kernbuildjob        build-amd64-pvops
+test-amd64-amd64-dom0pvh-xl-intel kernbuildjob        build-amd64-pvops
+test-amd64-amd64-dom0pvh-xl-amd   kernkind            pvops
+test-amd64-amd64-dom0pvh-xl-intel kernkind            pvops
+test-amd64-amd64-dom0pvh-xl-amd   toolstack           xl
+test-amd64-amd64-dom0pvh-xl-intel toolstack           xl
+test-amd64-amd64-dom0pvh-xl-amd   xen_boot_append     dom0=pvh,verbose
+test-amd64-amd64-dom0pvh-xl-intel xen_boot_append     dom0=pvh,verbose

---
Changes since v1:
- Request hosts with iommu flag.

---
 make-flight | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
make-flight +24 -0
index b08431dc..48f164cc 100755
@@ -753,6 +753,16 @@ test_matrix_do_one () {
   *)                test_shim=y ;;
   esac
 
+  # PVH dom0 tests for versions >= 4.13 only
+  case "$xenbranch" in
+  xen-3.*-testing)  test_dom0pvh=n ;;
+  xen-4.?-testing)  test_dom0pvh=n ;;
+  xen-4.10-testing) test_dom0pvh=n ;;
+  xen-4.11-testing) test_dom0pvh=n ;;
+  xen-4.12-testing) test_dom0pvh=n ;;
+  *)                test_dom0pvh=y ;;
+  esac
+
   # xend PV guest test on x86 only
   if [ x$test_xend = xy -a \( $dom0arch = "i386" -o $dom0arch = "amd64" \) ]; then
     job_create_test test-$xenarch$kern-$dom0arch-pv test-debian xend \
@@ -861,6 +871,20 @@ test_matrix_do_one () {
 
   fi
 
+  if [ x$test_dom0pvh = xy -a $xenarch = amd64 -a $dom0arch = amd64 ]; then
+
+    for cpuvendor in amd intel; do
+
+      job_create_test test-$xenarch$kern-$dom0arch-dom0pvh-xl-$cpuvendor \
+                test-debian xl $xenarch $dom0arch $debian_runvars \
+                all_hostflags=$most_hostflags,hvm-$cpuvendor,iommu \
+                xen_boot_append='dom0=pvh,verbose'
+
+    done
+
+  fi
+
+
   if [ x$test_shim = xy -a $xenarch = amd64 ]; then
 
     job_create_test test-$xenarch$kern-$dom0arch-xl-pvshim \
Signature · 7 lines
2.25.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v5 1/5
#11 lore raw
In reply to Roger Pau Monne

Roger Pau Monne writes ("[PATCH OSSTEST v5 1/5] host: introduce modify_host"):

Quoted 5 lines

Abstract the set_property checks and DB call into a helper.

No functional change.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel