Skip to contentSkip to conversation

Configure the Calxeda fabric on host boot

PATCH Superseded Ian Campbell 2 participants 5 messages Feb 12, 2014

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Configure the Calxeda fabric on host boot

The fabric on the Calxeda midway boxes (marilith-* in osstest) does not learn mac address (at least not with the firmware we have, and with Calxeda folding this seems unlikely to get fixed). This means that guests do not get network connectivity unless their mac addresses explicitly registered with the fabric.

Registrations can be done with the bridge(8) tool from the iproute2 package which unfortunately is only present in Jessie+ and not in Wheezy. So I have done my own backport and placed it in $images/wheezy-iproute2 and arranged for it to be installed along with the transitional iproute package (from the same source) which is needed to satisfy various dependencies.

The registrations are ephemeral and need to be renewed on each reboot, so add the necessary commands to rc.local during ts-xen-install.

This required leaking a certain amount of the implementation of select_ether. Unless we want to do the bodge on every ts-guest-satart, reboot, migrate etc then this seems to be the best way.

In my testing I've set NRCXFabricMACs to 8. test-armhf-armhf-xl now gets past ts-guest-start and as far as the migration tests.

---
 Osstest/CXFabric.pm    | 86 ++++++++++++++++++++++++++++++++++++++++++++++++++
 Osstest/TestSupport.pm | 21 ++++++++----
 ts-xen-install         |  3 ++
 3 files changed, 103 insertions(+), 7 deletions(-)
 create mode 100644 Osstest/CXFabric.pm
Osstest/CXFabric.pm new file +86 -0
new file mode 100644
index 0000000..690afba
@@ -0,0 +1,86 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2014 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package Osstest::CXFabric;
+
+use strict;
+use warnings;
+
+use Osstest;
+use Osstest::TestSupport;
+
+BEGIN {
+    use Exporter ();
+    our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+    $VERSION     = 1.00;
+    @ISA         = qw(Exporter);
+    @EXPORT      = qw(
+                      setup_cxfabric
+                      );
+    %EXPORT_TAGS = ( );
+
+    @EXPORT_OK   = qw();
+}
+
+sub setup_cxfabric($)
+{
+    my ($ho) = @_;
+
+    my $nr = get_host_property($ho, 'NRCXFabricMACs', 0);
+    return unless $nr;
+
+    die unless $nr < 2**16;
+
+    my $prefix = ether_prefix($ho);
+    logm("Registering $nr MAC addresses with CX fabric using prefix $prefix");
+
+    if ( $ho->{Suite} =~ m/wheezy/ )
+    {
+        # iproute2 is not in Wheezy nor wheezy-backports. Use our own backport.
+        my $images = "$c{Images}/wheezy-iproute2";
+        my $ver = '3.12.0-1~xen70+1';
+
+        target_putfile_root($ho, 10, "$images/iproute_${ver}_all.deb", "iproute_${ver}_all.deb");
+        target_putfile_root($ho, 10, "$images/iproute2_${ver}_armhf.deb", "iproute2_${ver}_armhf.deb");
+
+        target_cmd_root($ho, <<END);
+dpkg -i iproute_${ver}_all.deb iproute2_${ver}_armhf.deb
+END
+    } else {
+        target_install_packages($ho, qw(iproute2));
+    }
+
+    my $banner = '# osstest: register potential guest MACs with CX fabric';
+    my $rclocal = "$banner\n";
+    # Osstest::TestSupport::select_ether allocates sequentially from $prefix:00:01
+    my $i = 0;
+    while ( $i++ < $nr ) {
+        $rclocal .= sprintf("bridge fdb add $prefix:%02x:%02x dev eth0\n",
+                            $i >> 8, $i & 0xff);
+    }
+
+    target_editfile_root($ho, '/etc/rc.local', sub {
+        my $had_banner = 0;
+        while (<::EI>) {
+            $had_banner = 1 if m/^$banner$/;
+            print ::EO $rclocal if m/^exit 0$/ && !$had_banner;
+            print ::EO;
+        }
+    });
+
+}
+
+1;
Osstest/TestSupport.pm +14 -7
index a513540..98eb172 100644
@@ -97,6 +97,8 @@ BEGIN {
                       await_webspace_fetch_byleaf create_webfile
                       file_link_contents get_timeout
                       setup_pxeboot setup_pxeboot_local host_pxefile
+
+                      ether_prefix
                       );
     %EXPORT_TAGS = ( );
 
@@ -1245,6 +1247,17 @@ sub target_choose_vg ($$) {
     return $bestvg;
 }
 
+sub ether_prefix($) {
+    my ($ho) = @_;
+    my $prefix = get_host_property($ho, 'gen-ether-prefix-base');
+    $prefix =~ m/^(\w+:\w+):(\w+):(\w+)$/ or die "$prefix ?";
+    my $lhs = $1;
+    my $pv = (hex($2)<<8) | (hex($3));
+    $pv ^= $mjobdb->gen_ether_offset($ho,$flight);
+    $prefix = sprintf "%s:%02x:%02x", $lhs, ($pv>>8)&0xff, $pv&0xff;
+    return $prefix;
+}
+
 sub select_ether ($$) {
     my ($ho,$vn) = @_;
     # must be run outside transaction
@@ -1252,13 +1265,7 @@ sub select_ether ($$) {
     return $ether if defined $ether;
 
     db_retry($flight,'running', $dbh_tests,[qw(flights)], sub {
-	my $prefix = get_host_property($ho, 'gen-ether-prefix-base');
-	$prefix =~ m/^(\w+:\w+):(\w+):(\w+)$/ or die "$prefix ?";
-	my $lhs = $1;
-	my $pv = (hex($2)<<8) | (hex($3));
-	$pv ^= $mjobdb->gen_ether_offset($ho,$flight);
-	$prefix = sprintf "%s:%02x:%02x", $lhs, ($pv>>8)&0xff, $pv&0xff;
-
+	my $prefix = ether_prefix($ho);
 	my $glob_ether = $mjobdb->jobdb_db_glob('*_ether');
 
         my $previous= $dbh_tests->selectrow_array(<<END, {}, $flight);
ts-xen-install +3 -0
index fc96516..903ed45 100755
@@ -22,6 +22,7 @@ use File::Path;
 use POSIX;
 use Osstest::Debian;
 use Osstest::TestSupport;
+use Osstest::CXFabric;
 
 my $checkmode= 0;
 
@@ -122,6 +123,8 @@ sub adjustconfig () {
     });
 
     target_cmd_root($ho, 'mkdir -p /var/log/xen/console');
+
+    setup_cxfabric($ho);
 }
 
 sub setupboot () {
Signature · 1 line
1.8.5.2
#1 lore raw

Ian Campbell writes ("[PATCH OSSTEST] Configure the Calxeda fabric on host boot"):

Quoted 5 lines

The fabric on the Calxeda midway boxes (marilith-* in osstest) does not learn mac address (at least not with the firmware we have, and with Calxeda folding this seems unlikely to get fixed). This means that guests do not get network connectivity unless their mac addresses explicitly registered with the fabric.

This looks mostly fine (well, FSVO "fine"!) to me, but:

Quoted 1 line

+ my $nr = get_host_property($ho, 'NRCXFabricMACs', 0);

This shouldn't be a host property, but a global config option (or even hardcoded). There should be a host flag for the bodge, probably "need-calxeda-bridge-fixup" or something.

And you could wrap these lines, while you're at it:

Osstest/CXFabric.pm line 57

Quoted 2 lines
Osstest/CXFabric.pm new file +2 -0
+        target_putfile_root($ho, 10, "$images/iproute_${ver}_all.deb", "iproute_${ver}_all.deb");
+        target_putfile_root($ho, 10, "$images/iproute2_${ver}_armhf.deb", "iproute2_${ver}_armhf.deb");

Or replace this extensive code with something more condensed, e.g.:

  +        my @debs = map { "iproute2_${ver}_${_}.deb" } qw(all armhf);
  +        target_putfile_root($ho, 10, "$images/$_", $_) foreach @debs;
  +        target_cmd_root($ho, "dpkg -i @debs");

Ian.

#2 lore raw
Quoted 7 lines · Ian Jackson

Ian Campbell writes ("[PATCH OSSTEST] Configure the Calxeda fabric on host boot"):

Quoted 5 lines

The fabric on the Calxeda midway boxes (marilith-* in osstest) does not learn mac address (at least not with the firmware we have, and with Calxeda folding this seems unlikely to get fixed). This means that guests do not get network connectivity unless their mac addresses explicitly registered with the fabric.

Quoted 7 lines

This looks mostly fine (well, FSVO "fine"!) to me, but:

Quoted 1 line

+ my $nr = get_host_property($ho, 'NRCXFabricMACs', 0);

This shouldn't be a host property, but a global config option (or even hardcoded). There should be a host flag for the bodge, probably "need-calxeda-bridge-fixup" or something.

equiv-marilith isn't suitable I take it?

Quoted 4 lines

And you could wrap these lines, while you're at it:

Quoted 2 lines
Osstest/CXFabric.pm new file +2 -0
+        target_putfile_root($ho, 10, "$images/iproute_${ver}_all.deb", "iproute_${ver}_all.deb");
+        target_putfile_root($ho, 10, "$images/iproute2_${ver}_armhf.deb", "iproute2_${ver}_armhf.deb");

Sure.

Quoted 3 lines

Or replace this extensive code with something more condensed, e.g.:

+ my @debs = map { "iproute2_${ver}_${_}.deb" } qw(all armhf);

This misses the iproute vs iproute2 distinction in the names, but I'll see if I can get something like this to work.

Quoted 4 lines

+ target_putfile_root($ho, 10, "$images/$_", $_) foreach @debs;
+ target_cmd_root($ho, "dpkg -i @debs");

Ian.

#3 lore raw

Ian Campbell writes ("Re: [PATCH OSSTEST] Configure the Calxeda fabric on host boot"):

Quoted 6 lines · Ian Jackson
Quoted 3 lines

This shouldn't be a host property, but a global config option (or even hardcoded). There should be a host flag for the bodge, probably "need-calxeda-bridge-fixup" or something.

equiv-marilith isn't suitable I take it?

Wow, that's _really_ bodgy.

Quoted 5 lines
Quoted 4 lines

And you could wrap these lines, while you're at it:

Quoted 2 lines
Osstest/CXFabric.pm new file +2 -0
+        target_putfile_root($ho, 10, "$images/iproute_${ver}_all.deb", "iproute_${ver}_all.deb");
+        target_putfile_root($ho, 10, "$images/iproute2_${ver}_armhf.deb", "iproute2_${ver}_armhf.deb");
Quoted 8 lines

Sure.

Quoted 3 lines

Or replace this extensive code with something more condensed, e.g.:

+ my @debs = map { "iproute2_${ver}_${_}.deb" } qw(all armhf);

This misses the iproute vs iproute2 distinction in the names, but I'll see if I can get something like this to work.

Oh so it does. Then you need to say
+ my @debs = "iproute_${ver}_all.deb", "iproute2_${ver}_armhf.deb";
but you can at least spell that out only once rather than three times as you do...

Ian.
(trying to turn everything into Common Lisp)

#4 lore raw
Quoted 9 lines · Ian Jackson

Ian Campbell writes ("Re: [PATCH OSSTEST] Configure the Calxeda fabric on host boot"):

Quoted 6 lines · Ian Jackson
Quoted 3 lines

This shouldn't be a host property, but a global config option (or even hardcoded). There should be a host flag for the bodge, probably "need-calxeda-bridge-fixup" or something.

equiv-marilith isn't suitable I take it?

Wow, that's _really_ bodgy.

I'll take that as a no ;-)

Quoted 11 lines
Quoted 11 lines
Quoted 4 lines

And you could wrap these lines, while you're at it:

Quoted 2 lines
Osstest/CXFabric.pm new file +2 -0
+        target_putfile_root($ho, 10, "$images/iproute_${ver}_all.deb", "iproute_${ver}_all.deb");
+        target_putfile_root($ho, 10, "$images/iproute2_${ver}_armhf.deb", "iproute2_${ver}_armhf.deb");

Sure.

Quoted 3 lines

Or replace this extensive code with something more condensed, e.g.:

+ my @debs = map { "iproute2_${ver}_${_}.deb" } qw(all armhf);

Quoted 7 lines
Quoted 2 lines

This misses the iproute vs iproute2 distinction in the names, but I'll see if I can get something like this to work.

Oh so it does. Then you need to say
+ my @debs = "iproute_${ver}_all.deb", "iproute2_${ver}_armhf.deb";
but you can at least spell that out only once rather than three times as you do...

Ack

Quoted 2 lines

Ian.
(trying to turn everything into Common Lisp)

(Ack)

Ian.