QoS settings not mapped correctly per pkey ?

Hello,

Following the QoS experiments I carried out yesterday, I wanted to set up 3 IP networks, each one bound to a particular pkey, in order to achieve QoS for each network.
Unfortunately, it seems that something is not mapped properly in the ULP layers (vlarb tables are fine).

The settings are as follows:

opensm.conf:
------------

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:8,1:1,2:1,3:4,4:0,5:0
qos_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

The corresponding VLArb tables are fine on both the server (pichu16) and the client (pichu22):

[root@pichu22 network-scripts]# smpquery vlarb -D 0
# VLArbitration tables: DR path slid 65535; dlid 65535; 0 port 0 LowCap 8 HighCap 8
# Low priority VL Arbitration Table:
VL : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x0 |0x0 |
WEIGHT: |0x8 |0x1 |0x1 |0x4 |0x0 |0x0 |0x0 |0x0 |
# High priority VL Arbitration Table:
VL : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x0 |0x0 |
WEIGHT: |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |

[root@pichu16 ~]# smpquery vlarb -D 0
# VLArbitration tables: DR path slid 65535; dlid 65535; 0 port 0 LowCap 8 HighCap 8
# Low priority VL Arbitration Table:
VL : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x0 |0x0 |
WEIGHT: |0x8 |0x1 |0x1 |0x4 |0x0 |0x0 |0x0 |0x0 |
# High priority VL Arbitration Table:
VL : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x0 |0x0 |
WEIGHT: |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |

partitions.conf:
---------------

default=0x7fff,ipoib : ALL=full;
ip_backbone=0x0001,ipoib : ALL=full;
ip_admin=0x0002,ipoib : ALL=full;

qos-policy.conf:
---------------

qos-ulps

    default                : 0 # default SL
    ipoib, pkey 0x7FFF     : 1 # IP with default pkey 0x7FFF
    ipoib, pkey 0x1        : 2 # backbone IP with pkey 0x1
    ipoib, pkey 0x2        : 3 # admin IP with pkey 0x2

end-qos-ulps

Assigned IP addresses (in /etc/hosts):
-------------------------------------

10.12.1.4       pichu16-ic0             # default IPoIB network, pkey 0x7FFF
10.13.1.4       pichu16-backbone        # IPoIB backbone network, pkey 0x1
10.14.1.4       pichu16-admin           # IPoIB admin network, pkey 0x2
10.12.1.10      pichu22-ic0             # default IPoIB network, pkey 0x7FFF
10.13.1.10      pichu22-backbone        # IPoIB backbone network, pkey 0x1
10.14.1.10      pichu22-admin           # IPoIB admin network, pkey 0x2

Note that the netmask is /16, so the -ic0, -backbone and -admin networks cannot see each other.

IPoIB settings on server side:
------------------------------

[root@pichu16 ~]# tail -n 5 /etc/sysconfig/network-scripts/ifcfg-ib0* ==> /etc/sysconfig/network-scripts/ifcfg-ib0 <== BOOTPROTO=static
IPADDR=10.12.1.4
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2044

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8001 <== BOOTPROTO=static
IPADDR=10.13.1.4
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2044

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8002 <== BOOTPROTO=static
IPADDR=10.14.1.4
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2044

[root@pichu16 ~]# ip addr show ib0
4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast state UP qlen 256
link/infiniband
80:00:00:48:fe:80:00:00:00:00:00:00:2c:90:00:10:0d:00:05:6d brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

    inet 10.12.1.4/16 brd 10.12.255.255 scope global ib0
    inet 10.13.1.4/16 brd 10.13.255.255 scope global ib0
    inet 10.14.1.4/16 brd 10.14.255.255 scope global ib0
    inet6 fe80::2e90:10:d00:56d/64 scope link
       valid_lft forever preferred_lft forever

IPoIB settings on client side:
------------------------------

[root@pichu22 ~]# tail -n 5 /etc/sysconfig/network-scripts/ifcfg-ib0* ==> /etc/sysconfig/network-scripts/ifcfg-ib0 <== BOOTPROTO=static
IPADDR=10.12.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2044

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8001 <== BOOTPROTO=static
IPADDR=10.13.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2044

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8002 <== BOOTPROTO=static
IPADDR=10.14.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2044

[root@pichu22 ~]# ip addr show ib0
48: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast state UP qlen 256
link/infiniband
80:00:00:48:fe:80:00:00:00:00:00:00:2c:90:00:10:0d:00:06:79 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

    inet 10.12.1.10/16 brd 10.12.255.255 scope global ib0
    inet 10.13.1.10/16 brd 10.13.255.255 scope global ib0
    inet 10.14.1.10/16 brd 10.14.255.255 scope global ib0
    inet6 fe80::2e90:10:d00:679/64 scope link
       valid_lft forever preferred_lft forever

Iperf servers on server side:
-----------------------------

Quoting from iperf help:
-B, --bind <host> bind to <host>, an interface or multicast address
-s, --server run in server mode

Each iperf server is bound to a dedicated interface as follows:

[root@pichu16 ~]# iperf -s -B pichu16-backbone
[root@pichu16 ~]# iperf -s -B pichu16-admin
[root@pichu16 ~]# iperf -s -B pichu16-ic0

Iperf clients on client side:
-----------------------------

Quoting from iperf help:

  -c, --client    <host>   run in client mode, connecting to <host>
  -t, --time      #        time in seconds to transmit for (default 10 secs)

And each iperf client talks to the corresponding iperf server:

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t 100 2>&1; done | grep Gbits/sec

[  3]  0.0-100.0 sec  64.6 GBytes  5.55 Gbits/sec
[  3]  0.0-100.0 sec  64.5 GBytes  5.54 Gbits/sec
[  3]  0.0-100.0 sec  60.5 GBytes  5.20 Gbits/sec
[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-backbone
-t 100 2>&1; done | grep Gbits/sec
[  3]  0.0-100.0 sec  64.8 GBytes  5.57 Gbits/sec
[  3]  0.0-100.0 sec  56.7 GBytes  4.87 Gbits/sec
[  3]  0.0-100.0 sec  59.7 GBytes  5.13 Gbits/sec
[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-admin -t
100 2>&1; done | grep Gbits/sec
[  3]  0.0-100.0 sec  57.3 GBytes  4.92 Gbits/sec
[  3]  0.0-100.0 sec  61.6 GBytes  5.29 Gbits/sec
[  3]  0.0-100.0 sec  62.7 GBytes  5.38 Gbits/sec

Given the VLarb weights assigned (1 for *-ic0 on VL1, 1 for *-backbone on VL2 and 4 for *-admin on VL3), we would expect different b/w figures for the *-admin network.
As we can see, all iperf values are the same, showing that QoS is not enforced on a per pkey basis.
It seems to me that something is not mapped properly in the ULP layers. Could anyone tell me if I'm wrong here ? If not, is that a known issue ?

Thanks for your help,

Vincent

Hi Vincent,

Quoted 10 lines · Vincent Ficet

Hello,

Following the QoS experiments I carried out yesterday, I wanted to set up 3 IP networks, each one bound to a particular pkey, in order to achieve QoS for each network.
Unfortunately, it seems that something is not mapped properly in the ULP layers (vlarb tables are fine).

The settings are as follows:

Quoted 8 lines

opensm.conf:
------------

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:8,1:1,2:1,3:4,4:0,5:0
qos_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

Please check section 7 of the QoS_management_in_OpenSM.txt doc. It explains what exactly is the meaning of the values in the VLArb table. It also has explanation of the problem that you're seeing. Quoting from there:

"Keep in mind that ports usually transmit packets of
size equal to MTU. For instance, for 4KB MTU a single packet will require 64 credits, so in order to achieve effective VL arbitration for packets of 4KB MTU, the weighting values for each VL should be multiples of 64."

-- Yevgeny

Quoted 179 lines

The corresponding VLArb tables are fine on both the server (pichu16) and the client (pichu22):

[root@pichu22 network-scripts]# smpquery vlarb -D 0
# VLArbitration tables: DR path slid 65535; dlid 65535; 0 port 0 LowCap 8 HighCap 8
# Low priority VL Arbitration Table:
VL : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x0 |0x0 |
WEIGHT: |0x8 |0x1 |0x1 |0x4 |0x0 |0x0 |0x0 |0x0 |
# High priority VL Arbitration Table:
VL : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x0 |0x0 |
WEIGHT: |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |

[root@pichu16 ~]# smpquery vlarb -D 0
# VLArbitration tables: DR path slid 65535; dlid 65535; 0 port 0 LowCap 8 HighCap 8
# Low priority VL Arbitration Table:
VL : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x0 |0x0 |
WEIGHT: |0x8 |0x1 |0x1 |0x4 |0x0 |0x0 |0x0 |0x0 |
# High priority VL Arbitration Table:
VL : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x0 |0x0 |
WEIGHT: |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |

partitions.conf:
---------------

default=0x7fff,ipoib : ALL=full;
ip_backbone=0x0001,ipoib : ALL=full;
ip_admin=0x0002,ipoib : ALL=full;

qos-policy.conf:
---------------

qos-ulps

    default                : 0 # default SL
    ipoib, pkey 0x7FFF     : 1 # IP with default pkey 0x7FFF
    ipoib, pkey 0x1        : 2 # backbone IP with pkey 0x1
    ipoib, pkey 0x2        : 3 # admin IP with pkey 0x2

end-qos-ulps

Assigned IP addresses (in /etc/hosts):
-------------------------------------

10.12.1.4 pichu16-ic0 # default IPoIB network, pkey
0x7FFF
10.13.1.4 pichu16-backbone # IPoIB backbone network,
pkey 0x1

10.14.1.4       pichu16-admin           # IPoIB admin network, pkey 0x2
10.12.1.10      pichu22-ic0             # default IPoIB network, pkey

0x7FFF
10.13.1.10 pichu22-backbone # IPoIB backbone network,
pkey 0x1
10.14.1.10 pichu22-admin # IPoIB admin network, pkey 0x2

Note that the netmask is /16, so the -ic0, -backbone and -admin networks cannot see each other.

IPoIB settings on server side:
------------------------------

[root@pichu16 ~]# tail -n 5 /etc/sysconfig/network-scripts/ifcfg-ib0* ==> /etc/sysconfig/network-scripts/ifcfg-ib0 <== BOOTPROTO=static
IPADDR=10.12.1.4
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2044

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8001 <== BOOTPROTO=static
IPADDR=10.13.1.4
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2044

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8002 <== BOOTPROTO=static
IPADDR=10.14.1.4
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2044

[root@pichu16 ~]# ip addr show ib0
4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast state UP qlen 256
link/infiniband
80:00:00:48:fe:80:00:00:00:00:00:00:2c:90:00:10:0d:00:05:6d brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

    inet 10.12.1.4/16 brd 10.12.255.255 scope global ib0
    inet 10.13.1.4/16 brd 10.13.255.255 scope global ib0
    inet 10.14.1.4/16 brd 10.14.255.255 scope global ib0
    inet6 fe80::2e90:10:d00:56d/64 scope link
       valid_lft forever preferred_lft forever

IPoIB settings on client side:
------------------------------

[root@pichu22 ~]# tail -n 5 /etc/sysconfig/network-scripts/ifcfg-ib0* ==> /etc/sysconfig/network-scripts/ifcfg-ib0 <== BOOTPROTO=static
IPADDR=10.12.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2044

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8001 <== BOOTPROTO=static
IPADDR=10.13.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2044

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8002 <== BOOTPROTO=static
IPADDR=10.14.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2044

[root@pichu22 ~]# ip addr show ib0
48: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast state UP qlen 256
link/infiniband
80:00:00:48:fe:80:00:00:00:00:00:00:2c:90:00:10:0d:00:06:79 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

    inet 10.12.1.10/16 brd 10.12.255.255 scope global ib0
    inet 10.13.1.10/16 brd 10.13.255.255 scope global ib0
    inet 10.14.1.10/16 brd 10.14.255.255 scope global ib0
    inet6 fe80::2e90:10:d00:679/64 scope link
       valid_lft forever preferred_lft forever

Iperf servers on server side:
-----------------------------

Quoting from iperf help:
-B, --bind <host> bind to <host>, an interface or multicast
address
-s, --server run in server mode

Each iperf server is bound to a dedicated interface as follows:

[root@pichu16 ~]# iperf -s -B pichu16-backbone
[root@pichu16 ~]# iperf -s -B pichu16-admin
[root@pichu16 ~]# iperf -s -B pichu16-ic0

Iperf clients on client side:
-----------------------------

Quoting from iperf help:

  -c, --client    <host>   run in client mode, connecting to <host>
  -t, --time      #        time in seconds to transmit for (default

10 secs)

And each iperf client talks to the corresponding iperf server:

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t 100 2>&1; done | grep Gbits/sec

[  3]  0.0-100.0 sec  64.6 GBytes  5.55 Gbits/sec
[  3]  0.0-100.0 sec  64.5 GBytes  5.54 Gbits/sec
[  3]  0.0-100.0 sec  60.5 GBytes  5.20 Gbits/sec
[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-backbone
-t 100 2>&1; done | grep Gbits/sec
[  3]  0.0-100.0 sec  64.8 GBytes  5.57 Gbits/sec
[  3]  0.0-100.0 sec  56.7 GBytes  4.87 Gbits/sec
[  3]  0.0-100.0 sec  59.7 GBytes  5.13 Gbits/sec
[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-admin -t
100 2>&1; done | grep Gbits/sec
[  3]  0.0-100.0 sec  57.3 GBytes  4.92 Gbits/sec
[  3]  0.0-100.0 sec  61.6 GBytes  5.29 Gbits/sec
[  3]  0.0-100.0 sec  62.7 GBytes  5.38 Gbits/sec

Given the VLarb weights assigned (1 for *-ic0 on VL1, 1 for *-backbone on VL2 and 4 for *-admin on VL3), we would expect different b/w figures for the *-admin network.
As we can see, all iperf values are the same, showing that QoS is not enforced on a per pkey basis.
It seems to me that something is not mapped properly in the ULP layers. Could anyone tell me if I'm wrong here ? If not, is that a known issue ?

Quoted 8 lines

Thanks for your help,

Vincent

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Yevgeny,

Quoted 26 lines

Hi Vincent,

Quoted 10 lines · Vincent Ficet

Hello,

Following the QoS experiments I carried out yesterday, I wanted to set up 3 IP networks, each one bound to a particular pkey, in order to achieve QoS for each network.
Unfortunately, it seems that something is not mapped properly in the ULP layers (vlarb tables are fine).

The settings are as follows:

Quoted 8 lines

opensm.conf:
------------

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:8,1:1,2:1,3:4,4:0,5:0
qos_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

Please check section 7 of the QoS_management_in_OpenSM.txt doc. It explains what exactly is the meaning of the values in the VLArb table. It also has explanation of the problem that you're seeing. Quoting from there:

Quoted 6 lines

"Keep in mind that ports usually transmit packets of
size equal to MTU. For instance, for 4KB MTU a single packet will require 64 credits, so in order to achieve effective VL arbitration for packets of 4KB MTU, the weighting values for each VL should be multiples of 64."

OK, I see the point.

To check that it works as you said. we changed the IPoIB MTU from 2044 to 2000 in order to make sure that it fits into the IB MTU. which is set to 2K on our cluster.
In theory, such a 2K packet would require 32 packets (credits) of 64 bytes.

We changed the vlarb tables with increments of 32 (for VL 1,2,3):

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:8,1:32,2:64,3:96,4:0,5:0
qos_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

and we also tried increments of 64:

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:8,1:64,2:128,3:192,4:0,5:0
qos_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

But still, it does not make any difference:

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t
20 2>&1; done | grep Gbits/sec

[  3]  0.0-20.0 sec  13.0 GBytes  5.57 Gbits/sec
[  3]  0.0-20.0 sec  12.9 GBytes  5.53 Gbits/sec
[  3]  0.0-20.0 sec  12.0 GBytes  5.17 Gbits/sec

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-backbone -t 20 2>&1; done | grep Gbits/sec

[  3]  0.0-20.0 sec  13.1 GBytes  5.61 Gbits/sec
[  3]  0.0-20.0 sec  11.9 GBytes  5.09 Gbits/sec
[  3]  0.0-20.0 sec  9.43 GBytes  4.05 Gbits/sec

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-admin -t 20 2>&1; done | grep Gbits/sec

[  3]  0.0-20.0 sec  10.5 GBytes  4.50 Gbits/sec
[  3]  0.0-20.0 sec  12.3 GBytes  5.28 Gbits/sec
[  3]  0.0-20.0 sec  12.0 GBytes  5.15 Gbits/sec

Any other idea ?

Thanks for your help.

Vincent

Quoted 196 lines

-- Yevgeny

Quoted 187 lines

The corresponding VLArb tables are fine on both the server (pichu16) and the client (pichu22):

[root@pichu22 network-scripts]# smpquery vlarb -D 0
# VLArbitration tables: DR path slid 65535; dlid 65535; 0 port 0 LowCap 8 HighCap 8
# Low priority VL Arbitration Table:
VL : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x0 |0x0 |
WEIGHT: |0x8 |0x1 |0x1 |0x4 |0x0 |0x0 |0x0 |0x0 |
# High priority VL Arbitration Table:
VL : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x0 |0x0 |
WEIGHT: |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |

[root@pichu16 ~]# smpquery vlarb -D 0
# VLArbitration tables: DR path slid 65535; dlid 65535; 0 port 0 LowCap 8 HighCap 8
# Low priority VL Arbitration Table:
VL : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x0 |0x0 |
WEIGHT: |0x8 |0x1 |0x1 |0x4 |0x0 |0x0 |0x0 |0x0 |
# High priority VL Arbitration Table:
VL : |0x0 |0x1 |0x2 |0x3 |0x4 |0x5 |0x0 |0x0 |
WEIGHT: |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |

partitions.conf:
---------------

default=0x7fff,ipoib : ALL=full;
ip_backbone=0x0001,ipoib : ALL=full;
ip_admin=0x0002,ipoib : ALL=full;

qos-policy.conf:
---------------

qos-ulps

    default                : 0 # default SL
    ipoib, pkey 0x7FFF     : 1 # IP with default pkey 0x7FFF
    ipoib, pkey 0x1        : 2 # backbone IP with pkey 0x1
    ipoib, pkey 0x2        : 3 # admin IP with pkey 0x2

end-qos-ulps

Assigned IP addresses (in /etc/hosts):
-------------------------------------

10.12.1.4 pichu16-ic0 # default IPoIB network, pkey
0x7FFF
10.13.1.4 pichu16-backbone # IPoIB backbone network,
pkey 0x1

10.14.1.4       pichu16-admin           # IPoIB admin network, pkey 0x2
10.12.1.10      pichu22-ic0             # default IPoIB network, pkey

0x7FFF
10.13.1.10 pichu22-backbone # IPoIB backbone network,
pkey 0x1
10.14.1.10 pichu22-admin # IPoIB admin network, pkey 0x2

Note that the netmask is /16, so the -ic0, -backbone and -admin networks cannot see each other.

IPoIB settings on server side:
------------------------------

[root@pichu16 ~]# tail -n 5 /etc/sysconfig/network-scripts/ifcfg-ib0* ==> /etc/sysconfig/network-scripts/ifcfg-ib0 <== BOOTPROTO=static
IPADDR=10.12.1.4
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2044

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8001 <== BOOTPROTO=static
IPADDR=10.13.1.4
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2044

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8002 <== BOOTPROTO=static
IPADDR=10.14.1.4
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2044

[root@pichu16 ~]# ip addr show ib0
4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast state UP qlen 256
link/infiniband
80:00:00:48:fe:80:00:00:00:00:00:00:2c:90:00:10:0d:00:05:6d brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

    inet 10.12.1.4/16 brd 10.12.255.255 scope global ib0
    inet 10.13.1.4/16 brd 10.13.255.255 scope global ib0
    inet 10.14.1.4/16 brd 10.14.255.255 scope global ib0
    inet6 fe80::2e90:10:d00:56d/64 scope link
       valid_lft forever preferred_lft forever

IPoIB settings on client side:
------------------------------

[root@pichu22 ~]# tail -n 5 /etc/sysconfig/network-scripts/ifcfg-ib0* ==> /etc/sysconfig/network-scripts/ifcfg-ib0 <== BOOTPROTO=static
IPADDR=10.12.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2044

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8001 <== BOOTPROTO=static
IPADDR=10.13.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2044

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8002 <== BOOTPROTO=static
IPADDR=10.14.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2044

[root@pichu22 ~]# ip addr show ib0
48: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast state UP qlen 256
link/infiniband
80:00:00:48:fe:80:00:00:00:00:00:00:2c:90:00:10:0d:00:06:79 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

    inet 10.12.1.10/16 brd 10.12.255.255 scope global ib0
    inet 10.13.1.10/16 brd 10.13.255.255 scope global ib0
    inet 10.14.1.10/16 brd 10.14.255.255 scope global ib0
    inet6 fe80::2e90:10:d00:679/64 scope link
       valid_lft forever preferred_lft forever

Iperf servers on server side:
-----------------------------

Quoting from iperf help:
-B, --bind <host> bind to <host>, an interface or multicast
address
-s, --server run in server mode

Each iperf server is bound to a dedicated interface as follows:

[root@pichu16 ~]# iperf -s -B pichu16-backbone
[root@pichu16 ~]# iperf -s -B pichu16-admin
[root@pichu16 ~]# iperf -s -B pichu16-ic0

Iperf clients on client side:
-----------------------------

Quoting from iperf help:

  -c, --client    <host>   run in client mode, connecting to <host>
  -t, --time      #        time in seconds to transmit for (default

10 secs)

And each iperf client talks to the corresponding iperf server:

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t 100 2>&1; done | grep Gbits/sec

[  3]  0.0-100.0 sec  64.6 GBytes  5.55 Gbits/sec
[  3]  0.0-100.0 sec  64.5 GBytes  5.54 Gbits/sec
[  3]  0.0-100.0 sec  60.5 GBytes  5.20 Gbits/sec
[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-backbone
-t 100 2>&1; done | grep Gbits/sec
[  3]  0.0-100.0 sec  64.8 GBytes  5.57 Gbits/sec
[  3]  0.0-100.0 sec  56.7 GBytes  4.87 Gbits/sec
[  3]  0.0-100.0 sec  59.7 GBytes  5.13 Gbits/sec
[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-admin -t
100 2>&1; done | grep Gbits/sec
[  3]  0.0-100.0 sec  57.3 GBytes  4.92 Gbits/sec
[  3]  0.0-100.0 sec  61.6 GBytes  5.29 Gbits/sec
[  3]  0.0-100.0 sec  62.7 GBytes  5.38 Gbits/sec

Given the VLarb weights assigned (1 for *-ic0 on VL1, 1 for *-backbone on VL2 and 4 for *-admin on VL3), we would expect different b/w figures for the *-admin network.
As we can see, all iperf values are the same, showing that QoS is not enforced on a per pkey basis.
It seems to me that something is not mapped properly in the ULP layers. Could anyone tell me if I'm wrong here ? If not, is that a known issue ?

Thanks for your help,

Vincent

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

OK, so there are three possible reasons that I can think of:
1. Something is wrong in the configuration.
2. The application does not saturate the link, thus QoS and the whole VL arbitration thing doesn't kick in.
3. There's some bug, somewhere.

Let's start with reason no. 1.
Please shut off each of the SLs one by one, and make sure that the application gets zero BW on these SLs. You can do it by mapping SL to VL15:

qos_sl2vl 0,15,2,3,4,5,6,7,8,9,10,11,12,13,14,15

and then
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

and then
qos_sl2vl 0,1,2,15,4,5,6,7,8,9,10,11,12,13,14,15

If this part works well, then we will continue to reason no. 2.

Yevgeny,

Quoted 11 lines

OK, so there are three possible reasons that I can think of:
1. Something is wrong in the configuration.
2. The application does not saturate the link, thus QoS and the whole VL arbitration thing doesn't kick in.
3. There's some bug, somewhere.

Let's start with reason no. 1.
Please shut off each of the SLs one by one, and make sure that the application gets zero BW on these SLs. You can do it by mapping SL to VL15:

Quoted 1 line

qos_sl2vl 0,15,2,3,4,5,6,7,8,9,10,11,12,13,14,15

If I shut down this SL by moving it to VL15, the interfaces stop pinging. This is probably because some IPoIB multicast traffic gets cut off for pkey 0x7fff .. ?

So no results for this one.

Quoted 3 lines

and then
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

With this setup, and the following QoS settings:

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:1,1:64,2:128,3:192,4:0,5:0
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

I get roughly the same values for SL 1 to SL3:

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.0 sec  6.15 GBytes  5.28 Gbits/sec
[SUM]  0.0-10.0 sec  6.00 GBytes  5.16 Gbits/sec
[SUM]  0.0-10.1 sec  5.38 GBytes  4.59 Gbits/sec

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-backbone -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.0 sec  6.09 GBytes  5.23 Gbits/sec
[SUM]  0.0-10.0 sec  6.41 GBytes  5.51 Gbits/sec
[SUM]  0.0-10.0 sec  4.72 GBytes  4.05 Gbits/sec

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-admin -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.1 sec  6.96 GBytes  5.92 Gbits/sec
[SUM]  0.0-10.1 sec  5.89 GBytes  5.00 Gbits/sec
[SUM]  0.0-10.0 sec  5.35 GBytes  4.58 Gbits/sec

Quoted 2 lines

and then
qos_sl2vl 0,1,2,15,4,5,6,7,8,9,10,11,12,13,14,15

Same results as the previous 0,1,15,3,... SL2vl mapping.

Quoted 2 lines

If this part works well, then we will continue to reason no. 2.

In the above tests, I used -P8 to force 8 threads on the client side for each test.
I have one quad core CPU(Intel E55400).
This makes 24 iperf threads on 4 cores, which __should__ be fine (well I suppose ...)

And regarding reason #3. I still get the error I got yesterday, which you told me was not important because the SL's set in partitions.conf would override what was read from qos-policy.conf in the first place.

Nov 25 13:13:05 664690 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
AC15: pkey 0x0002 in match rule - overriding partition SL (0) with QoS
Level SL (3)
Nov 25 13:13:05 664681 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
AC15: pkey 0x0001 in match rule - overriding partition SL (0) with QoS
Level SL (2)
Nov 25 13:13:05 664670 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR

AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with QoS Level SL (1)

Thanks for your help.

Vincent

Quoted 11 lines

Quoted 11 lines

OK, so there are three possible reasons that I can think of:
1. Something is wrong in the configuration.
2. The application does not saturate the link, thus QoS and the whole VL arbitration thing doesn't kick in.
3. There's some bug, somewhere.

Let's start with reason no. 1.
Please shut off each of the SLs one by one, and make sure that the application gets zero BW on these SLs. You can do it by mapping SL to VL15:

Quoted 5 lines

Quoted 1 line

qos_sl2vl 0,15,2,3,4,5,6,7,8,9,10,11,12,13,14,15

If I shut down this SL by moving it to VL15, the interfaces stop pinging.
This is probably because some IPoIB multicast traffic gets cut off for pkey 0x7fff .. ?

Could be, or because ALL interfaces are mapped to SL1, which is what the results below suggest.

Quoted 13 lines

So no results for this one.

Quoted 3 lines

and then
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

With this setup, and the following QoS settings:

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:1,1:64,2:128,3:192,4:0,5:0
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

Quoted 1 line

I get roughly the same values for SL 1 to SL3:

That doesn't look right.
You have shut off SL2, so you can't see same BW for this SL. Looks like there is a problem in configuration (or bug in SM).

Have you validated somehow that the interfaces have been mapped to the right SLs?

Quoted 21 lines

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.0 sec  6.15 GBytes  5.28 Gbits/sec
[SUM]  0.0-10.0 sec  6.00 GBytes  5.16 Gbits/sec
[SUM]  0.0-10.1 sec  5.38 GBytes  4.59 Gbits/sec

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-backbone -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.0 sec  6.09 GBytes  5.23 Gbits/sec
[SUM]  0.0-10.0 sec  6.41 GBytes  5.51 Gbits/sec
[SUM]  0.0-10.0 sec  4.72 GBytes  4.05 Gbits/sec

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-admin -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.1 sec  6.96 GBytes  5.92 Gbits/sec
[SUM]  0.0-10.1 sec  5.89 GBytes  5.00 Gbits/sec
[SUM]  0.0-10.0 sec  5.35 GBytes  4.58 Gbits/sec

Quoted 2 lines

and then
qos_sl2vl 0,1,2,15,4,5,6,7,8,9,10,11,12,13,14,15

Quoted 8 lines

Same results as the previous 0,1,15,3,... SL2vl mapping.

Quoted 2 lines

If this part works well, then we will continue to reason no. 2.

In the above tests, I used -P8 to force 8 threads on the client side for each test.
I have one quad core CPU(Intel E55400).
This makes 24 iperf threads on 4 cores, which __should__ be fine (well I suppose ...)

Best would be having one qperf per CPU core, which is 4 qperf's in your case.

What is your subnet setup?

From ca {0x2c9000100d00056c} portnum 1 lid 49-49 "pichu16 HCA-1"

-- Yevgeny

Quoted 18 lines

And regarding reason #3. I still get the error I got yesterday, which you told me was not important because the SL's set in partitions.conf would override what was read from qos-policy.conf in the first place.

Nov 25 13:13:05 664690 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
AC15: pkey 0x0002 in match rule - overriding partition SL (0) with QoS
Level SL (3)
Nov 25 13:13:05 664681 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
AC15: pkey 0x0001 in match rule - overriding partition SL (0) with QoS
Level SL (2)
Nov 25 13:13:05 664670 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR

AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with QoS Level SL (1)

Thanks for your help.

Vincent

Hello Yevgeny,

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

OK, so there are three possible reasons that I can think of:
1. Something is wrong in the configuration.
2. The application does not saturate the link, thus QoS and the whole VL arbitration thing doesn't kick in.
3. There's some bug, somewhere.

Let's start with reason no. 1.
Please shut off each of the SLs one by one, and make sure that the application gets zero BW on these SLs. You can do it by mapping SL to VL15:

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

qos_sl2vl 0,15,2,3,4,5,6,7,8,9,10,11,12,13,14,15

If I shut down this SL by moving it to VL15, the interfaces stop pinging.
This is probably because some IPoIB multicast traffic gets cut off for pkey 0x7fff .. ?

Could be, or because ALL interfaces are mapped to SL1, which is what the results below suggest.

Yes, you are right (see below).

Quoted 12 lines

Quoted 12 lines

So no results for this one.

Quoted 3 lines

and then
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

With this setup, and the following QoS settings:

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:1,1:64,2:128,3:192,4:0,5:0
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

Quoted 5 lines

Quoted 1 line

I get roughly the same values for SL 1 to SL3:

That doesn't look right.
You have shut off SL2, so you can't see same BW for this SL. Looks like there is a problem in configuration (or bug in SM).

Yes, that's correct: There could be a configuration issue or a bug in SM:

Current setup and results:

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:1,1:64,2:128,3:192,4:0,5:0
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.1 sec  9.78 GBytes  8.28 Gbits/sec
[SUM]  0.0-10.0 sec  5.69 GBytes  4.89 Gbits/sec
[SUM]  0.0-10.0 sec  4.30 GBytes  3.69 Gbits/sec
[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-backbone
-t 10 -P 8 2>&1; done | grep SUM
[SUM]  0.0-10.2 sec  6.44 GBytes  5.45 Gbits/sec
[SUM]  0.0-10.1 sec  6.64 GBytes  5.66 Gbits/sec
[SUM]  0.0-10.0 sec  6.03 GBytes  5.15 Gbits/sec
[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-admin -t
10 -P 8 2>&1; done | grep SUM
[SUM]  0.0-10.0 sec  5.80 GBytes  4.98 Gbits/sec
[SUM]  0.0-10.0 sec  7.04 GBytes  6.02 Gbits/sec
[SUM]  0.0-10.0 sec  6.60 GBytes  5.67 Gbits/sec

The -backbone bandwidth should be 0 here.

Quoted 2 lines

Have you validated somehow that the interfaces have been mapped to the right SLs?

Two things:
1/ Either the interface have not been mapped properly to the right SL's, but given the config files below, I doubt it:

[root@pichu22 ~]# tail -n 5 /etc/sysconfig/network-scripts/ifcfg-ib0* ==> /etc/sysconfig/network-scripts/ifcfg-ib0 <== BOOTPROTO=static
IPADDR=10.12.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8001 <== BOOTPROTO=static
IPADDR=10.13.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8002 <== BOOTPROTO=static
IPADDR=10.14.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000

partitions.conf:
-----------------

default=0x7fff,ipoib : ALL=full;
ip_backbone=0x0001,ipoib : ALL=full;
ip_admin=0x0002,ipoib : ALL=full;

qos-policy.conf:
----------------
qos-ulps

        default            : 0 # default SL
    ipoib, pkey 0x7FFF    : 1 # IP with default pkey 0x7FFF
    ipoib, pkey 0x1        : 2 # backbone IP with pkey 0x1
    ipoib, pkey 0x2        : 3 # admin IP with pkey 0x2

end-qos-ulps

ib0.8001 maps to pkey 1 (with MSB set to 1 due to full membership => 0x8001 = (1<<16 | 1)
ib0.8002 maps to pkey 2 (with MSB set to 1 due to full membership => 0x8002 = (1<<16 | 2)

2/ Somehow, the qos policy parsing does not map pkeys as we would expect, which is what the opensm messages would suggest:

Nov 25 13:13:05 664690 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
AC15: pkey 0x0002 in match rule - overriding partition SL (0) with QoS
Level SL (3)
Nov 25 13:13:05 664681 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
AC15: pkey 0x0001 in match rule - overriding partition SL (0) with QoS
Level SL (2)
Nov 25 13:13:05 664670 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR

AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with QoS Level SL (1)

If the messages are correct and do reflect what opensm is actually doing, this would explain why shutting down SL1 (by moving it to VL15) prevented all interfaces from running.

Quoted 28 lines

Quoted 20 lines

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.0 sec  6.15 GBytes  5.28 Gbits/sec
[SUM]  0.0-10.0 sec  6.00 GBytes  5.16 Gbits/sec
[SUM]  0.0-10.1 sec  5.38 GBytes  4.59 Gbits/sec

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-backbone -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.0 sec  6.09 GBytes  5.23 Gbits/sec
[SUM]  0.0-10.0 sec  6.41 GBytes  5.51 Gbits/sec
[SUM]  0.0-10.0 sec  4.72 GBytes  4.05 Gbits/sec

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-admin -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.1 sec  6.96 GBytes  5.92 Gbits/sec
[SUM]  0.0-10.1 sec  5.89 GBytes  5.00 Gbits/sec
[SUM]  0.0-10.0 sec  5.35 GBytes  4.58 Gbits/sec

Quoted 2 lines

and then
qos_sl2vl 0,1,2,15,4,5,6,7,8,9,10,11,12,13,14,15

Quoted 8 lines

Same results as the previous 0,1,15,3,... SL2vl mapping.

Quoted 2 lines

If this part works well, then we will continue to reason no. 2.

In the above tests, I used -P8 to force 8 threads on the client side for each test.
I have one quad core CPU(Intel E55400).
This makes 24 iperf threads on 4 cores, which __should__ be fine (well I suppose ...)

Quoted 4 lines

Best would be having one qperf per CPU core, which is 4 qperf's in your case.

What is your subnet setup?

Nothing fancy for this test: I just bounce the taffic through a switch;

[root@pichu16 ~]# ibtracert 49 53

Quoted 1 line

From ca {0x2c9000100d00056c} portnum 1 lid 49-49 "pichu16 HCA-1"

[1] -> switch port {0x0002c9000100d0d4}[22] lid 58-58 "bullX chassis 36 port QDR switch"
[28] -> ca port {0x2c9000100d000679}[1] lid 53-53 "pichu22 HCA-1" To ca {0x2c9000100d000678} portnum 1 lid 53-53 "pichu22 HCA-1"

Vincent

Quoted 23 lines

-- Yevgeny

Quoted 18 lines

And regarding reason #3. I still get the error I got yesterday, which you told me was not important because the SL's set in partitions.conf would override what was read from qos-policy.conf in the first place.

Nov 25 13:13:05 664690 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
AC15: pkey 0x0002 in match rule - overriding partition SL (0) with QoS
Level SL (3)
Nov 25 13:13:05 664681 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
AC15: pkey 0x0001 in match rule - overriding partition SL (0) with QoS
Level SL (2)
Nov 25 13:13:05 664670 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR

AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with QoS Level SL (1)

Thanks for your help.

Vincent

Quoted 129 lines · Vincent Ficet

Hello Yevgeny,

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

OK, so there are three possible reasons that I can think of:
1. Something is wrong in the configuration.
2. The application does not saturate the link, thus QoS and the whole VL arbitration thing doesn't kick in.
3. There's some bug, somewhere.

Let's start with reason no. 1.
Please shut off each of the SLs one by one, and make sure that the application gets zero BW on these SLs. You can do it by mapping SL to VL15:

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

qos_sl2vl 0,15,2,3,4,5,6,7,8,9,10,11,12,13,14,15

If I shut down this SL by moving it to VL15, the interfaces stop pinging.
This is probably because some IPoIB multicast traffic gets cut off for pkey 0x7fff .. ?

Could be, or because ALL interfaces are mapped to SL1, which is what the results below suggest.

Yes, you are right (see below).

Quoted 12 lines

Quoted 12 lines

So no results for this one.

Quoted 3 lines

and then
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

With this setup, and the following QoS settings:

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:1,1:64,2:128,3:192,4:0,5:0
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

Quoted 5 lines

Quoted 1 line

I get roughly the same values for SL 1 to SL3:

That doesn't look right.
You have shut off SL2, so you can't see same BW for this SL. Looks like there is a problem in configuration (or bug in SM).

Yes, that's correct: There could be a configuration issue or a bug in SM:

Current setup and results:

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:1,1:64,2:128,3:192,4:0,5:0
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.1 sec  9.78 GBytes  8.28 Gbits/sec
[SUM]  0.0-10.0 sec  5.69 GBytes  4.89 Gbits/sec
[SUM]  0.0-10.0 sec  4.30 GBytes  3.69 Gbits/sec
[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-backbone
-t 10 -P 8 2>&1; done | grep SUM
[SUM]  0.0-10.2 sec  6.44 GBytes  5.45 Gbits/sec
[SUM]  0.0-10.1 sec  6.64 GBytes  5.66 Gbits/sec
[SUM]  0.0-10.0 sec  6.03 GBytes  5.15 Gbits/sec
[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-admin -t
10 -P 8 2>&1; done | grep SUM
[SUM]  0.0-10.0 sec  5.80 GBytes  4.98 Gbits/sec
[SUM]  0.0-10.0 sec  7.04 GBytes  6.02 Gbits/sec
[SUM]  0.0-10.0 sec  6.60 GBytes  5.67 Gbits/sec

The -backbone bandwidth should be 0 here.

Quoted 2 lines

Have you validated somehow that the interfaces have been mapped to the right SLs?

Two things:
1/ Either the interface have not been mapped properly to the right SL's, but given the config files below, I doubt it:

[root@pichu22 ~]# tail -n 5 /etc/sysconfig/network-scripts/ifcfg-ib0* ==> /etc/sysconfig/network-scripts/ifcfg-ib0 <== BOOTPROTO=static
IPADDR=10.12.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8001 <== BOOTPROTO=static
IPADDR=10.13.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8002 <== BOOTPROTO=static
IPADDR=10.14.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000

partitions.conf:
-----------------

default=0x7fff,ipoib : ALL=full;
ip_backbone=0x0001,ipoib : ALL=full;
ip_admin=0x0002,ipoib : ALL=full;

qos-policy.conf:
----------------
qos-ulps

        default            : 0 # default SL
    ipoib, pkey 0x7FFF    : 1 # IP with default pkey 0x7FFF
    ipoib, pkey 0x1        : 2 # backbone IP with pkey 0x1
    ipoib, pkey 0x2        : 3 # admin IP with pkey 0x2

end-qos-ulps

ib0.8001 maps to pkey 1 (with MSB set to 1 due to full membership => 0x8001 = (1<<16 | 1)
ib0.8002 maps to pkey 2 (with MSB set to 1 due to full membership => 0x8002 = (1<<16 | 2)

2/ Somehow, the qos policy parsing does not map pkeys as we would expect, which is what the opensm messages would suggest:

Nov 25 13:13:05 664690 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
AC15: pkey 0x0002 in match rule - overriding partition SL (0) with QoS
Level SL (3)
Nov 25 13:13:05 664681 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
AC15: pkey 0x0001 in match rule - overriding partition SL (0) with QoS
Level SL (2)
Nov 25 13:13:05 664670 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR

AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with QoS Level SL (1)

Quoted 3 lines

If the messages are correct and do reflect what opensm is actually doing, this would explain why shutting down SL1 (by moving it to VL15) prevented all interfaces from running.

What SM are you using?

Does it have the following bug fix:

http://www.openfabrics.org/git/?p=~sashak/management.git;a=commit;h=ef4c8ac3fdd50bb0b7af06887abdb5b73b7ed8c3

-- Yevgeny

Quoted 75 lines

Quoted 32 lines

Quoted 25 lines

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.0 sec  6.15 GBytes  5.28 Gbits/sec
[SUM]  0.0-10.0 sec  6.00 GBytes  5.16 Gbits/sec
[SUM]  0.0-10.1 sec  5.38 GBytes  4.59 Gbits/sec

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-backbone
-t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.0 sec  6.09 GBytes  5.23 Gbits/sec
[SUM]  0.0-10.0 sec  6.41 GBytes  5.51 Gbits/sec
[SUM]  0.0-10.0 sec  4.72 GBytes  4.05 Gbits/sec

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-admin -t
10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.1 sec  6.96 GBytes  5.92 Gbits/sec
[SUM]  0.0-10.1 sec  5.89 GBytes  5.00 Gbits/sec
[SUM]  0.0-10.0 sec  5.35 GBytes  4.58 Gbits/sec

Quoted 2 lines

and then
qos_sl2vl 0,1,2,15,4,5,6,7,8,9,10,11,12,13,14,15

Same results as the previous 0,1,15,3,... SL2vl mapping.

Quoted 2 lines

If this part works well, then we will continue to reason no. 2.

Quoted 7 lines

In the above tests, I used -P8 to force 8 threads on the client side for
each test.
I have one quad core CPU(Intel E55400).
This makes 24 iperf threads on 4 cores, which __should__ be fine (well I
suppose ...)

Quoted 4 lines

Best would be having one qperf per CPU core, which is 4 qperf's in your case.

What is your subnet setup?

Nothing fancy for this test: I just bounce the taffic through a switch;

[root@pichu16 ~]# ibtracert 49 53

Quoted 1 line

From ca {0x2c9000100d00056c} portnum 1 lid 49-49 "pichu16 HCA-1"

[1] -> switch port {0x0002c9000100d0d4}[22] lid 58-58 "bullX chassis 36 port QDR switch"
[28] -> ca port {0x2c9000100d000679}[1] lid 53-53 "pichu22 HCA-1" To ca {0x2c9000100d000678} portnum 1 lid 53-53 "pichu22 HCA-1"

Vincent

Quoted 26 lines

-- Yevgeny

Quoted 21 lines

And regarding reason #3. I still get the error I got yesterday, which you told me was not important because the SL's set in partitions.conf would override what was read from qos-policy.conf in the first place.

Nov 25 13:13:05 664690 [373E910] 0x01 ->
__qos_policy_validate_pkey: ERR
AC15: pkey 0x0002 in match rule - overriding partition SL (0) with QoS
Level SL (3)
Nov 25 13:13:05 664681 [373E910] 0x01 ->
__qos_policy_validate_pkey: ERR
AC15: pkey 0x0001 in match rule - overriding partition SL (0) with QoS
Level SL (2)
Nov 25 13:13:05 664670 [373E910] 0x01 ->

__qos_policy_validate_pkey: ERR
AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with QoS Level SL (1)

Thanks for your help.

Vincent

Quoted 130 lines · Yevgeny Kliteynik

Quoted 129 lines · Vincent Ficet

Hello Yevgeny,

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

OK, so there are three possible reasons that I can think of:
1. Something is wrong in the configuration.
2. The application does not saturate the link, thus QoS and the whole VL arbitration thing doesn't kick in.
3. There's some bug, somewhere.

Let's start with reason no. 1.
Please shut off each of the SLs one by one, and make sure that the application gets zero BW on these SLs. You can do it by mapping SL to VL15:

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

qos_sl2vl 0,15,2,3,4,5,6,7,8,9,10,11,12,13,14,15

If I shut down this SL by moving it to VL15, the interfaces stop pinging.
This is probably because some IPoIB multicast traffic gets cut off for pkey 0x7fff .. ?

Could be, or because ALL interfaces are mapped to SL1, which is what the results below suggest.

Yes, you are right (see below).

Quoted 12 lines

Quoted 12 lines

So no results for this one.

Quoted 3 lines

and then
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

With this setup, and the following QoS settings:

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:1,1:64,2:128,3:192,4:0,5:0
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

Quoted 5 lines

Quoted 1 line

I get roughly the same values for SL 1 to SL3:

That doesn't look right.
You have shut off SL2, so you can't see same BW for this SL. Looks like there is a problem in configuration (or bug in SM).

Yes, that's correct: There could be a configuration issue or a bug in SM:

Current setup and results:

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:1,1:64,2:128,3:192,4:0,5:0
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.1 sec  9.78 GBytes  8.28 Gbits/sec
[SUM]  0.0-10.0 sec  5.69 GBytes  4.89 Gbits/sec
[SUM]  0.0-10.0 sec  4.30 GBytes  3.69 Gbits/sec
[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-backbone
-t 10 -P 8 2>&1; done | grep SUM
[SUM]  0.0-10.2 sec  6.44 GBytes  5.45 Gbits/sec
[SUM]  0.0-10.1 sec  6.64 GBytes  5.66 Gbits/sec
[SUM]  0.0-10.0 sec  6.03 GBytes  5.15 Gbits/sec
[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-admin -t
10 -P 8 2>&1; done | grep SUM
[SUM]  0.0-10.0 sec  5.80 GBytes  4.98 Gbits/sec
[SUM]  0.0-10.0 sec  7.04 GBytes  6.02 Gbits/sec
[SUM]  0.0-10.0 sec  6.60 GBytes  5.67 Gbits/sec

The -backbone bandwidth should be 0 here.

Quoted 2 lines

Have you validated somehow that the interfaces have been mapped to the right SLs?

Two things:
1/ Either the interface have not been mapped properly to the right SL's, but given the config files below, I doubt it:

[root@pichu22 ~]# tail -n 5 /etc/sysconfig/network-scripts/ifcfg-ib0* ==> /etc/sysconfig/network-scripts/ifcfg-ib0 <== BOOTPROTO=static
IPADDR=10.12.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8001 <== BOOTPROTO=static
IPADDR=10.13.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8002 <== BOOTPROTO=static
IPADDR=10.14.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000

partitions.conf:
-----------------

default=0x7fff,ipoib : ALL=full;
ip_backbone=0x0001,ipoib : ALL=full;
ip_admin=0x0002,ipoib : ALL=full;

qos-policy.conf:
----------------
qos-ulps

        default            : 0 # default SL
    ipoib, pkey 0x7FFF    : 1 # IP with default pkey 0x7FFF
    ipoib, pkey 0x1        : 2 # backbone IP with pkey 0x1
    ipoib, pkey 0x2        : 3 # admin IP with pkey 0x2

end-qos-ulps

ib0.8001 maps to pkey 1 (with MSB set to 1 due to full membership => 0x8001 = (1<<16 | 1)
ib0.8002 maps to pkey 2 (with MSB set to 1 due to full membership => 0x8002 = (1<<16 | 2)

2/ Somehow, the qos policy parsing does not map pkeys as we would expect, which is what the opensm messages would suggest:

Nov 25 13:13:05 664690 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
AC15: pkey 0x0002 in match rule - overriding partition SL (0) with QoS
Level SL (3)
Nov 25 13:13:05 664681 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
AC15: pkey 0x0001 in match rule - overriding partition SL (0) with QoS
Level SL (2)
Nov 25 13:13:05 664670 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR

AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with QoS Level SL (1)

Quoted 4 lines

Quoted 3 lines

If the messages are correct and do reflect what opensm is actually doing, this would explain why shutting down SL1 (by moving it to VL15) prevented all interfaces from running.

What SM are you using?

OpenSM 3.3.2

Quoted 2 lines

Does it have the following bug fix:

Quoted 2 lines

http://www.openfabrics.org/git/?p=~sashak/management.git;a=commit;h=ef4c8ac3fdd50bb0b7af06887abdb5b73b7ed8c3

Yes it does.

The most recent git commit (sorted by date) is for this rpm is:
* Sun Aug 23 2009 Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/[email protected]> commit 3f4954c73add5e7b598883242782607f87c482b4

Apart from the following commit (with a bogus date):
* Tue Jul 24 2035 Keshetti Mahesh <[email protected]> commit a0c23ed2194e96816744a075d405ff34c8373fa3

Thanks,

Vincent

Quoted 77 lines

-- Yevgeny

Quoted 73 lines

Quoted 32 lines

Quoted 25 lines

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.0 sec  6.15 GBytes  5.28 Gbits/sec
[SUM]  0.0-10.0 sec  6.00 GBytes  5.16 Gbits/sec
[SUM]  0.0-10.1 sec  5.38 GBytes  4.59 Gbits/sec

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-backbone
-t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.0 sec  6.09 GBytes  5.23 Gbits/sec
[SUM]  0.0-10.0 sec  6.41 GBytes  5.51 Gbits/sec
[SUM]  0.0-10.0 sec  4.72 GBytes  4.05 Gbits/sec

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-admin -t
10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.1 sec  6.96 GBytes  5.92 Gbits/sec
[SUM]  0.0-10.1 sec  5.89 GBytes  5.00 Gbits/sec
[SUM]  0.0-10.0 sec  5.35 GBytes  4.58 Gbits/sec

Quoted 2 lines

and then
qos_sl2vl 0,1,2,15,4,5,6,7,8,9,10,11,12,13,14,15

Same results as the previous 0,1,15,3,... SL2vl mapping.

Quoted 2 lines

If this part works well, then we will continue to reason no. 2.

Quoted 7 lines

In the above tests, I used -P8 to force 8 threads on the client side for
each test.
I have one quad core CPU(Intel E55400).
This makes 24 iperf threads on 4 cores, which __should__ be fine (well I
suppose ...)

Quoted 4 lines

Best would be having one qperf per CPU core, which is 4 qperf's in your case.

What is your subnet setup?

Nothing fancy for this test: I just bounce the taffic through a switch;

[root@pichu16 ~]# ibtracert 49 53

Quoted 1 line

From ca {0x2c9000100d00056c} portnum 1 lid 49-49 "pichu16 HCA-1"

[1] -> switch port {0x0002c9000100d0d4}[22] lid 58-58 "bullX chassis 36 port QDR switch"
[28] -> ca port {0x2c9000100d000679}[1] lid 53-53 "pichu22 HCA-1" To ca {0x2c9000100d000678} portnum 1 lid 53-53 "pichu22 HCA-1"

Vincent

Quoted 25 lines

-- Yevgeny

Quoted 21 lines

And regarding reason #3. I still get the error I got yesterday, which you told me was not important because the SL's set in partitions.conf would override what was read from qos-policy.conf in the first place.

Nov 25 13:13:05 664690 [373E910] 0x01 ->
__qos_policy_validate_pkey: ERR
AC15: pkey 0x0002 in match rule - overriding partition SL (0) with QoS
Level SL (3)
Nov 25 13:13:05 664681 [373E910] 0x01 ->
__qos_policy_validate_pkey: ERR
AC15: pkey 0x0001 in match rule - overriding partition SL (0) with QoS
Level SL (2)
Nov 25 13:13:05 664670 [373E910] 0x01 ->

__qos_policy_validate_pkey: ERR
AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with QoS Level SL (1)

Thanks for your help.

Vincent

Quoted 138 lines · Vincent Ficet

Quoted 130 lines · Yevgeny Kliteynik

Quoted 129 lines · Vincent Ficet

Hello Yevgeny,

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

OK, so there are three possible reasons that I can think of:
1. Something is wrong in the configuration.
2. The application does not saturate the link, thus QoS and the whole VL arbitration thing doesn't kick in.
3. There's some bug, somewhere.

Let's start with reason no. 1.
Please shut off each of the SLs one by one, and make sure that the application gets zero BW on these SLs. You can do it by mapping SL to VL15:

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

qos_sl2vl 0,15,2,3,4,5,6,7,8,9,10,11,12,13,14,15

If I shut down this SL by moving it to VL15, the interfaces stop pinging.
This is probably because some IPoIB multicast traffic gets cut off for pkey 0x7fff .. ?

Could be, or because ALL interfaces are mapped to SL1, which is what the results below suggest.

Yes, you are right (see below).

Quoted 12 lines

Quoted 12 lines

So no results for this one.

Quoted 3 lines

and then
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

With this setup, and the following QoS settings:

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:1,1:64,2:128,3:192,4:0,5:0
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

Quoted 5 lines

Quoted 1 line

I get roughly the same values for SL 1 to SL3:

That doesn't look right.
You have shut off SL2, so you can't see same BW for this SL. Looks like there is a problem in configuration (or bug in SM).

Yes, that's correct: There could be a configuration issue or a bug in SM:

Current setup and results:

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:1,1:64,2:128,3:192,4:0,5:0
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.1 sec  9.78 GBytes  8.28 Gbits/sec
[SUM]  0.0-10.0 sec  5.69 GBytes  4.89 Gbits/sec
[SUM]  0.0-10.0 sec  4.30 GBytes  3.69 Gbits/sec
[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-backbone
-t 10 -P 8 2>&1; done | grep SUM
[SUM]  0.0-10.2 sec  6.44 GBytes  5.45 Gbits/sec
[SUM]  0.0-10.1 sec  6.64 GBytes  5.66 Gbits/sec
[SUM]  0.0-10.0 sec  6.03 GBytes  5.15 Gbits/sec
[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-admin -t
10 -P 8 2>&1; done | grep SUM
[SUM]  0.0-10.0 sec  5.80 GBytes  4.98 Gbits/sec
[SUM]  0.0-10.0 sec  7.04 GBytes  6.02 Gbits/sec
[SUM]  0.0-10.0 sec  6.60 GBytes  5.67 Gbits/sec

The -backbone bandwidth should be 0 here.

Quoted 2 lines

Have you validated somehow that the interfaces have been mapped to the right SLs?

Two things:
1/ Either the interface have not been mapped properly to the right SL's, but given the config files below, I doubt it:

[root@pichu22 ~]# tail -n 5 /etc/sysconfig/network-scripts/ifcfg-ib0* ==> /etc/sysconfig/network-scripts/ifcfg-ib0 <== BOOTPROTO=static
IPADDR=10.12.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8001 <== BOOTPROTO=static
IPADDR=10.13.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8002 <== BOOTPROTO=static
IPADDR=10.14.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000

partitions.conf:
-----------------

default=0x7fff,ipoib : ALL=full;
ip_backbone=0x0001,ipoib : ALL=full;
ip_admin=0x0002,ipoib : ALL=full;

qos-policy.conf:
----------------
qos-ulps

        default            : 0 # default SL
    ipoib, pkey 0x7FFF    : 1 # IP with default pkey 0x7FFF
    ipoib, pkey 0x1        : 2 # backbone IP with pkey 0x1
    ipoib, pkey 0x2        : 3 # admin IP with pkey 0x2

end-qos-ulps

ib0.8001 maps to pkey 1 (with MSB set to 1 due to full membership => 0x8001 = (1<<16 | 1)
ib0.8002 maps to pkey 2 (with MSB set to 1 due to full membership => 0x8002 = (1<<16 | 2)

2/ Somehow, the qos policy parsing does not map pkeys as we would expect, which is what the opensm messages would suggest:

Nov 25 13:13:05 664690 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
AC15: pkey 0x0002 in match rule - overriding partition SL (0) with QoS
Level SL (3)
Nov 25 13:13:05 664681 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
AC15: pkey 0x0001 in match rule - overriding partition SL (0) with QoS
Level SL (2)
Nov 25 13:13:05 664670 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR

AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with QoS Level SL (1)

Quoted 4 lines

Quoted 3 lines

If the messages are correct and do reflect what opensm is actually doing, this would explain why shutting down SL1 (by moving it to VL15) prevented all interfaces from running.

What SM are you using?

OpenSM 3.3.2

Quoted 2 lines

Does it have the following bug fix:

Quoted 8 lines

Quoted 3 lines

http://www.openfabrics.org/git/?p=~sashak/management.git;a=commit;h=ef4c8ac3fdd50bb0b7af06887abdb5b73b7ed8c3

Yes it does.

The most recent git commit (sorted by date) is for this rpm is:
* Sun Aug 23 2009 Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/[email protected]> commit 3f4954c73add5e7b598883242782607f87c482b4

OK, in that case I ran out of ideas. Need to debug. We can do it here, but best would be if you open a bug at bugzilla.
Please run opensm as follows:

opensm -Q -Y <qos_policy_file> -P <partition_config_file> -e -V -s 0 -d1 &

Wait a minute or so, try your test, and attach OSM log to the issue.

-- Yevgeny

Quoted 78 lines

Apart from the following commit (with a bogus date):
* Tue Jul 24 2035 Keshetti Mahesh <[email protected]> commit a0c23ed2194e96816744a075d405ff34c8373fa3

Thanks,

Vincent

Quoted 71 lines

-- Yevgeny

Quoted 69 lines

Quoted 32 lines

Quoted 25 lines

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.0 sec  6.15 GBytes  5.28 Gbits/sec
[SUM]  0.0-10.0 sec  6.00 GBytes  5.16 Gbits/sec
[SUM]  0.0-10.1 sec  5.38 GBytes  4.59 Gbits/sec

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-backbone
-t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.0 sec  6.09 GBytes  5.23 Gbits/sec
[SUM]  0.0-10.0 sec  6.41 GBytes  5.51 Gbits/sec
[SUM]  0.0-10.0 sec  4.72 GBytes  4.05 Gbits/sec

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-admin -t
10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.1 sec  6.96 GBytes  5.92 Gbits/sec
[SUM]  0.0-10.1 sec  5.89 GBytes  5.00 Gbits/sec
[SUM]  0.0-10.0 sec  5.35 GBytes  4.58 Gbits/sec

Quoted 2 lines

and then
qos_sl2vl 0,1,2,15,4,5,6,7,8,9,10,11,12,13,14,15

Same results as the previous 0,1,15,3,... SL2vl mapping.

Quoted 2 lines

If this part works well, then we will continue to reason no. 2.

Quoted 7 lines

In the above tests, I used -P8 to force 8 threads on the client side for
each test.
I have one quad core CPU(Intel E55400).
This makes 24 iperf threads on 4 cores, which __should__ be fine (well I
suppose ...)

Quoted 4 lines

Best would be having one qperf per CPU core, which is 4 qperf's in your case.

What is your subnet setup?

Nothing fancy for this test: I just bounce the taffic through a switch;

[root@pichu16 ~]# ibtracert 49 53

Quoted 1 line

From ca {0x2c9000100d00056c} portnum 1 lid 49-49 "pichu16 HCA-1"

[1] -> switch port {0x0002c9000100d0d4}[22] lid 58-58 "bullX chassis 36 port QDR switch"
[28] -> ca port {0x2c9000100d000679}[1] lid 53-53 "pichu22 HCA-1" To ca {0x2c9000100d000678} portnum 1 lid 53-53 "pichu22 HCA-1"

Vincent

Quoted 22 lines

-- Yevgeny

Quoted 19 lines

And regarding reason #3. I still get the error I got yesterday, which you told me was not important because the SL's set in partitions.conf would override what was read from qos-policy.conf in the first place.

Nov 25 13:13:05 664690 [373E910] 0x01 ->
__qos_policy_validate_pkey: ERR
AC15: pkey 0x0002 in match rule - overriding partition SL (0) with QoS
Level SL (3)
Nov 25 13:13:05 664681 [373E910] 0x01 ->
__qos_policy_validate_pkey: ERR
AC15: pkey 0x0001 in match rule - overriding partition SL (0) with QoS
Level SL (2)
Nov 25 13:13:05 664670 [373E910] 0x01 ->

__qos_policy_validate_pkey: ERR
AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with QoS Level SL (1)

Thanks for your help.

Quoted 8 lines

Quoted 6 lines

Quoted 4 lines

Quoted 3 lines

Quoted 2 lines

Vincent

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Sebastien,

I noticed that you found the problem in IPoIB child interfaces configuration. Glad that this worked out well.

My question is about the note that you left in the issue:

" It looks like in 'datagram' mode, the SL weights
do not seem to be applied, or maybe this is an artifact of IPoIB in 'datagram mode' "

Have you checked that in this mode you do get the right SL for each child interface by shutting off the relevant SL (mapping it to VL15)?

If yes, then what you're saying is that you see that interfaces use the right SL and VL, but you don't see any arbitration between VLs?

-- Yevgeny

Quoted 251 lines · Yevgeny Kliteynik

Quoted 145 lines · Vincent Ficet

Quoted 137 lines · Yevgeny Kliteynik

Quoted 136 lines · Vincent Ficet

Hello Yevgeny,

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

OK, so there are three possible reasons that I can think of:
1. Something is wrong in the configuration.
2. The application does not saturate the link, thus QoS and the whole VL arbitration thing doesn't kick in.
3. There's some bug, somewhere.

Let's start with reason no. 1.
Please shut off each of the SLs one by one, and make sure that the application gets zero BW on these SLs. You can do it by mapping SL to VL15:

Quoted 8 lines

Quoted 6 lines

Quoted 1 line

qos_sl2vl 0,15,2,3,4,5,6,7,8,9,10,11,12,13,14,15

If I shut down this SL by moving it to VL15, the interfaces stop pinging.
This is probably because some IPoIB multicast traffic gets cut off for
pkey 0x7fff .. ?

Could be, or because ALL interfaces are mapped to SL1, which is what the results below suggest.

Yes, you are right (see below).

Quoted 12 lines

Quoted 12 lines

So no results for this one.

Quoted 3 lines

and then
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

With this setup, and the following QoS settings:

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:1,1:64,2:128,3:192,4:0,5:0
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

Quoted 5 lines

Quoted 1 line

I get roughly the same values for SL 1 to SL3:

That doesn't look right.
You have shut off SL2, so you can't see same BW for this SL. Looks like there is a problem in configuration (or bug in SM).

Yes, that's correct: There could be a configuration issue or a bug in SM:

Current setup and results:

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:1,1:64,2:128,3:192,4:0,5:0
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.1 sec  9.78 GBytes  8.28 Gbits/sec
[SUM]  0.0-10.0 sec  5.69 GBytes  4.89 Gbits/sec
[SUM]  0.0-10.0 sec  4.30 GBytes  3.69 Gbits/sec
[root@pichu22 ~]# while test -e keep_going; do iperf -c 
pichu16-backbone
-t 10 -P 8 2>&1; done | grep SUM
[SUM]  0.0-10.2 sec  6.44 GBytes  5.45 Gbits/sec
[SUM]  0.0-10.1 sec  6.64 GBytes  5.66 Gbits/sec
[SUM]  0.0-10.0 sec  6.03 GBytes  5.15 Gbits/sec
[root@pichu22 ~]# while test -e keep_going; do iperf -c 
pichu16-admin -t
10 -P 8 2>&1; done | grep SUM
[SUM]  0.0-10.0 sec  5.80 GBytes  4.98 Gbits/sec
[SUM]  0.0-10.0 sec  7.04 GBytes  6.02 Gbits/sec
[SUM]  0.0-10.0 sec  6.60 GBytes  5.67 Gbits/sec

The -backbone bandwidth should be 0 here.

Quoted 2 lines

Have you validated somehow that the interfaces have been mapped to the right SLs?

Two things:
1/ Either the interface have not been mapped properly to the right SL's,
but given the config files below, I doubt it:

[root@pichu22 ~]# tail -n 5 /etc/sysconfig/network-scripts/ifcfg-ib0* ==> /etc/sysconfig/network-scripts/ifcfg-ib0 <== BOOTPROTO=static
IPADDR=10.12.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8001 <== BOOTPROTO=static
IPADDR=10.13.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8002 <== BOOTPROTO=static
IPADDR=10.14.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000

partitions.conf:
-----------------

default=0x7fff,ipoib : ALL=full;
ip_backbone=0x0001,ipoib : ALL=full;
ip_admin=0x0002,ipoib : ALL=full;

qos-policy.conf:
----------------
qos-ulps

        default            : 0 # default SL
    ipoib, pkey 0x7FFF    : 1 # IP with default pkey 0x7FFF
    ipoib, pkey 0x1        : 2 # backbone IP with pkey 0x1
    ipoib, pkey 0x2        : 3 # admin IP with pkey 0x2

end-qos-ulps

ib0.8001 maps to pkey 1 (with MSB set to 1 due to full membership => 0x8001 = (1<<16 | 1)
ib0.8002 maps to pkey 2 (with MSB set to 1 due to full membership => 0x8002 = (1<<16 | 2)

2/ Somehow, the qos policy parsing does not map pkeys as we would expect, which is what the opensm messages would suggest:

Nov 25 13:13:05 664690 [373E910] 0x01 -> __qos_policy_validate_pkey: 
ERR
AC15: pkey 0x0002 in match rule - overriding partition SL (0) with QoS
Level SL (3)
Nov 25 13:13:05 664681 [373E910] 0x01 -> __qos_policy_validate_pkey: 
ERR
AC15: pkey 0x0001 in match rule - overriding partition SL (0) with QoS
Level SL (2)
Nov 25 13:13:05 664670 [373E910] 0x01 -> __qos_policy_validate_pkey: 

ERR
AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with QoS Level SL (1)

Quoted 4 lines

Quoted 3 lines

If the messages are correct and do reflect what opensm is actually doing, this would explain why shutting down SL1 (by moving it to VL15) prevented all interfaces from running.

What SM are you using?

OpenSM 3.3.2

Quoted 2 lines

Does it have the following bug fix:

Quoted 8 lines

Quoted 3 lines

http://www.openfabrics.org/git/?p=~sashak/management.git;a=commit;h=ef4c8ac3fdd50bb0b7af06887abdb5b73b7ed8c3

Yes it does.

The most recent git commit (sorted by date) is for this rpm is:
* Sun Aug 23 2009 Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/[email protected]> commit 3f4954c73add5e7b598883242782607f87c482b4

OK, in that case I ran out of ideas. Need to debug. We can do it here, but best would be if you open a bug at bugzilla.
Please run opensm as follows:

opensm -Q -Y <qos_policy_file> -P <partition_config_file> -e -V -s 0 -d1 &

Wait a minute or so, try your test, and attach OSM log to the issue.

-- Yevgeny

Quoted 82 lines

Apart from the following commit (with a bogus date):
* Tue Jul 24 2035 Keshetti Mahesh <[email protected]> commit a0c23ed2194e96816744a075d405ff34c8373fa3

Thanks,

Vincent

Quoted 75 lines

-- Yevgeny

Quoted 73 lines

Quoted 33 lines

Quoted 26 lines

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t
10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.0 sec  6.15 GBytes  5.28 Gbits/sec
[SUM]  0.0-10.0 sec  6.00 GBytes  5.16 Gbits/sec
[SUM]  0.0-10.1 sec  5.38 GBytes  4.59 Gbits/sec

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-backbone
-t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.0 sec  6.09 GBytes  5.23 Gbits/sec
[SUM]  0.0-10.0 sec  6.41 GBytes  5.51 Gbits/sec
[SUM]  0.0-10.0 sec  4.72 GBytes  4.05 Gbits/sec

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-admin -t
10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.1 sec  6.96 GBytes  5.92 Gbits/sec
[SUM]  0.0-10.1 sec  5.89 GBytes  5.00 Gbits/sec
[SUM]  0.0-10.0 sec  5.35 GBytes  4.58 Gbits/sec

Quoted 2 lines

and then
qos_sl2vl 0,1,2,15,4,5,6,7,8,9,10,11,12,13,14,15

Same results as the previous 0,1,15,3,... SL2vl mapping.

Quoted 2 lines

If this part works well, then we will continue to reason no. 2.

Quoted 7 lines

In the above tests, I used -P8 to force 8 threads on the client side for
each test.
I have one quad core CPU(Intel E55400).
This makes 24 iperf threads on 4 cores, which __should__ be fine (well I
suppose ...)

Quoted 4 lines

Best would be having one qperf per CPU core, which is 4 qperf's in your case.

What is your subnet setup?

Nothing fancy for this test: I just bounce the taffic through a switch;

[root@pichu16 ~]# ibtracert 49 53

Quoted 1 line

From ca {0x2c9000100d00056c} portnum 1 lid 49-49 "pichu16 HCA-1"

[1] -> switch port {0x0002c9000100d0d4}[22] lid 58-58 "bullX chassis 36 port QDR switch"
[28] -> ca port {0x2c9000100d000679}[1] lid 53-53 "pichu22 HCA-1" To ca {0x2c9000100d000678} portnum 1 lid 53-53 "pichu22 HCA-1"

Vincent

Quoted 25 lines

-- Yevgeny

Quoted 22 lines

And regarding reason #3. I still get the error I got yesterday, which you told me was not important because the SL's set in partitions.conf would override what was read from qos-policy.conf in the first place.

Nov 25 13:13:05 664690 [373E910] 0x01 ->

__qos_policy_validate_pkey: ERR
AC15: pkey 0x0002 in match rule - overriding partition SL (0) with QoS
Level SL (3)

Nov 25 13:13:05 664681 [373E910] 0x01 ->

__qos_policy_validate_pkey: ERR
AC15: pkey 0x0001 in match rule - overriding partition SL (0) with QoS
Level SL (2)

Nov 25 13:13:05 664670 [373E910] 0x01 ->

__qos_policy_validate_pkey: ERR
AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with QoS
Level SL (1)

Thanks for your help.

Quoted 14 lines

Quoted 8 lines

Quoted 6 lines

Quoted 4 lines

Quoted 3 lines

Quoted 2 lines

Vincent

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

" It looks like in 'datagram' mode, the SL weights
do not seem to be applied, or maybe this is an artifact of IPoIB in 'datagram mode' "

Have you checked that in this mode you do get the right SL for each child interface by shutting off the relevant SL (mapping it to VL15)?

ifdown ib1
echo 1 > /sys/module/ib_ipoib/parameters/debug_level ifup ib1
ping 10.10.0.91
dmesg | grep ib1

ib1: Start path record lookup for fe80:0000:0000:0000:0008:f104:0399:3c92 MTU > 0
ib1: PathRec LID 0x0006 for GID fe80:0000:0000:0000:0008:f104:0399:3c92
ib1: Created ah ffff81021ddda180
ib1: created address handle ffff81021ddda500 for LID 0x0006, SL 0

# ip neigh show dev ib1
10.10.0.91 lladdr 80:00:00:49:fe:80:00:00:00:00:00:00:00:08:f1:04:03:99:3c:92 REACHABLE

Hi Yevgeny,

Quoted 6 lines · On Thu, 03 Dec 2009 10:01:28 +0200 Yevgeny Kliteynik…

Sebastien,

I noticed that you found the problem in IPoIB child interfaces configuration. Glad that this worked out well.

My question is about the note that you left in the issue:

Quoted 7 lines

" It looks like in 'datagram' mode, the SL weights
do not seem to be applied, or maybe this is an artifact of IPoIB in 'datagram mode' "

Have you checked that in this mode you do get the right SL for each child interface by shutting off the relevant SL (mapping it to VL15)?

Yes, SL to VL mapping is OK.

Quoted 3 lines

If yes, then what you're saying is that you see that interfaces use the right SL and VL, but you don't see any arbitration between VLs?

Right, whatever the weights I put in the vlarbs tables have absolutely
no effect when IPoIB is in datagram mode. I don't know if it's an arbitration problem (don't think so) or an IPoIB problem.

Could be that due to the 2044 bytes MTU in datagram mode, iperf
spends much time not doing transfers and fails to provide enough data to the interfaces. Don't know.

Once I switched to connected mode, with a 65520 bytes MTU, things
started to work OK with a much better overall combined bandwidth.

Thanks,

Sébastien.

Quoted 256 lines

-- Yevgeny

Quoted 237 lines · Yevgeny Kliteynik

Quoted 138 lines · Vincent Ficet

Quoted 130 lines · Yevgeny Kliteynik

Quoted 129 lines · Vincent Ficet

Hello Yevgeny,

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

OK, so there are three possible reasons that I can think of:
1. Something is wrong in the configuration.
2. The application does not saturate the link, thus QoS and the whole VL arbitration thing doesn't kick in.
3. There's some bug, somewhere.

Let's start with reason no. 1.
Please shut off each of the SLs one by one, and make sure that the application gets zero BW on these SLs. You can do it by mapping SL to VL15:

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

qos_sl2vl 0,15,2,3,4,5,6,7,8,9,10,11,12,13,14,15

If I shut down this SL by moving it to VL15, the interfaces stop pinging.
This is probably because some IPoIB multicast traffic gets cut off for pkey 0x7fff .. ?

Could be, or because ALL interfaces are mapped to SL1, which is what the results below suggest.

Yes, you are right (see below).

Quoted 12 lines

Quoted 12 lines

So no results for this one.

Quoted 3 lines

and then
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

With this setup, and the following QoS settings:

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:1,1:64,2:128,3:192,4:0,5:0
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

Quoted 5 lines

Quoted 1 line

I get roughly the same values for SL 1 to SL3:

That doesn't look right.
You have shut off SL2, so you can't see same BW for this SL. Looks like there is a problem in configuration (or bug in SM).

Yes, that's correct: There could be a configuration issue or a bug in SM:

Current setup and results:

qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:1,1:64,2:128,3:192,4:0,5:0
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.1 sec  9.78 GBytes  8.28 Gbits/sec
[SUM]  0.0-10.0 sec  5.69 GBytes  4.89 Gbits/sec
[SUM]  0.0-10.0 sec  4.30 GBytes  3.69 Gbits/sec
[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-backbone
-t 10 -P 8 2>&1; done | grep SUM
[SUM]  0.0-10.2 sec  6.44 GBytes  5.45 Gbits/sec
[SUM]  0.0-10.1 sec  6.64 GBytes  5.66 Gbits/sec
[SUM]  0.0-10.0 sec  6.03 GBytes  5.15 Gbits/sec
[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-admin -t
10 -P 8 2>&1; done | grep SUM
[SUM]  0.0-10.0 sec  5.80 GBytes  4.98 Gbits/sec
[SUM]  0.0-10.0 sec  7.04 GBytes  6.02 Gbits/sec
[SUM]  0.0-10.0 sec  6.60 GBytes  5.67 Gbits/sec

The -backbone bandwidth should be 0 here.

Quoted 2 lines

Have you validated somehow that the interfaces have been mapped to the right SLs?

Two things:
1/ Either the interface have not been mapped properly to the right SL's, but given the config files below, I doubt it:

[root@pichu22 ~]# tail -n 5 /etc/sysconfig/network-scripts/ifcfg-ib0* ==> /etc/sysconfig/network-scripts/ifcfg-ib0 <== BOOTPROTO=static
IPADDR=10.12.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8001 <== BOOTPROTO=static
IPADDR=10.13.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000

==> /etc/sysconfig/network-scripts/ifcfg-ib0.8002 <== BOOTPROTO=static
IPADDR=10.14.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000

partitions.conf:
-----------------

default=0x7fff,ipoib : ALL=full;
ip_backbone=0x0001,ipoib : ALL=full;
ip_admin=0x0002,ipoib : ALL=full;

qos-policy.conf:
----------------
qos-ulps

        default            : 0 # default SL
    ipoib, pkey 0x7FFF    : 1 # IP with default pkey 0x7FFF
    ipoib, pkey 0x1        : 2 # backbone IP with pkey 0x1
    ipoib, pkey 0x2        : 3 # admin IP with pkey 0x2

end-qos-ulps

ib0.8001 maps to pkey 1 (with MSB set to 1 due to full membership => 0x8001 = (1<<16 | 1)
ib0.8002 maps to pkey 2 (with MSB set to 1 due to full membership => 0x8002 = (1<<16 | 2)

2/ Somehow, the qos policy parsing does not map pkeys as we would expect, which is what the opensm messages would suggest:

Nov 25 13:13:05 664690 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
AC15: pkey 0x0002 in match rule - overriding partition SL (0) with QoS
Level SL (3)
Nov 25 13:13:05 664681 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
AC15: pkey 0x0001 in match rule - overriding partition SL (0) with QoS
Level SL (2)
Nov 25 13:13:05 664670 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR

AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with QoS Level SL (1)

Quoted 4 lines

Quoted 3 lines

If the messages are correct and do reflect what opensm is actually doing, this would explain why shutting down SL1 (by moving it to VL15) prevented all interfaces from running.

What SM are you using?

OpenSM 3.3.2

Quoted 2 lines

Does it have the following bug fix:

Quoted 8 lines

Quoted 3 lines

http://www.openfabrics.org/git/?p=~sashak/management.git;a=commit;h=ef4c8ac3fdd50bb0b7af06887abdb5b73b7ed8c3

Yes it does.

The most recent git commit (sorted by date) is for this rpm is:
* Sun Aug 23 2009 Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/[email protected]> commit 3f4954c73add5e7b598883242782607f87c482b4

OK, in that case I ran out of ideas. Need to debug. We can do it here, but best would be if you open a bug at bugzilla.
Please run opensm as follows:

opensm -Q -Y <qos_policy_file> -P <partition_config_file> -e -V -s 0 -d1 &

Wait a minute or so, try your test, and attach OSM log to the issue.

-- Yevgeny

Quoted 76 lines

Apart from the following commit (with a bogus date):
* Tue Jul 24 2035 Keshetti Mahesh <[email protected]> commit a0c23ed2194e96816744a075d405ff34c8373fa3

Thanks,

Vincent

Quoted 69 lines

-- Yevgeny

Quoted 67 lines

Quoted 32 lines

Quoted 25 lines

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.0 sec  6.15 GBytes  5.28 Gbits/sec
[SUM]  0.0-10.0 sec  6.00 GBytes  5.16 Gbits/sec
[SUM]  0.0-10.1 sec  5.38 GBytes  4.59 Gbits/sec

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-backbone
-t 10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.0 sec  6.09 GBytes  5.23 Gbits/sec
[SUM]  0.0-10.0 sec  6.41 GBytes  5.51 Gbits/sec
[SUM]  0.0-10.0 sec  4.72 GBytes  4.05 Gbits/sec

[root@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-admin -t
10 -P 8 2>&1; done | grep SUM

[SUM]  0.0-10.1 sec  6.96 GBytes  5.92 Gbits/sec
[SUM]  0.0-10.1 sec  5.89 GBytes  5.00 Gbits/sec
[SUM]  0.0-10.0 sec  5.35 GBytes  4.58 Gbits/sec

Quoted 2 lines

and then
qos_sl2vl 0,1,2,15,4,5,6,7,8,9,10,11,12,13,14,15

Same results as the previous 0,1,15,3,... SL2vl mapping.

Quoted 2 lines

If this part works well, then we will continue to reason no. 2.

Quoted 7 lines

In the above tests, I used -P8 to force 8 threads on the client side for
each test.
I have one quad core CPU(Intel E55400).
This makes 24 iperf threads on 4 cores, which __should__ be fine (well I
suppose ...)

Quoted 4 lines

Best would be having one qperf per CPU core, which is 4 qperf's in your case.

What is your subnet setup?

Nothing fancy for this test: I just bounce the taffic through a switch;

[root@pichu16 ~]# ibtracert 49 53

Quoted 1 line

From ca {0x2c9000100d00056c} portnum 1 lid 49-49 "pichu16 HCA-1"

[1] -> switch port {0x0002c9000100d0d4}[22] lid 58-58 "bullX chassis 36 port QDR switch"
[28] -> ca port {0x2c9000100d000679}[1] lid 53-53 "pichu22 HCA-1" To ca {0x2c9000100d000678} portnum 1 lid 53-53 "pichu22 HCA-1"

Vincent

Quoted 20 lines

-- Yevgeny

Quoted 17 lines

And regarding reason #3. I still get the error I got yesterday, which you told me was not important because the SL's set in partitions.conf would override what was read from qos-policy.conf in the first place.

Nov 25 13:13:05 664690 [373E910] 0x01 ->
__qos_policy_validate_pkey: ERR
AC15: pkey 0x0002 in match rule - overriding partition SL (0) with QoS
Level SL (3)
Nov 25 13:13:05 664681 [373E910] 0x01 ->
__qos_policy_validate_pkey: ERR
AC15: pkey 0x0001 in match rule - overriding partition SL (0) with QoS
Level SL (2)
Nov 25 13:13:05 664670 [373E910] 0x01 ->

__qos_policy_validate_pkey: ERR
AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with QoS Level SL (1)

Quoted 11 lines

Quoted 6 lines

Quoted 5 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Thanks for your help.

Vincent

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 3 lines · Yevgeny Kliteynik

" It looks like in 'datagram' mode, the SL weights
do not seem to be applied, or maybe this is an artifact of IPoIB in 'datagram mode' "

yes, there's no reason for connected mode to behave differently wrt to QoS/SL assignment from the SM, as both modes get their SL from the path record provided by the SM and both mode use the same code for the path query...

Quoted 4 lines

Have you checked that in this mode you do get the right SL for each child interface by shutting off the relevant SL (mapping it to VL15)?

seeing what SL is provided by the SM in return to the path query is trivial, either through the opensm logs or the ipoib ones, e.g here you see that ib1 got SL 0 on its Path to GID fe80:0000:0000:0000:0008:f104:0399:3c92 LID 0x0006 which is
10.10.0.91

Quoted 6 lines

ifdown ib1
echo 1 > /sys/module/ib_ipoib/parameters/debug_level ifup ib1
ping 10.10.0.91
dmesg | grep ib1

Quoted 7 lines

ib1: Start path record lookup for fe80:0000:0000:0000:0008:f104:0399:3c92 MTU > 0
ib1: PathRec LID 0x0006 for GID fe80:0000:0000:0000:0008:f104:0399:3c92
ib1: Created ah ffff81021ddda180
ib1: created address handle ffff81021ddda500 for LID 0x0006, SL 0

# ip neigh show dev ib1
10.10.0.91 lladdr 80:00:00:49:fe:80:00:00:00:00:00:00:00:08:f1:04:03:99:3c:92 REACHABLE

Quoted 34 lines · sebastien dugue

Hi Yevgeny,

Quoted 7 lines · On Thu, 03 Dec 2009 10:01:28 +0200 Yevgeny Kliteynik…

Sebastien,

I noticed that you found the problem in IPoIB child interfaces configuration. Glad that this worked out well.

My question is about the note that you left in the issue:

Quoted 7 lines

" It looks like in 'datagram' mode, the SL weights
do not seem to be applied, or maybe this is an artifact of IPoIB in 'datagram mode' "

Have you checked that in this mode you do get the right SL for each child interface by shutting off the relevant SL (mapping it to VL15)?

Yes, SL to VL mapping is OK.

Quoted 3 lines

If yes, then what you're saying is that you see that interfaces use the right SL and VL, but you don't see any arbitration between VLs?

Right, whatever the weights I put in the vlarbs tables have absolutely
no effect when IPoIB is in datagram mode. I don't know if it's an arbitration problem (don't think so) or an IPoIB problem.

Could be that due to the 2044 bytes MTU in datagram mode, iperf
spends much time not doing transfers and fails to provide enough data to the interfaces. Don't know.

Quoted 2 lines

Once I switched to connected mode, with a 65520 bytes MTU, things
started to work OK with a much better overall combined bandwidth.

OK, then "a much better overall combined bandwidth" is an answer here. VL arbitration kicks in only when you saturate the link. If you don't, there's no point doing arbitration, because HW is able so serve any packet that comes w/o the need to prioritize.