Skip to contentSkip to conversation

mlx4: Fix unneeded return error in eth_link_query_port

PATCH Applied Eli Cohen 4 participants 29 messages Oct 24, 2010 – Oct 26, 2010 16 messages not in the archive

Subsystems (2)
InfiniBand Mellanox MLX4 IB driver
lore
Share & tools
lore

Eli Cohen

author
#0 lore raw

PATCH 1/1 mlx4: Fix unneeded return error in eth_link_query_port

eth_link_query_port() returns error if a netdevice was not yet associated with the IBoE port. This is not required since we already initialize the link as down. On the other hand, we need other information that the query provides. Specifically, this can cause a failure to initilize an IBoE device after this commit 5eb620c8, which calls ib_query_port().
Fix this by always returning success.

Signed-off-by Eli Cohen <eli-VPRAkNaXOzVS1MOuV/[email protected]>
---
 drivers/infiniband/hw/mlx4/main.c |    7 ++-----
 1 files changed, 2 insertions(+), 5 deletions(-)
drivers/infiniband/hw/mlx4/main.c +2 -5
index 30cd111..bf3e20c 100644
@@ -217,7 +217,6 @@ static int eth_link_query_port(struct ib_device *ibdev, u8 port,
 {
 	struct mlx4_ib_iboe *iboe = &to_mdev(ibdev)->iboe;
 	struct net_device *ndev;
-	int err = 0;
 	enum ib_mtu tmp;
 
 	props->active_width	= IB_WIDTH_4X;

@@ -237,10 +236,8 @@ static int eth_link_query_port(struct ib_device *ibdev, u8 port,
 	props->active_mtu	= IB_MTU_256;
 	spin_lock(&iboe->lock);
 	ndev = iboe->netdevs[port - 1];
-	if (!ndev) {
-		err = -ENOMEM;
+	if (!ndev)
 		goto out;
-	}
 
 	tmp = iboe_get_mtu(ndev->mtu);
 	props->active_mtu = tmp ? min(props->max_mtu, tmp) : IB_MTU_256;

@@ -251,7 +248,7 @@ static int eth_link_query_port(struct ib_device *ibdev, u8 port,
 
 out:
 	spin_unlock(&iboe->lock);
-	return err;
+	return 0;
 }
 
 static int mlx4_ib_query_port(struct ib_device *ibdev, u8 port,
Signature · 6 lines
1.7.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#3 lore raw
Quoted 3 lines · Roland Dreier

applied, thanks.

(I didn't introduce this bug in my monkeying with the patches, did I?)

No you did not. It was there already but we never noticed before Yossi's patch.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw
Quoted 2 lines

No you did not. It was there already but we never noticed before Yossi's patch.

But AFAICT Yossi's patch (5eb620c8) went into 2.6.22 about 2.5 years ago... wasn't that already there way before the IBoE stuff started?

- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#6 lore raw
Quoted 6 lines · Roland Dreier

 > No you did not. It was there already but we never noticed before > Yossi's patch.

But AFAICT Yossi's patch (5eb620c8) went into 2.6.22 about 2.5 years ago... wasn't that already there way before the IBoE stuff started?

I see... I think the reason it started failing comes from this portion of patch 8:

+ mlx4_ib_port_link_layer(ibdev, port) == IB_LINK_LAYER_INFINIBAND ?
+ ib_link_query_port(ibdev, port, props, out_mad) :
+ eth_link_query_port(ibdev, port, props, out_mad);

I discarded the value returned and you added it as it should be. So the fix is valid anyway, but the changelog has to be changed, probably to something like:

eth_link_query_port() returns error if a netdevice was not yet associated with the IBoE port. This is not required since we already initialize the link as down. Failure to do so can cause mlx4_ib loading to fail.
Fix this by always returning success.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): AANLkTimb++kFYFXCWajBGACpA1OpCXyyeyD-98Ed3uTu-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#8 lore raw

OK... since my "fix" for the missing return of the error value ended up breaking things, I just decided to fold this second fix into the original patch. The end result is the same code but I didn't think keeping the buggy point in the middle of history helped anything.

- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#10 lore raw
Quoted 5 lines · Roland Dreier

OK... since my "fix" for the missing return of the error value ended up breaking things, I just decided to fold this second fix into the original patch. The end result is the same code but I didn't think keeping the buggy point in the middle of history helped anything.

Sure, makes sense.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#11 lore raw
In reply to a message not in the archive
Quoted 6 lines · Eli Cohen
Quoted 3 lines · Roland Dreier
Quoted 1 line

No you did not. It was there already but we never noticed before Yossi's patch.

But AFAICT Yossi's patch (5eb620c8) went into 2.6.22 about 2.5 years ago... wasn't that already there way before the IBoE stuff started?

I see... I think the reason it started failing comes from this portion of patch 8:

I pulled/built/booted with the for-next branch of Roland's tree, and I can't get IB link for the node, I don't think this is my problem, since I'm on L2 IB and not Eth, but should this work with pre 2.7 firmware?! if not, maybe patch the mlx4 driver to print some error,

Quoted 106 lines
# ibv_devinfo
hca_id: mlx4_0
        transport:                      InfiniBand (0)
        fw_ver:                         2.6.818
        node_guid:                      0002:c903:0002:6be2
        sys_image_guid:                 0002:c903:0002:6be5
        vendor_id:                      0x02c9
        vendor_part_id:                 26418
        hw_ver:                         0xA0
        board_id:                       MT_0A50110002
        phys_port_cnt:                  2
                port:   1
                        state:                  PORT_INIT (2)
                        max_mtu:                2048 (4)
                        active_mtu:             2048 (4)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                port:   2
                        state:                  PORT_INIT (2)
                        max_mtu:                2048 (4)
                        active_mtu:             2048 (4)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
# dmesg
mlx4_core: Mellanox ConnectX core driver v0.01 (May 1, 2007)
mlx4_core: Initializing 0000:0b:00.0
mlx4_core 0000:0b:00.0: PCI INT A -> GSI 30 (level, low) -> IRQ 30
mlx4_core 0000:0b:00.0: setting latency timer to 64
mlx4_core 0000:0b:00.0: FW version 2.6.818 (cmd intf rev 3), max commands 16
mlx4_core 0000:0b:00.0: Catastrophic error buffer at 0x1f020, size 0x10, BAR 0
mlx4_core 0000:0b:00.0: FW size 385 KB
mlx4_core 0000:0b:00.0: Clear int @ f0058, BAR 0
mlx4_core 0000:0b:00.0: Mapped 26 chunks/6168 KB for FW.
mlx4_core 0000:0b:00.0: BlueFlame available (reg size 512, regs/page 256)
mlx4_core 0000:0b:00.0: Base MM extensions: flags 00000cc0, rsvd L_Key 00000500
mlx4_core 0000:0b:00.0: Max ICM size 4294967296 MB
mlx4_core 0000:0b:00.0: Max QPs: 16777216, reserved QPs: 64, entry size: 256
mlx4_core 0000:0b:00.0: Max SRQs: 16777216, reserved SRQs: 64, entry size: 128
mlx4_core 0000:0b:00.0: Max CQs: 16777216, reserved CQs: 128, entry size: 128
mlx4_core 0000:0b:00.0: Max EQs: 512, reserved EQs: 4, entry size: 128
mlx4_core 0000:0b:00.0: reserved MPTs: 16, reserved MTTs: 16
mlx4_core 0000:0b:00.0: Max PDs: 8388608, reserved PDs: 4, reserved UARs: 1
mlx4_core 0000:0b:00.0: Max QP/MCG: 8388608, reserved MGMs: 0
mlx4_core 0000:0b:00.0: Max CQEs: 4194304, max WQEs: 16384, max SRQ WQEs: 16384
mlx4_core 0000:0b:00.0: Local CA ACK delay: 15, max MTU: 4096, port width cap: 3
mlx4_core 0000:0b:00.0: Max SQ desc size: 1008, max SQ S/G: 62
mlx4_core 0000:0b:00.0: Max RQ desc size: 512, max RQ S/G: 32
mlx4_core 0000:0b:00.0: Max GSO size: 131072
mlx4_core 0000:0b:00.0: DEV_CAP flags:
mlx4_core 0000:0b:00.0:     RC transport
mlx4_core 0000:0b:00.0:     UC transport
mlx4_core 0000:0b:00.0:     UD transport
mlx4_core 0000:0b:00.0:     XRC transport
mlx4_core 0000:0b:00.0:     FCoIB support
mlx4_core 0000:0b:00.0:     SRQ support
mlx4_core 0000:0b:00.0:     IPoIB checksum offload
mlx4_core 0000:0b:00.0:     P_Key violation counter
mlx4_core 0000:0b:00.0:     Q_Key violation counter
mlx4_core 0000:0b:00.0:     Big LSO headers
mlx4_core 0000:0b:00.0:     APM support
mlx4_core 0000:0b:00.0:     Atomic ops support
mlx4_core 0000:0b:00.0:     Address vector port checking support
mlx4_core 0000:0b:00.0:     UD multicast support
mlx4_core 0000:0b:00.0:     Router support
mlx4_core 0000:0b:00.0:     IBoE support
mlx4_core 0000:0b:00.0:   profile[ 0] (  CMPT): 2^26 entries @ 0x         0, size 0x 100000000
mlx4_core 0000:0b:00.0:   profile[ 1] (RDMARC): 2^21 entries @ 0x 100000000, size 0x   4000000
mlx4_core 0000:0b:00.0:   profile[ 2] (   MTT): 2^20 entries @ 0x 104000000, size 0x   4000000
mlx4_core 0000:0b:00.0:   profile[ 3] (    QP): 2^17 entries @ 0x 108000000, size 0x   2000000
mlx4_core 0000:0b:00.0:   profile[ 4] (  ALTC): 2^17 entries @ 0x 10a000000, size 0x    800000
mlx4_core 0000:0b:00.0:   profile[ 5] (   SRQ): 2^16 entries @ 0x 10a800000, size 0x    800000
mlx4_core 0000:0b:00.0:   profile[ 6] (    CQ): 2^16 entries @ 0x 10b000000, size 0x    800000
mlx4_core 0000:0b:00.0:   profile[ 7] (  DMPT): 2^17 entries @ 0x 10b800000, size 0x    800000
mlx4_core 0000:0b:00.0:   profile[ 8] (   MCG): 2^13 entries @ 0x 10c000000, size 0x    200000
mlx4_core 0000:0b:00.0:   profile[ 9] (  AUXC): 2^17 entries @ 0x 10c200000, size 0x     20000
mlx4_core 0000:0b:00.0:   profile[10] (    EQ): 2^04 entries @ 0x 10c220000, size 0x      1000
mlx4_core 0000:0b:00.0: HCA context memory: reserving 4393092 KB
mlx4_core 0000:0b:00.0: 4393092 KB of HCA context requires 8620 KB aux memory.
mlx4_core 0000:0b:00.0: Mapped 37 chunks/8620 KB for ICM aux.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/256 KB at 0 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/256 KB at 40000000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/256 KB at 80000000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/4 KB at c0000000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/4 KB at 10c220000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/256 KB at 104000000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/256 KB at 10b800000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/256 KB at 108000000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/128 KB at 10c200000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/256 KB at 10a000000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/256 KB at 100000000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/256 KB at 10b000000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/256 KB at 10a800000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/256 KB at 10c000000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/256 KB at 10c040000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/256 KB at 10c080000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/256 KB at 10c0c0000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/256 KB at 10c100000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/256 KB at 10c140000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/256 KB at 10c180000 for ICM.
mlx4_core 0000:0b:00.0: Mapped 1 chunks/256 KB at 10c1c0000 for ICM.
mlx4_core 0000:0b:00.0: irq 82 for MSI/MSI-X
mlx4_core 0000:0b:00.0: irq 83 for MSI/MSI-X
Quoted 8 lines
mlx4_core 0000:0b:00.0: irq 84 for MSI/MSI-X
mlx4_core 0000:0b:00.0: irq 85 for MSI/MSI-X
mlx4_core 0000:0b:00.0: irq 86 for MSI/MSI-X
mlx4_core 0000:0b:00.0: NOP command IRQ test passed
mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.4.1.1 (June 2009)
mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (April 4, 2008)
ADDRCONF(NETDEV_UP): ib0: link is not ready
ADDRCONF(NETDEV_UP): ib1: link is not ready

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4CC54D4E.7050203-hKgKHo2Ms0FWk0Htik3J/[email protected]

#13 lore raw
Quoted 3 lines

I pulled/built/booted with the for-next branch of Roland's tree, and I can't get IB link for the node, I don't think this is my problem, since I'm on L2 IB and not Eth, but should this work with pre 2.7 firmware?! if not, maybe patch the mlx4 driver to print some error,

Oh, I also see an error printed by some script provided by the RHEL5 libmlx4 package,

Quoted 20 lines
# modprobe mlx4_ib
awk: /etc/ofed/setup-mlx4.awk:6: (FILENAME=/etc/ofed/mlx4.conf FNR=21) fatal: cannot open file `/sbin/setup-mlx4' for reading (No such file or directory)
# rpm -qf /etc/ofed/setup-mlx4.awk
libmlx4-1.0.1-5.el5
# rpm -ql libmlx4-1.0.1-5.el5
/etc/libibverbs.d/mlx4.driver
/etc/modprobe.d/libmlx4.conf
/etc/ofed/mlx4.conf
/etc/ofed/setup-mlx4.awk
/usr/lib64/libmlx4-rdmav2.so
/usr/lib64/libmlx4.so
/usr/share/doc/libmlx4-1.0.1
/usr/share/doc/libmlx4-1.0.1/AUTHORS
/usr/share/doc/libmlx4-1.0.1/COPYING
/usr/share/doc/libmlx4-1.0.1/README
/etc/libibverbs.d/mlx4.driver
/etc/modprobe.d/libmlx4.conf
Quoted 8 lines
/etc/ofed/mlx4.conf
/etc/ofed/setup-mlx4.awk
/usr/lib/libmlx4-rdmav2.so
/usr/lib/libmlx4.so
/usr/share/doc/libmlx4-1.0.1
/usr/share/doc/libmlx4-1.0.1/AUTHORS
/usr/share/doc/libmlx4-1.0.1/COPYING
/usr/share/doc/libmlx4-1.0.1/README

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4CC54ED7.6030303-hKgKHo2Ms0FWk0Htik3J/[email protected]

author
#15 lore raw
Quoted 3 lines · Or Gerlitz

Oh, I also see an error printed by some script provided by the RHEL5 libmlx4 package,

You mean you're using for-next on a RHEL5 filesystem?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#16 lore raw

Re: can't get IB link with the for-next branch / IBoE patches (was "mlx4: Fix unneeded return error...")

In reply to a message not in the archive
Quoted 3 lines

I pulled/built/booted with the for-next branch of Roland's tree, and I can't get IB link for the node, I don't think this is my problem, since I'm on L2 IB and not Eth, but should this work with pre 2.7 firmware?! if not, maybe patch the mlx4 driver to print some error,

okay, I verified that with 2.6.36 this node gets IB link and IPoIB is working fine, so it must be something in or related to the for-next branch, I assume around the IBoE patches that touch mlx4 which make this failure to happen. With 2.6.36 I also see the "awk: /etc/ofed/setup-mlx4.awk:6: (FILENAME=/etc/ofed/mlx4.conf FNR=21) fatal: cannot open file `/sbin/setup-mlx4' for reading (No such file or directory)" warning when loading mlx4_ib, but it doesn't disruptive in the sense that the node works fine, IB wise.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4CC5604D.2080803-hKgKHo2Ms0FWk0Htik3J/[email protected]

#18 lore raw

Re: can't get IB link with the for-next branch / IBoE patches

Quoted 8 lines

okay, I verified that with 2.6.36 this node gets IB link and IPoIB is working fine, so it must be something in or related to the for-next branch, I assume around the IBoE patches that touch mlx4 which make this failure to happen. With 2.6.36 I also see the "awk:
/etc/ofed/setup-mlx4.awk:6: (FILENAME=/etc/ofed/mlx4.conf FNR=21)
fatal: cannot open file `/sbin/setup-mlx4' for reading (No such file or directory)" warning when loading mlx4_ib, but it doesn't disruptive in the sense that the node works fine, IB wise.

I suspect I broke either the UD header packing or the build_mlx_header function when I "cleaned up" the patches. I see the same problem, I'll take a look today.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#20 lore raw

Re: can't get IB link with the for-next branch / IBoE patches

Quoted 4 lines · Roland Dreier

I suspect I broke either the UD header packing or the build_mlx_header function when I "cleaned up" the patches. I see the same problem, I'll take a look today.

I think this will fix things up. The + operator has precedence over the ? operator so we end up with packet_length equal IB_GRH_BYTES / 4 which is wrong.

drivers/infiniband/core/ud_header.c +1 -1
index 7e5d224..bb7e192 100644
@@ -241,7 +241,7 @@ void ib_ud_header_init(int     		    payload_bytes,
 		packet_length = (IB_LRH_BYTES	+
 				 IB_BTH_BYTES	+
 				 IB_DETH_BYTES	+
-				 grh_present ? IB_GRH_BYTES : 0 +
+				 (grh_present ? IB_GRH_BYTES : 0) +
 				 payload_bytes	+
 				 4		+ /* ICRC     */
 				 3) / 4;	  /* round up */

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#21 lore raw

Re: can't get IB link with the for-next branch / IBoE patches

Quoted 9 lines · Eli Cohen
Quoted 3 lines · Roland Dreier

I suspect I broke either the UD header packing or the build_mlx_header function when I "cleaned up" the patches.  I see the same problem, I'll take a look today.

I think this will fix things up. The + operator has precedence over the ? operator so we end up with packet_length equal IB_GRH_BYTES / 4 which is wrong.

Once you guys feel to have a fix I would be happy to give Roland's for-next bits some further basic kernel (e.g IB link up/down, IPoIB, running SM on a node with IBoE patches) testing and a bit of more advanced (e.g IB/iSER, IB/RDS [Andy]) testing to see that things are in place with L2 IB, I would recommend also the iWARP folks to do the same as the addr/rdma-cm modules were also modified.

The merge window still has about 9 days, so we're okay with delaying the push in 1-2 days, thoughts people?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#22 lore raw

Re: can't get IB link with the for-next branch / IBoE patches

In reply to Eli Cohen
Quoted 3 lines

I think this will fix things up. The + operator has precedence over the ? operator so we end up with packet_length equal IB_GRH_BYTES / 4 which is wrong.

Yep, looks like that's where my cleanup broke things. I rolled this in and pushed it out; I'm testing it myself now.

- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

#24 lore raw

Re: can't get IB link with the for-next branch / IBoE patches

Quoted 2 lines

Yep, looks like that's where my cleanup broke things. I rolled this in and pushed it out; I'm testing it myself now.

My IB port comes to active now, I think that fixed things.

- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

#26 lore raw

Re: can't get IB link with the for-next branch / IBoE patches

Quoted 4 lines · Roland Dreier
Quoted 2 lines

Yep, looks like that's where my cleanup broke things. I rolled this in and pushed it out; I'm testing it myself now.

My IB port comes to active now, I think that fixed things.

same here, I have IB port coming to active and basic IPoIB, opensm working okay on the node with the current for-next/IBoE bits

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4CC6A051.3010703-hKgKHo2Ms0FWk0Htik3J/[email protected]

#28 lore raw

Re: can't get IB link with the for-next branch / IBoE patches

Quoted 2 lines

I have IB port coming to active and basic IPoIB, opensm working okay on the node with the current for-next/IBoE bits

doing a little bit stress testing, I came across the below oops, when running IPoIB and couple of iperf/udp sessions, it doesn't look like a problem in the IB stack.

Also with rds, using rds-stress from rds-tools-1.5-1.el5 and "rds-stress -s 192.168.20.18 -p 4000 -t 1
-q 1K -a 1K -D 1M" on the client side, the node running the for-next/IBoE bits and acting as the passive side
of the test, got hanged. Also here, this could be a bug in RDS and not in the IBoE patches, I know that the rds guys queued about a hundred! patches for 2.6.37 so with these patches things might be better. I have the oops trace in jpg, will send to Andy, Roland and Eli. I guess we can continue these tests for 2-3 days and have the push over the weekend, or push it before and get fixes if needed through the -rc cycle.

Oct 26 12:36:30 nsg2 kernel: BUG: spinlock bad magic on CPU#0, iperf/20845
Oct 26 12:36:30 nsg2 kernel:  lock: ffffffff81663ef8, .magic: 00000000, .owner: <none>/-1, .owner_cpu: 0
Oct 26 12:36:30 nsg2 kernel: Pid: 20845, comm: iperf Not tainted 2.6.36-rc5-42052-gce806e1 #1
Oct 26 12:36:30 nsg2 kernel: Call Trace:
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff811542b7>] ? do_raw_spin_lock+0x22/0x122
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff81268b2b>] ? dev_queue_xmit+0x10d/0x346
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff8128ca13>] ? ip_push_pending_frames+0x2bf/0x318
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff812a7e66>] ? udp_push_pending_frames+0x2d2/0x351
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff812a970c>] ? udp_sendmsg+0x4b0/0x59c
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff8112e9f7>] ? cap_socket_sendmsg+0x0/0x3
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff812e7d8e>] ? common_interrupt+0xe/0x13
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff8112e9f7>] ? cap_socket_sendmsg+0x0/0x3
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff81256bbb>] ? sock_aio_write+0xf5/0x10d
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff810029ae>] ? reschedule_interrupt+0xe/0x20
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff812e7d8e>] ? common_interrupt+0xe/0x13
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff812e7d8e>] ? common_interrupt+0xe/0x13
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff810b9b49>] ? do_sync_write+0xab/0xeb
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff812e7abf>] ? _raw_spin_unlock_irq+0x9/0xd
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff8112e83f>] ? security_file_permission+0x18/0x6b
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff810ba1f7>] ? vfs_write+0xbe/0x132
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff810ba754>] ? sys_write+0x45/0x6e
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff81001e6b>] ? system_call_fastpath+0x16/0x1b

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4CC6C75F.8030103-hKgKHo2Ms0FWk0Htik3J/[email protected]

#30 lore raw

Re: can't get IB link with the for-next branch / IBoE patches

Quoted 2 lines

doing a little bit stress testing, I came across the below oops, when running IPoIB and couple of iperf/udp sessions, it doesn't look like a problem in the IB stack.

To trigger this I run from client node the following "iperf -uc 192.168.21.18 -l 64000 -t 72000 -i 1 -b 40g -d -P 4" where the server node (21.18 here) was the one that has the IBoE patches and got this oops

Quoted 14 lines
Oct 26 12:36:30 nsg2 kernel: BUG: spinlock bad magic on CPU#0, iperf/20845
Oct 26 12:36:30 nsg2 kernel:  lock: ffffffff81663ef8, .magic: 00000000, .owner: <none>/-1, .owner_cpu: 0
Oct 26 12:36:30 nsg2 kernel: Pid: 20845, comm: iperf Not tainted 2.6.36-rc5-42052-gce806e1 #1
Oct 26 12:36:30 nsg2 kernel: Call Trace:
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff811542b7>] ? do_raw_spin_lock+0x22/0x122
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff81268b2b>] ? dev_queue_xmit+0x10d/0x346
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff8128ca13>] ? ip_push_pending_frames+0x2bf/0x318
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff812a7e66>] ? udp_push_pending_frames+0x2d2/0x351
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff812a970c>] ? udp_sendmsg+0x4b0/0x59c
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff8112e9f7>] ? cap_socket_sendmsg+0x0/0x3
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff812e7d8e>] ? common_interrupt+0xe/0x13
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff8112e9f7>] ? cap_socket_sendmsg+0x0/0x3
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff81256bbb>] ? sock_aio_write+0xf5/0x10d
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff810029ae>] ? reschedule_interrupt+0xe/0x20
Quoted 8 lines
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff812e7d8e>] ? common_interrupt+0xe/0x13
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff812e7d8e>] ? common_interrupt+0xe/0x13
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff810b9b49>] ? do_sync_write+0xab/0xeb
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff812e7abf>] ? _raw_spin_unlock_irq+0x9/0xd
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff8112e83f>] ? security_file_permission+0x18/0x6b
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff810ba1f7>] ? vfs_write+0xbe/0x132
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff810ba754>] ? sys_write+0x45/0x6e
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff81001e6b>] ? system_call_fastpath+0x16/0x1b

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#31 lore raw

Re: can't get IB link with the for-next branch / IBoE patches

In reply to a message not in the archive

I'll try to reproduce here. Anyway, I don't see how it could possibly be related to IBoE.

Quoted 25 lines · Or Gerlitz
Quoted 2 lines

I have IB port coming to active and basic IPoIB, opensm working okay on the node with the current for-next/IBoE bits

doing a little bit stress testing, I came across the below oops, when running IPoIB and couple of iperf/udp sessions, it doesn't look like a problem in the IB stack.

Also with rds, using rds-stress from rds-tools-1.5-1.el5 and "rds-stress -s 192.168.20.18 -p 4000 -t 1
-q 1K -a 1K -D 1M" on the client side, the node running the for-next/IBoE bits and acting as the passive side
of the test, got hanged. Also here, this could be a bug in RDS and not in the IBoE patches, I know that the rds guys queued about a hundred! patches for 2.6.37 so with these patches things might be better. I have the oops trace in jpg, will send to Andy, Roland and Eli. I guess we can continue these tests for 2-3 days and have the push over the weekend, or push it before and get fixes if needed through the -rc cycle.

Oct 26 12:36:30 nsg2 kernel: BUG: spinlock bad magic on CPU#0, iperf/20845
Oct 26 12:36:30 nsg2 kernel:  lock: ffffffff81663ef8, .magic: 00000000, .owner: <none>/-1, .owner_cpu: 0
Oct 26 12:36:30 nsg2 kernel: Pid: 20845, comm: iperf Not tainted 2.6.36-rc5-42052-gce806e1 #1
Oct 26 12:36:30 nsg2 kernel: Call Trace:
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff811542b7>] ? do_raw_spin_lock+0x22/0x122
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff81268b2b>] ? dev_queue_xmit+0x10d/0x346
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff8128ca13>] ? ip_push_pending_frames+0x2bf/0x318
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff812a7e66>] ? udp_push_pending_frames+0x2d2/0x351
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff812a970c>] ? udp_sendmsg+0x4b0/0x59c
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff8112e9f7>] ? cap_socket_sendmsg+0x0/0x3
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff812e7d8e>] ? common_interrupt+0xe/0x13
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff8112e9f7>] ? cap_socket_sendmsg+0x0/0x3
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff81256bbb>] ? sock_aio_write+0xf5/0x10d
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff810029ae>] ? reschedule_interrupt+0xe/0x20
Quoted 8 lines
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff812e7d8e>] ? common_interrupt+0xe/0x13
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff812e7d8e>] ? common_interrupt+0xe/0x13
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff810b9b49>] ? do_sync_write+0xab/0xeb
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff812e7abf>] ? _raw_spin_unlock_irq+0x9/0xd
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff8112e83f>] ? security_file_permission+0x18/0x6b
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff810ba1f7>] ? vfs_write+0xbe/0x132
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff810ba754>] ? sys_write+0x45/0x6e
Oct 26 12:36:30 nsg2 kernel:  [<ffffffff81001e6b>] ? system_call_fastpath+0x16/0x1b

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#32 lore raw

Re: can't get IB link with the for-next branch / IBoE patches (was "mlx4: Fix unneeded return error...")

In reply to a message not in the archive
Quoted 3 lines

I pulled/built/booted with the for-next branch of Roland's tree, and I can't get IB link for the node, I don't think this is my problem, since I'm on L2 IB and not Eth, but should this work with pre 2.7 firmware?! if not, maybe patch the mlx4 driver to print some error,

okay, I verified that with 2.6.36 this node gets IB link and IPoIB is working fine, so it must be something in or related to the for-next branch, I assume around the IBoE patches that touch mlx4 which make this failure to happen. With 2.6.36 I also see the "awk: /etc/ofed/setup-mlx4.awk:6: (FILENAME=/etc/ofed/mlx4.conf FNR=21) fatal: cannot open file `/sbin/setup-mlx4' for reading (No such file or directory)" warning when loading mlx4_ib, but it doesn't disruptive in the sense that the node works fine, IB wise.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#33 lore raw
In reply to a message not in the archive
Quoted 5 lines · Or Gerlitz

I pulled/built/booted with the for-next branch of Roland's tree, and I can't get IB link for the node, I don't think this is my problem, since I'm on L2 IB and not Eth, but should this work with pre 2.7 firmware?! if not, maybe patch the mlx4 driver to print some error,

Hi Or,
IBoE will not work with firmware prior to 2.7.000. I don't think an error message is required in this case.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#34 lore raw
Quoted 2 lines · Eli Cohen

IBoE will not work with firmware prior to 2.7.000. I don't think an error message is required in this case.

But I'm on **IB** not IBoE, I don't think you mean that the Linux kernel IB stack is not functional over pre-2.7 firmware with the IBoE patches?! are you?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): AANLkTi=ZxB4b463OOS6YGxTJSKxjyCj8vy0rNtj0n+uA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

author
#36 lore raw
Quoted 5 lines · Or Gerlitz

On Mon, Oct 25, 2010 at 1:34 PM, Eli Cohen <[email protected]> wrote:

But I'm on **IB** not IBoE, I don't think you mean that the Linux kernel IB stack is not functional over pre-2.7 firmware with the IBoE patches?! are you?

Of course not. I just noticed that the IB link for IB link layer does come up, is that what you're seeing?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#37 lore raw
Quoted 2 lines · Eli Cohen

Of course not. I just noticed that the IB link for IB link layer does come up, is that what you're seeing?

No, I didn't have IB Link when I used the for-next bits
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): AANLkTimaEcFZMnYE+G3osTWzPkfxuBpRMtrrXF7xUPYv-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#40 lore raw
Quoted 3 lines · Eli Cohen
Quoted 1 line · Or Gerlitz

No, I didn't have IB Link when I used the for-next bits

Can you summarize what is the problem that you're seeing?

Eli, this is pretty simple, I do the following
1. pull/build/boot Roland's for-next
2. modprobe mlx4_ib
--> the port state is INIT forever, is that clear?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): AANLkTik_4OzMLMWXud89m_rF47OQ3Wji9R_Bye+0DcTV-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

author
#42 lore raw

First, I suggest that you sync your tree again.
Second, I assume your link layer is IB since Ethernet will show either Down or Active so I think you should be fine after syncing. If that does not answer you, please don't spare words to explain what is the problem that you're seeing.

Quoted 12 lines · Or Gerlitz
Quoted 3 lines · Eli Cohen
Quoted 1 line · Or Gerlitz

No, I didn't have IB Link when I used the for-next bits

Can you summarize what is the problem that you're seeing?

Eli, this is pretty simple, I do the following
1. pull/build/boot Roland's for-next
2. modprobe mlx4_ib
--> the port state is INIT forever, is that clear?

Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): AANLkTi=yZUoexwVUCfbeGypEWC_8=oZaMu9mBTF+VJgq-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#44 lore raw
Quoted 1 line · Eli Cohen

First, I suggest that you sync your tree again.

I'll be able to do that tomorrow

Quoted 1 line

Second, I assume your link layer is IB since

I wrote black-over-white that its IB, what wasn't clear here?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html