CQ overrun with ib_send_bw

DISCUSSION Sumeet Lahorani 5 participants 8 messages Aug 13, 2010 – Aug 17, 2010 7 messages not in the archive InfiniBand

lore

Share & tools

lore

Sumeet Lahorani

author

2010-08-13 18:44 UTC #0 lore raw

Hi,

If I run ib_send_bw with the -a option, we seem to be getting CQ overrun errors.

Server :
[root@dscbad01 ~]# ib_send_bw
------------------------------------------------------------------
Send BW Test
Connection type : RC
Inline data is used up to 1 bytes message
local address: LID 0x24, QPN 0x1c004c, PSN 0x85c292 remote address: LID 0x2a, QPN 0x14004a, PSN 0x858358
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] ------------------------------------------------------------------

Client :
[root@dscbad03 ~]# ib_send_bw -a dscbad01
------------------------------------------------------------------
Send BW Test
Connection type : RC
Inline data is used up to 1 bytes message
local address: LID 0x2a, QPN 0x14004a, PSN 0x858358 remote address: LID 0x24, QPN 0x1c004c, PSN 0x85c292
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec]
2 1000 5.99 5.45
Completion wth error at client:
Failed status 12: wr_id 1 syndrom 0x81 scnt=600, ccnt=300

and on the client console

mlx4_core 0000:13:00.0: CQ overrun on CQN 000086
mlx4_core 0000:13:00.0: Internal error detected:

mlx4_core 0000:13:00.0:   buf[00]: 00328f6f
mlx4_core 0000:13:00.0:   buf[01]: 00000000
mlx4_core 0000:13:00.0:   buf[02]: 20070000
mlx4_core 0000:13:00.0:   buf[03]: 00000000
mlx4_core 0000:13:00.0:   buf[04]: 00328f3c
mlx4_core 0000:13:00.0:   buf[05]: 0014004a
mlx4_core 0000:13:00.0:   buf[06]: 00340000
mlx4_core 0000:13:00.0:   buf[07]: 00000044
mlx4_core 0000:13:00.0:   buf[08]: 00000804
mlx4_core 0000:13:00.0:   buf[09]: 00000804
mlx4_core 0000:13:00.0:   buf[0a]: 00000000
mlx4_core 0000:13:00.0:   buf[0b]: 00000000
mlx4_core 0000:13:00.0:   buf[0c]: 00000000
mlx4_core 0000:13:00.0:   buf[0d]: 00000000
mlx4_core 0000:13:00.0:   buf[0e]: 00000000
mlx4_core 0000:13:00.0:   buf[0f]: 00000000

This is with OFED 1.5.1 but it also happens with OFED 1.4.2. Sometimes, the node crashes because it runs out of memory but most of the time, I see just the above errors. What could be wrong?

- Sumeet

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

Ralph Campbell

2010-08-13 19:06 UTC #2 lore raw

I know there is a bug with "ib_send_bw -b" (bi-directional) since it doesn't create a CQ that is large enough for all the posted sends *and* receives. I have tried several times to get the following patch applied but I never got a reply and nothing was done.

send_bw.c +2 -0

index ddd2b73..e3f644a 100644
@@ -746,6 +746,8 @@ static struct pingpong_context *pp_init_ctx(struct ibv_device *ib_dev,
 	if (user_parm->use_mcg && !user_parm->servername) {
 		cq_rx_depth *= user_parm->num_of_clients_mcg;
 	}
+	if (user_parm->duplex)
+		cq_rx_depth += ctx->tx_depth;
 	ctx->cq = ibv_create_cq(ctx->context,cq_rx_depth, NULL, ctx->channel, 0);
 	if (!ctx->cq) {
 		fprintf(stderr, "Couldn't create CQ\n");

There should be enough CQEs in the normal case though.

Quoted 59 lines · Sumeet Lahorani

Hi,
If I run ib_send_bw with the -a option, we seem to be getting CQ overrun errors.
Server :
[root@dscbad01 ~]# ib_send_bw
------------------------------------------------------------------
Send BW Test
Connection type : RC
Inline data is used up to 1 bytes message
local address: LID 0x24, QPN 0x1c004c, PSN 0x85c292 remote address: LID 0x2a, QPN 0x14004a, PSN 0x858358
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec]
------------------------------------------------------------------
Client :
[root@dscbad03 ~]# ib_send_bw -a dscbad01
------------------------------------------------------------------
Send BW Test
Connection type : RC
Inline data is used up to 1 bytes message
local address: LID 0x2a, QPN 0x14004a, PSN 0x858358 remote address: LID 0x24, QPN 0x1c004c, PSN 0x85c292
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec]
2 1000 5.99 5.45
Completion wth error at client:
Failed status 12: wr_id 1 syndrom 0x81 scnt=600, ccnt=300
and on the client console
mlx4_core 0000:13:00.0: CQ overrun on CQN 000086
mlx4_core 0000:13:00.0: Internal error detected:
mlx4_core 0000:13:00.0:   buf[00]: 00328f6f
mlx4_core 0000:13:00.0:   buf[01]: 00000000
mlx4_core 0000:13:00.0:   buf[02]: 20070000
mlx4_core 0000:13:00.0:   buf[03]: 00000000
mlx4_core 0000:13:00.0:   buf[04]: 00328f3c
mlx4_core 0000:13:00.0:   buf[05]: 0014004a
mlx4_core 0000:13:00.0:   buf[06]: 00340000
mlx4_core 0000:13:00.0:   buf[07]: 00000044
mlx4_core 0000:13:00.0:   buf[08]: 00000804
mlx4_core 0000:13:00.0:   buf[09]: 00000804
mlx4_core 0000:13:00.0:   buf[0a]: 00000000
mlx4_core 0000:13:00.0:   buf[0b]: 00000000
mlx4_core 0000:13:00.0:   buf[0c]: 00000000
mlx4_core 0000:13:00.0:   buf[0d]: 00000000
mlx4_core 0000:13:00.0:   buf[0e]: 00000000
mlx4_core 0000:13:00.0:   buf[0f]: 00000000
This is with OFED 1.5.1 but it also happens with OFED 1.4.2. Sometimes, the node crashes because it runs out of memory but most of the time, I see just the above errors. What could be wrong?

Quoted 7 lines

- Sumeet
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 1281726396.2313.44.camel-/[email protected]

Hefty, Sean

2010-08-13 19:14 UTC #4 lore raw

Quoted 5 lines

I know there is a bug with "ib_send_bw -b" (bi-directional) since it doesn't create a CQ that is large enough for all the posted sends *and* receives. I have tried several times to get the following patch applied but I never got a reply and nothing was done.

Who's the maintainer of these tests?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): CF9C39F99A89134C9CF9C4CCB68B8DDF25A96887A2-osO9UTpF0USkrb+BlOpmy7fspsVTdybXVpNB7YpNyf8@public.gmane.org

Ralph Campbell

2010-08-13 19:21 UTC #6 lore raw

Quoted 7 lines · Sean

Quoted 5 lines
I know there is a bug with "ib_send_bw -b" (bi-directional) since it doesn't create a CQ that is large enough for all the posted sends *and* receives. I have tried several times to get the following patch applied but I never got a reply and nothing was done.
Who's the maintainer of these tests?

I believe it is:

Ido Shamai <[email protected]>

git://git.openfabrics.org/~shamoya/perftest.git

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 1281727297.2313.47.camel-/[email protected]

Tziporet Koren

2010-08-17 11:19 UTC #8 lore raw

Quoted 7 lines · Ralph Campbell

Quoted 6 lines · Sean
Quoted 5 lines
I know there is a bug with "ib_send_bw -b" (bi-directional) since it doesn't create a CQ that is large enough for all the posted sends *and* receives. I have tried several times to get the following patch applied but I never got a reply and nothing was done.

Quoted 9 lines

Quoted 1 line
Who's the maintainer of these tests?
I believe it is:
Ido Shamai <[email protected]>
git://git.openfabrics.org/~shamoya/perftest.git

Yes Ido is the maintainer, however he is on vacation till Sep. I add Amir that may help for now

Tziporet
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): E113D394D7C5DB4F8FF691FA7EE9DB443B5668DE17-WQlSmcKwN8Te+A/[email protected]

Amir Ancel

2010-08-17 11:36 UTC #10 lore raw

Hi Sean,

We've seen this issue as well.

Can you send the patch directly to us ?

Added Raz from my team which replaces Ido while he is OOO.

Thanks,

Amir Ancel
Performance Team Manager
Mellanox Technologies

-----Original Message-----
From: Tziporet Koren
Sent: Tuesday, August 17, 2010 2:19 PM
To: Ralph Campbell; Hefty, Sean; Ido Shamay; Amir Ancel
Cc: Sumeet Lahorani; [email protected]
Subject: RE: CQ overrun with ib_send_bw

Quoted 7 lines · Ralph Campbell

Quoted 6 lines · Sean
Quoted 5 lines
I know there is a bug with "ib_send_bw -b" (bi-directional) since it doesn't create a CQ that is large enough for all the posted sends *and* receives. I have tried several times to get the following patch applied but I never got a reply and nothing was done.

Quoted 9 lines

Quoted 1 line
Who's the maintainer of these tests?
I believe it is:
Ido Shamai <[email protected]>
git://git.openfabrics.org/~shamoya/perftest.git

Yes Ido is the maintainer, however he is on vacation till Sep. I add Amir that may help for now

Tziporet
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 1EEC75D0B27041449A1EEA2927D1B145380145A7DA-WQlSmcKwN8Te+A/[email protected]

Ralph Campbell

2010-08-17 18:59 UTC #12 lore raw

The patch is attached.

Quoted 36 lines · Amir Ancel

Hi Sean,
We've seen this issue as well.
Can you send the patch directly to us ?
Added Raz from my team which replaces Ido while he is OOO.
Thanks,
Amir Ancel
Performance Team Manager
Mellanox Technologies
-----Original Message-----
From: Tziporet Koren
Sent: Tuesday, August 17, 2010 2:19 PM
To: Ralph Campbell; Hefty, Sean; Ido Shamay; Amir Ancel
Cc: Sumeet Lahorani; [email protected]
Subject: RE: CQ overrun with ib_send_bw
Quoted 13 lines · Ralph Campbell
Quoted 7 lines · Sean
Quoted 5 lines
I know there is a bug with "ib_send_bw -b" (bi-directional) since it doesn't create a CQ that is large enough for all the posted sends *and* receives. I have tried several times to get the following patch applied but I never got a reply and nothing was done.
Who's the maintainer of these tests?
I believe it is:
Ido Shamai <[email protected]>

Quoted 9 lines

Quoted 3 lines
git://git.openfabrics.org/~shamoya/perftest.git
Yes Ido is the maintainer, however he is on vacation till Sep. I add Amir that may help for now
Tziporet

send_bw.c +2 -0

index ddd2b73..e3f644a 100644
@@ -746,6 +746,8 @@ static struct pingpong_context *pp_init_ctx(struct ibv_device *ib_dev,
 	if (user_parm->use_mcg && !user_parm->servername) {
 		cq_rx_depth *= user_parm->num_of_clients_mcg;
 	}
+	if (user_parm->duplex)
+		cq_rx_depth += ctx->tx_depth;
 	ctx->cq = ibv_create_cq(ctx->context,cq_rx_depth, NULL, ctx->channel, 0);
 	if (!ctx->cq) {
 		fprintf(stderr, "Couldn't create CQ\n");

A message that is not in the archive (sent off-list or not yet received): 1282071547.2313.100.camel-/[email protected]

Amir Ancel

2010-08-17 19:08 UTC #14 lore raw

Thanks,

We'll apply it soon.

Amir Ancel
Performance Team Manager
Mellanox Technologies

-----Original Message-----
From: Ralph Campbell [mailto:[email protected]]
Sent: Tuesday, August 17, 2010 9:59 PM
To: Amir Ancel
Cc: Tziporet Koren; Hefty, Sean; Ido Shamay; Sumeet Lahorani; [email protected]; Raz Baussi
Subject: RE: CQ overrun with ib_send_bw

The patch is attached.

Quoted 36 lines · Amir Ancel

Hi Sean,
We've seen this issue as well.
Can you send the patch directly to us ?
Added Raz from my team which replaces Ido while he is OOO.
Thanks,
Amir Ancel
Performance Team Manager
Mellanox Technologies
-----Original Message-----
From: Tziporet Koren
Sent: Tuesday, August 17, 2010 2:19 PM
To: Ralph Campbell; Hefty, Sean; Ido Shamay; Amir Ancel
Cc: Sumeet Lahorani; [email protected]
Subject: RE: CQ overrun with ib_send_bw
Quoted 13 lines · Ralph Campbell
Quoted 7 lines · Sean
Quoted 5 lines
I know there is a bug with "ib_send_bw -b" (bi-directional) since it doesn't create a CQ that is large enough for all the posted sends *and* receives. I have tried several times to get the following patch applied but I never got a reply and nothing was done.
Who's the maintainer of these tests?
I believe it is:
Ido Shamai <[email protected]>

Quoted 9 lines

Quoted 3 lines
git://git.openfabrics.org/~shamoya/perftest.git
Yes Ido is the maintainer, however he is on vacation till Sep. I add Amir that may help for now
Tziporet

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html