IB atomic operations

Hi,
I am trying to use IB atomic operations, specifically fetch and add.
I tried a small example but I am unable to get it to work. The WC status is IBV_WC_REM_INV_REQ_ERR. My sample code is as follows:

sge.addr = (uintptr_t) &data;
sge.length = sizeof (uint64_t);
sge.lkey = ib_mr->lkey;
wr.opcode = IBV_WR_ATOMIC_FETCH_AND_ADD;
wr.send_flags = IBV_SEND_SIGNALED;
wr.next = NULL;
wr.num_sge = 1;
wr.sg_list = &sge;
wr.wr.atomic.remote_addr = vaddr;
wr.wr.atomic.compare_add = 1;
//wr.wr.atomic.swap = 0;
wr.wr.atomic.rkey = rkey;
wr.wr_id = 1;
ret = ibv_post_send (ib_qp, &wr, &wr_bad);

do {
	c = ibv_poll_cq (ib_cq, 1, &wc);
} while (c == 0);

if (c < 0) {
	printf ("ERROR: ibv_poll_cq failed.\n");
      return -1;
}

if (wc.status != IBV_WC_SUCCESS) {
	// print wc info.
	// print qp status.
}

Any help on this appreciated.

Regards,

Bharath

Did you register your memory with IBV_ACCESS_REMOTE_ATOMIC ?

Frank.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I have registered both the remote and local data with IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_ATOMIC | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE

I am not sure if the vendor err code is of any help but it is 138. I am also using Mellanox ConnectX-2 QDR cards.

Regards,

Bharath

Do you create your QPs with RDMA CM or do you transition them by hand ? In the later case there is a couple flags to set.

Frank.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I transition them by hand, my QP creation is done as follows:

memset (&init_attr, 0, sizeof (struct ibv_qp_init_attr)); init_attr.recv_cq = ib_cq;
init_attr.send_cq = ib_cq; init_attr.cap.max_recv_wr = IB_MAX_WR; init_attr.cap.max_send_wr = IB_MAX_WR; init_attr.cap.max_recv_sge = IB_MAX_SGE; init_attr.cap.max_send_sge = IB_MAX_SGE; init_attr.qp_type = IBV_QPT_RC;
qp = ibv_create_qp (ib_pd, &init_attr);

if (qp == NULL) {
	// QP creation error.
}

I transition the QP as follows:

// Transition QP to INIT state.
attr.qp_state = IBV_QPS_INIT;
attr.pkey_index = 0;
attr.port_num = IB_PORT;
attr.qp_access_flags = IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_ATOMIC | IBV_ACCESS_REMOTE_READ

	| IBV_ACCESS_REMOTE_WRITE;
if (ibv_modify_qp (qp, &attr, IBV_QP_STATE | IBV_QP_PKEY_INDEX |
		IBV_QP_PORT | IBV_QP_ACCESS_FLAGS) != 0) {
	// QP modify error.
}

memset (&attr, 0, sizeof (struct ibv_qp_attr));
attr.qp_state = IBV_QPS_RTR;
attr.path_mtu = ib_mtu;
attr.dest_qp_num = qpn;
attr.rq_psn = 0;
attr.max_dest_rd_atomic = IB_MAX_RD_ATOMIC;
attr.min_rnr_timer = IB_RNR_TIMER;
attr.ah_attr.is_global = 0;
attr.ah_attr.dlid = lid;
attr.ah_attr.sl = 0;
attr.ah_attr.src_path_bits = 0;
attr.ah_attr.port_num = IB_PORT;
if (ibv_modify_qp (qp, &attr, IBV_QP_STATE | IBV_QP_AV | IBV_QP_PATH_MTU | IBV_QP_DEST_QPN |
IBV_QP_RQ_PSN | IBV_QP_MAX_DEST_RD_ATOMIC |

IBV_QP_MIN_RNR_TIMER) != 0) {
	// QP modify error.
}

// Transition the QP to RTS.
attr.timeout = IB_TIMEOUT;
attr.retry_cnt = IB_RETRY_CNT;
attr.rnr_retry = IB_RNR_RETRY;
attr.qp_state = IBV_QPS_RTS;
attr.sq_psn = 0;
attr.max_rd_atomic = IB_MAX_RD_ATOMIC;
if (ibv_modify_qp (qp, &attr, IBV_QP_STATE | IBV_QP_SQ_PSN | IBV_QP_MAX_QP_RD_ATOMIC |

		IBV_QP_RETRY_CNT | IBV_QP_RNR_RETRY | IBV_QP_TIMEOUT) != 0)
{
	// QP modify error.
}

Regards,

Bharath

Sorry, I don't see what's wrong.
Frank.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Thanks, I really appreciate you taking time to help me with the issue.

Regards,

Bharath

Could there be a reason why it would fail on certain nodes but run on other nodes. I picked two other nodes and ran the attached test code. It runs fine there without any issues, the hardware is same across all nodes of the cluster.

Regards,

Bharath