mlx4: IB_EVENT_PATH_MIG not generated on path migration

Hello,

We are using Linux 3.14.29 with mlx4 drivers and trying to use automatic path migration for RC connections.
The path migration works, but no IB_EVENT_PATH_MIG* events are generated when it happens.

APM is configured on machine A by the following code snippet:

        [..]
        err = ib_query_qp(qp, &qp_attr, 0, &init_attr);                                                                                            
        if (err)                                                                                                                                   
                return err;                                                                                                                        
                                                                                                                                                   
        memcpy(&qp_attr.alt_ah_attr, &qp_attr.ah_attr,
               sizeof(qp_attr.alt_ah_attr));
                                                                                                                                                   
        qp_attr.alt_pkey_index          = qp_attr.pkey_index;
        qp_attr.alt_timeout             = qp_attr.timeout;
        qp_attr.path_mig_state          = IB_MIG_REARM;
        qp_attr.alt_ah_attr.dlid        = dlid; /* peer's 2. port dlid */

        err = ib_modify_qp(con->qp, &qp_attr,
                           IB_QP_ALT_PATH | IB_QP_PATH_MIG_STATE);

(I experiment with setting also the ah_attr.port_num and alt_port_num attributes, they seem to have no effect.)

The QP event handler is registered when calling rdma_create_qp():

        [..]
        init_attr.event_handler = qp_event_handler;
        init_attr.qp_context = con;
        ret = rdma_create_qp(cm_id, pd, &init_attr);

Both machines have MT26428 HCAs.
1 Port of machine A, both ports of machine B are connected to a switch. We establish the connection to the 1. port of machine B, unplug the corresponding cable and the path is migrated transparently to the second port of machine B.
The connection can be used like nothing happens.

As soon as the migration happens I expect that a IB_EVENT_PATH_MIG* is generated on the QP event handler, but I don't receive any QP events.

We need the event notification to rearm the path migration state machine for another migration.

Does anybody have an idea what could be wrong?
Are the PATH_MIG* notifications with mlx4 drivers are working for somebody?

thanks

Fabian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

IIRC rdmacm does not forward these events.

There is a lot missing to support APM with RDMA CM, it doesn't do any of the CM state machine operations to coordinate loading alternate paths, for instance.

qp_event_handler is a RDMA CM event handler, it handles RDMA_CM_EVENT_* and there is no RDMA_CM_EVENT_PATH_MIGRATED.

You'd need to listen to the verbs level events directly, I don't recall if this is possible with RDMA CM or not, sorry.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

2015-02-10 19:00 GMT+01:00 Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>:

Thanks Jason for reply.

We are working in kernel module, we subscribe qp_event_handler when create qp, we've checked mlx4 driver(mlx4_core & mlx4_ib), looks they do report qp event. But strangely we never receive the call in our callback function.

Forgot to mention, the switch is from Qlogic, in case it matters.

Regards,
Jack

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hello Jason,

thanks for your answer.

Alternate paths can be set directly at the QP.

The RDMA CM event handler is registered by the rdma_create_id() call.

The event handler specified in the struct ib_qp_init_attr * parameter of rdma_create_qp() is registered as QP event handler. rdma_create_qp() calls ib_create_qp() and sets qp->event_handler to ib_qp_init_attr->event_handler.

Fabian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

APM requires a certain amount of synchronization between the local and remote QP state, particularly if you want it to work more than once. That is supposed to be done with CMA mads.

Okay, right, I was thinking of something else.

We had exactly this working a few months ago for a demo, in kernel using Lustre and RDMA CM. I checked the patches we made and yes, the event_handler was being properly called with IB_EVENT_PATH_MIG on mlx4 and qib cards.

IIRC we may have needed the latest firmware on the mlx side.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Jason,

Ok, we discovered functions to configure APM in ib_cm.h. We will investigate if we can use them to setup APM.

We will also try a firmware upgrade.

thanks a lot!

Fabian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hello Jason,

[..]

are your patches for APM support with RDMA CM available somewhere? We would be glad to test them :-)

regards

Fabian