Skip to contentSkip to conversation

Bad behavior by rdma-core ?

DISCUSSION Bob Pearson 2 participants 5 messages Oct 14, 2021 InfiniBand

lore
Share & tools
lore
#0 lore raw

I have been chasing a bug in the rxe driver seen in the python tests (test_cq_events_ud). The following occurs

The first time I execute this test it creates two AHs which are allocated by rdma-core and passed to rxe_create_ah. The test attempts to destroy them (i.e. rxe_destroy_ah is called in the provider driver) but rdma-core does not destroy them (i.e. rxe_destroy_ah is not called in the kernel).

The rxe driver saves the AV state and some metadata for these AHs and keeps it since it thinks they are still active.

The second or third time I execute this test two new AHs are created by rxe_create_ah but the memory passed in from rdma-core is the same as the first test. I.e. it has recycled them but they are still active in the driver so the result is chaos.

Somehow rdma-core thinks it has destroyed the AHs but it does not call down to the driver. This only occurs for AHs AFAIK.

Bob

#1 lore raw
Quoted 16 lines · Bob Pearson

I have been chasing a bug in the rxe driver seen in the python tests (test_cq_events_ud). The following occurs

The first time I execute this test it creates two AHs which are allocated by rdma-core and passed to rxe_create_ah. The test attempts to destroy them (i.e. rxe_destroy_ah is called in the provider driver) but rdma-core does not destroy them (i.e. rxe_destroy_ah is not called in the kernel).

The rxe driver saves the AV state and some metadata for these AHs and keeps it since it thinks they are still active.

The second or third time I execute this test two new AHs are created by rxe_create_ah but the memory passed in from rdma-core is the same as the first test. I.e. it has recycled them but they are still active in the driver so the result is chaos.

Quoted 5 lines

Somehow rdma-core thinks it has destroyed the AHs but it does not call down to the driver. This only occurs for AHs AFAIK.

Bob

The cause seems simple enough.

In uverbs_cmd.c ib_uverbs_create_ah() calls rdma_create_user_ah() which eventually calls device->ops.create_user_ah() or device->ops.create_ah().

But ib_uverbs_destroy_ah does *not* call rdma_uverbs_destroy_ah() it just deletes the object.

#2 lore raw
Quoted 23 lines · Bob Pearson
Quoted 16 lines · Bob Pearson

I have been chasing a bug in the rxe driver seen in the python tests (test_cq_events_ud). The following occurs

The first time I execute this test it creates two AHs which are allocated by rdma-core and passed to rxe_create_ah. The test attempts to destroy them (i.e. rxe_destroy_ah is called in the provider driver) but rdma-core does not destroy them (i.e. rxe_destroy_ah is not called in the kernel).

The rxe driver saves the AV state and some metadata for these AHs and keeps it since it thinks they are still active.

The second or third time I execute this test two new AHs are created by rxe_create_ah but the memory passed in from rdma-core is the same as the first test. I.e. it has recycled them but they are still active in the driver so the result is chaos.

Quoted 5 lines

Somehow rdma-core thinks it has destroyed the AHs but it does not call down to the driver. This only occurs for AHs AFAIK.

Bob

Quoted 6 lines

The cause seems simple enough.

In uverbs_cmd.c ib_uverbs_create_ah() calls rdma_create_user_ah() which eventually calls device->ops.create_user_ah() or device->ops.create_ah().

But ib_uverbs_destroy_ah does *not* call rdma_uverbs_destroy_ah() it just

should be rdma_destroy_user_ah()

Quoted 2 lines

deletes the object.

maintainer
#3 lore raw
In reply to Bob Pearson
Quoted 2 lines · Bob Pearson

But ib_uverbs_destroy_ah does *not* call rdma_uverbs_destroy_ah() it just deletes the object.

ib_uverbs_destroy_ah
uobj_perform_destroy
__uobj_perform_destroy
__uobj_get_destroy

    uobj_destroy
     uverbs_destroy_uobject:

    	} else if (uobj->object) {
		ret = uobj->uapi_object->type_class->destroy_hw(uobj, reason,
								attrs);

Which calls

destroy_hw_idr_uobject
int ret = idr_type->destroy_object(uobj, why, attrs);

Which links to this:

DECLARE_UVERBS_NAMED_OBJECT(UVERBS_OBJECT_AH,

			    UVERBS_TYPE_ALLOC_IDR(uverbs_free_ah),
			    &UVERBS_METHOD(UVERBS_METHOD_AH_DESTROY));

And thus calls

static int uverbs_free_ah(struct ib_uobject *uobject,
enum rdma_remove_reason why, struct uverbs_attr_bundle *attrs)

{
	return rdma_destroy_ah_user((struct ib_ah *)uobject->object,
				    RDMA_DESTROY_AH_SLEEPABLE,
				    &attrs->driver_udata);
}

So, look along that path and find out where it goes wrong?

Jason

#4 lore raw
Quoted 38 lines · Jason Gunthorpe
Quoted 2 lines · Bob Pearson

But ib_uverbs_destroy_ah does *not* call rdma_uverbs_destroy_ah() it just deletes the object.

ib_uverbs_destroy_ah
uobj_perform_destroy
__uobj_perform_destroy
__uobj_get_destroy

    uobj_destroy
     uverbs_destroy_uobject:

    	} else if (uobj->object) {
		ret = uobj->uapi_object->type_class->destroy_hw(uobj, reason,
								attrs);

Which calls

destroy_hw_idr_uobject
int ret = idr_type->destroy_object(uobj, why, attrs);

Which links to this:

DECLARE_UVERBS_NAMED_OBJECT(UVERBS_OBJECT_AH,

			    UVERBS_TYPE_ALLOC_IDR(uverbs_free_ah),
			    &UVERBS_METHOD(UVERBS_METHOD_AH_DESTROY));

And thus calls

static int uverbs_free_ah(struct ib_uobject *uobject,
enum rdma_remove_reason why, struct uverbs_attr_bundle *attrs)

{
	return rdma_destroy_ah_user((struct ib_ah *)uobject->object,
				    RDMA_DESTROY_AH_SLEEPABLE,
				    &attrs->driver_udata);
}
Quoted 4 lines

So, look along that path and find out where it goes wrong?

Jason

Thanks

I had more or less figured that out. I looked at other objects and saw a similar pattern. I think I've traced the problem back to myself.

Bob