Skip to contentSkip to conversation

RDMA/addr: create addr_wq with WQ_MEM_RECLAIM flag

PATCH New Potnuri Bharat Teja 1 participant 1 message Mar 13, 2019 4 messages not in the archive NVMe

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 5f5a1e4e90f3625cea57ffa79fc0e5bcb7efe09d.1548963371.git.swise@opengridcomputing.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw

PATCH 1/1 RDMA/addr: create addr_wq with WQ_MEM_RECLAIM flag

Quoted 27 lines · Steve Wise
Quoted 25 lines · Steve Wise
Quoted 13 lines · Jason Gunthorpe
Quoted 12 lines · Steve Wise

While running NVMe/oF wire unplug tests, we hit this warning in kernel/workqueue.c:check_flush_dependency():

WARN_ONCE(worker && ((worker->current_pwq->wq->flags &

		      (WQ_MEM_RECLAIM | __WQ_LEGACY)) == WQ_MEM_RECLAIM),
	  "workqueue: WQ_MEM_RECLAIM %s:%pf is flushing !WQ_MEM_RECLAIM %s:%pf",
	  worker->current_pwq->wq->name, worker->current_func,
	  target_wq->name, target_func);

Which I think means we're flushing a workq that doesn't have WQ_MEM_RECLAIM set, from workqueue context that does have it set.

Quoted 7 lines
Quoted 4 lines

Looking at rdma_addr_cancel() which is doing the flushing, it flushes the addr_wq which doesn't have MEM_RECLAIM set. Yet rdma_addr_cancel() is being called by the nvme host connection timeout/reconnect workqueue thread that does have WQ_MEM_RECLAIM set.

Since we haven't learned anything more, I think you should look to remove either the WQ_MEM_RECLAIM or the rdma_addr_cancel() from the nvme side.

The nvme code is just calling rdma_destroy_id(), which in turn calls rdma_addr_cancel(),? so I'll have to remove WQ_MEM_RECLAIM from the workqueue.

Quoted 8 lines
Quoted 1 line

I'll post this patch then.

What a mess.? If I remove RECLAIM for nvme_wq, then I'll regress this change, I think:

c669ccdc50c2 ("nvme: queue ns scanning and async request from nvme_wq")

Hi All,
I see this issue with some basic tests. Looks like the discussion ended here. Please suggest where this ideally needs to be fixed so that I can try fixing it.

Thanks,
Bharat