net: mana: concurrent HWC requests and dynamic queue depth

This series adds concurrent HWC request ownership and negotiated HWC queue depth to MANA. It starts from depth-one bootstrap queues, keeps setup private until the final channel is tested, and preserves the distinction between CPU lifetime and PF ownership of DMA mappings.

I am continuing Long Li's four-patch series. Original per-patch author identity, dates and sign-offs are retained.

Changes in v8 (v7 -> v8):
- Rebase onto net-next 05b5019b01fb9443b09ae4ed34af1135a80a84c8.
- Withdraw the table and drain RCU readers before freeing the HWC bundle, in patch 1. Add final PCI/CDX software-only disposal of retained state without reclaiming unacknowledged PF-owned DMA.
- Initialize caller state under the per-slot lock. Do not introduce a temporary channel-wide timeout latch in intermediate patch 2.
- Preserve recovery signaling when abandoned responses exhaust admission, while healthy saturation remains ordinary pressure.
- Include the candidate functional test in bounded bootstrap fallback and preserve a live retry budget without reviving cancellation.
- Share the effective TX budget with GDMA/RDMA request producers; bound RDMA batches without changing RX allocation/indexing strides.
- Preserve stopped-channel errors before admission and fail malformed doorbell initialization without depending on a later INIT_DONE.
- Correct cleanup comments and commit-message rationale. Review both Sashiko report streams rather than treating either as exhaustive.

This series remains independent of queue-set replacement. That work addresses individual Ethernet CQ publication/retirement. The shared CQ-table hunks must be reconciled while preserving both contracts, whichever series lands second. Neither series is silently stacked.
https://lore.kernel.org/all/[email protected]/

Known separate concerns include individual RDMA/datapath CQ lifetime, malformed-response RX reposting and wider DMA failure/device-lifecycle policy. This revision does not claim to fix them. Standalone net/stable backports of older-kernel fixes require separate extraction/validation.

Changes in v7 (v6 -> v7):
- Rebase onto net-next d8674294aefef02266c4d47ad10131f1bffbe534, following Simon Horman's request after the v6 CI application failure.
- Preserve the upstream GDMA core/PCI transport split and CDX support. Initialize the HWC lifetime lock on both transport probe paths and perform PCI recovery's timeout cancellation through a GDMA core helper which protects the published HWC pointer across the module boundary.
- Retain the four-patch lineage and the v6 ownership, request, timeout, publication, depth-validation and fallback policies.

Rebase request:
https://lore.kernel.org/all/[email protected]/

Changes in v6 (v5 -> v6):
- Keep the four-patch v5 structure while folding review-required HWC ownership, teardown, timeout, publication and fallback safety into their owning feature patches.
- Serialize complete ESTABLISH_HWC and DESTROY_HWC shared-memory transactions. Record possible PF ownership before MMIO submission and clear it only after DESTROY_HWC is acknowledged.
- Retain PF-visible queues after an indeterminate destroy. Fence the dedicated HWC EQ and withdraw its CQ from dispatch without freeing the mappings, then retry teardown before assigning fresh queues.
- Destroy the HWC EQ before CQ and completion state. Release-publish the CQ table and wait for IRQ-side RCU readers before freeing it, without adding per-completion refcount operations.
- Withdraw the caller's response buffer before return and quarantine a timed-out slot until its late response arrives or teardown cancels it.
- Return -EBUSY for healthy slot pressure and reserve -ETIMEDOUT for an admitted request which receives no response.
- Publish slots under the inflight lock, protect response state with the per-slot lock, and retain sender/response references until both sides finish.
- Synchronize hwc_timeout updates and publish the HWC only after queues, admission state, caller contexts and the direct EQ test are ready.
- Replace the goto-heavy request and channel-creation flows called out in review with focused admission, submission, wait, build, validation, rebuild and fallback helpers.
- Treat reported depth as a maximum: rebuild exactly for 2 through 128, retain untouched bootstrap queues for 1 or above 128, and require the rebuilt report to repeat the selected depth.
- Enforce directional request/response maxima against both the report and allocated buffers. Rebuild only for exact 4096-byte reports while permitting bootstrap operation for other nonzero maxima.
- Validate BAR page geometry and every stored doorbell. Invalidate the old HWC doorbell before re-establishment and suppress MMIO while the invalid sentinel is present, without adding full validation to each data-path doorbell ring.
- Remove the proposed capability bit because driver-version advertisement occurs after HWC establishment and cannot negotiate creation-time behavior.
- Preserve current net-next behavior which always uses the hardware-reported dest_vrq_id and dest_vrcq_id.

Changes in v5 (v4 -> v5):
- Shortened comments and documented the existing error paths in patches 1 and 2.
- Rejected apparent success from a response accepted before request submission.
- Guarded CQ unpublication when establishment failed before cq_table allocation.
- Required successful retry teardown before creating fresh bootstrap queues after an initial destroy failure.

Changes in v4 (return to net-next after the v3 split):
- Reworked the feature series into four standalone patches for handover, per-slot completion state, concurrency and dynamic depth.
- Added bounded admission, guarded sender accounting, timeout quarantine, queue-depth bounds, message-buffer sizing and rebuild validation.
- Reset routing IDs for each establish and rejected a missing doorbell.

Changes in v3 (v2 -> net v3, historical fixes-only posting):
- Split the combined series after maintainer feedback, posting fixes to net and deferring concurrency and dynamic depth.
- Added response-buffer withdrawal and stale-response handling.
- Removed pcie_flr() teardown fallback and retained resources after an unacknowledged destroy.

Changes in v2 (v1 -> v2):
- Bounds-checked the SGE pointer derived from inline OOB size.
- Protected active-sender accounting and drained senders during teardown and failure exits.
- Rebased on net-next e354f7d60f14.

v1:
- Combined CQ-table lifetime, queue sizing, RX validation, teardown, concurrent-request and dynamic-depth work in seven patches.

Validation:
Source and build qualification:
Per-intermediate GCC/Clang W=1 builds, patch style checks, focused actual-source C cases and paired full kernel/image/module qualification are complete. Sparse retains a baseline warning; Smatch retains baseline frontend diagnostics and is not counted as a clean pass.

Ordinary baseline/release coverage on VM5 and VM6 is complete. Each VM/fixture has 1,600 type-2 PD/MR and 320 type-3 PD/CQ/MR operations across the initial and three post-PCI rounds, native channels 8->4->8, native/upper MTUs 1500->1400->1500, three PCI remove/rescan cycles, bounded ping and three-second TCP/1Mbit UDP with positive receivers. Both VMs returned to their exact SAFE baselines. Owned fixtures/helpers and processes were removed; guest and central reservations were released.

Evidence is explicitly composed, not one uninterrupted four-trial run. The completed VM5 baseline uses retained exact runtime receipts and a corrected evaluation of iperf3's duplicate, identical UDP bitrate metadata. That original harness attempt remains PROFILE_FAILED in its history. VM5 release and both VM6 fixtures completed in a new three-trial run. Fixture identity and SAFE continuity are checked across the two scopes.

No measured simultaneous outstanding firmware-command count, negotiated depth proof, performance comparison, connected RDMA workload, CDX hardware, fault injection, sanitizer, DIM or queue-set qualification is claimed.

The final intermediate-history correction leaves the entire final source tree identical to the release build snapshot a1454f411cdc. Evidence retains the literal build commit and kernelrelease rather than relabeling an image with a later history-only commit ID.

v7:
https://lore.kernel.org/all/[email protected]/
v6:
https://lore.kernel.org/all/[email protected]/
v5:
https://lore.kernel.org/all/[email protected]/
v4:
https://lore.kernel.org/all/[email protected]/
v3:
https://lore.kernel.org/all/[email protected]/
v2:
https://lore.kernel.org/all/[email protected]/
v1:
https://lore.kernel.org/all/[email protected]/

Long Li (4):
net: mana: prepare HWC ownership for safe reinitialization
net: mana: give each HWC message slot its own completion state
net: mana: support concurrent HWC requests
net: mana: add dynamic HWC queue depth with reinit path

 drivers/infiniband/hw/mana/main.c             |   19 +-
 .../net/ethernet/microsoft/mana/gdma_cdx.c    |    5 +-
 .../net/ethernet/microsoft/mana/gdma_main.c   |  154 ++-
 .../net/ethernet/microsoft/mana/gdma_pci.c    |   15 +-
 .../net/ethernet/microsoft/mana/hw_channel.c  | 1019 ++++++++++++++---
 .../net/ethernet/microsoft/mana/shm_channel.c |   58 +-
 include/net/mana/gdma.h                       |   17 +
 include/net/mana/hw_channel.h                 |   69 +-
 include/net/mana/shm_channel.h                |   10 +-
 9 files changed, 1150 insertions(+), 216 deletions(-)

base-commit: 05b5019b01fb9443b09ae4ed34af1135a80a84c8

2.43.0

From: Long Li <[email protected]>

Dynamic HWC queue sizing tears down the bootstrap queues and establishes a second channel before publishing it. Prepare the existing HWC ownership and teardown paths so that sequence cannot free or reuse state still reachable by either the PF or an EQ handler.

The shared-memory aperture carries one request at a time. Serialize the complete ESTABLISH_HWC and DESTROY_HWC transactions, including possession polling and response validation. Record setup_active immediately before the MMIO submission and clear it only after DESTROY_HWC is acknowledged.

Treat an unacknowledged destroy as retained PF ownership. Fence the HWC EQ and withdraw its CQ from dispatch, but keep all queue memory and wrappers for a later teardown retry. A new channel creation retries that retained context before assigning fresh queues to the PF.

When a PCI or CDX transport permanently discards its gdma_context, no such retry is possible. After IRQ and service-work teardown, release the software-only HWC state through a common final-disposal helper. Leave the coherent DMA allocations unreclaimed rather than exposing freed memory to a PF whose teardown was not acknowledged.

Snapshot the bootstrap CQ count and CQ ID after INIT_DONE. Publish the fixed bound with the allocated table so later init events cannot grow the bound past the allocation. Pair annotated CQ-slot reads with annotated clears.

On acknowledged teardown, withdraw the CQ table and wait for all existing IRQ-side RCU readers before freeing the HWC queue bundle. This also covers a completion dispatched through a different EQ, without a per-event reference. Fence the dedicated EQ's remaining callbacks before releasing the CQ, completion buffer and work queues. Individual Ethernet and RDMA CQ lifetime handling remains separate.

---
 .../net/ethernet/microsoft/mana/gdma_cdx.c    |   4 +-
 .../net/ethernet/microsoft/mana/gdma_main.c   |  37 +++-
 .../net/ethernet/microsoft/mana/gdma_pci.c    |   4 +-
 .../net/ethernet/microsoft/mana/hw_channel.c  | 190 ++++++++++++++++--
 .../net/ethernet/microsoft/mana/shm_channel.c |  58 +++++-
 include/net/mana/gdma.h                       |   4 +
 include/net/mana/hw_channel.h                 |   5 +
 include/net/mana/shm_channel.h                |  10 +-
 8 files changed, 273 insertions(+), 39 deletions(-)

index 0d01e3d2f0d9..2e0dcb902d7e 100644
@@ -279,7 +279,7 @@ static int mana_cdx_gd_probe(struct cdx_device *cdx_dev)
 	iounmap(bar0_va);
 free_gc:
 	dev_set_drvdata(&cdx_dev->dev, NULL);
-	vfree(gc);
+	mana_gd_free_context(gc);
 clear_master:
 	cdx_clear_master(cdx_dev);
 	return err;

@@ -298,7 +298,7 @@ static int mana_cdx_gd_remove(struct cdx_device *cdx_dev)
 	iounmap(gc->bar0_va);
 
 	dev_set_drvdata(&cdx_dev->dev, NULL);
-	vfree(gc);
+	mana_gd_free_context(gc);
 	return 0;
 }
 

index ae0ed700b3b9..d11068f1dc05 100644
@@ -11,6 +11,7 @@
 #include <linux/irqdomain.h>
 #include <linux/export.h>
 #include <linux/uaccess.h>
+#include <linux/vmalloc.h>
 
 #include <net/mana/mana.h>
 #include <net/mana/hw_channel.h>

@@ -754,6 +755,7 @@ static void mana_gd_process_eqe(struct gdma_queue *eq)
 	union gdma_eqe_info eqe_info;
 	enum gdma_eqe_type type;
 	struct gdma_event event;
+	struct gdma_queue **cq_table;
 	struct gdma_queue *cq;
 	struct gdma_eqe *eqe;
 	u32 cq_id;

@@ -765,11 +767,16 @@ static void mana_gd_process_eqe(struct gdma_queue *eq)
 	switch (type) {
 	case GDMA_EQE_COMPLETION:
 		cq_id = eqe->details[0] & 0xFFFFFF;
-		if (WARN_ON_ONCE(cq_id >= gc->max_num_cqs))
+		/* The IRQ handler's RCU read-side section protects the table
+		 * until HWC teardown has fenced its EQ and waited for readers.
+		 */
+		cq_table = smp_load_acquire(&gc->cq_table);
+		if (!cq_table || cq_id >= READ_ONCE(gc->max_num_cqs))
 			break;
 
-		cq = gc->cq_table[cq_id];
-		if (WARN_ON_ONCE(!cq || cq->type != GDMA_CQ || cq->id != cq_id))
+		cq = READ_ONCE(cq_table[cq_id]);
+		if (!cq || WARN_ON_ONCE(cq->type != GDMA_CQ ||
+					cq->id != cq_id))
 			break;
 
 		if (cq->cq.callback)

@@ -993,23 +1000,32 @@ int mana_gd_test_eq(struct gdma_context *gc, struct gdma_queue *eq)
 	return err;
 }
 
-static void mana_gd_destroy_eq(struct gdma_context *gc, bool flush_evenets,
+static void mana_gd_destroy_eq(struct gdma_context *gc, bool flush_events,
 			       struct gdma_queue *queue)
 {
 	int err;
 
-	if (flush_evenets) {
+	if (queue->eq.msix_index == INVALID_PCI_MSIX_INDEX)
+		return;
+
+	if (flush_events) {
 		err = mana_gd_test_eq(gc, queue);
 		if (err && mana_need_log(gc, err))
 			dev_warn(gc->dev, "Failed to flush EQ: %d\n", err);
 	}
 
 	mana_gd_deregister_irq(queue);
+	queue->eq.msix_index = INVALID_PCI_MSIX_INDEX;
 
 	if (queue->eq.disable_needed)
 		mana_gd_disable_queue(queue);
 }
 
+void mana_gd_fence_eq(struct gdma_context *gc, struct gdma_queue *queue)
+{
+	mana_gd_destroy_eq(gc, false, queue);
+}
+
 static int mana_gd_create_eq(struct gdma_dev *gd,
 			     const struct gdma_queue_spec *spec,
 			     bool create_hwq, struct gdma_queue *queue)

@@ -1078,10 +1094,10 @@ static void mana_gd_destroy_cq(struct gdma_context *gc,
 	if (id >= gc->max_num_cqs)
 		return;
 
-	if (!gc->cq_table[id])
+	if (!READ_ONCE(gc->cq_table[id]))
 		return;
 
-	gc->cq_table[id] = NULL;
+	WRITE_ONCE(gc->cq_table[id], NULL);
 }
 
 int mana_gd_create_hwc_queue(struct gdma_dev *gd,

@@ -2008,6 +2024,13 @@ void mana_gd_cleanup(struct gdma_context *gc)
 }
 EXPORT_SYMBOL_NS(mana_gd_cleanup, "NET_MANA");
 
+void mana_gd_free_context(struct gdma_context *gc)
+{
+	mana_hwc_free_retained_channel(gc);
+	vfree(gc);
+}
+EXPORT_SYMBOL_NS(mana_gd_free_context, "NET_MANA");
+
 static int __init gdma_core_init(void)
 {
 	mana_debugfs_root = debugfs_create_dir("mana", NULL);

index ed8925296c8a..7935cb274e36 100644
@@ -725,7 +725,7 @@ static int mana_gd_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	pci_iounmap(pdev, bar0_va);
 free_gc:
 	pci_set_drvdata(pdev, NULL);
-	vfree(gc);
+	mana_gd_free_context(gc);
 release_region:
 	pci_release_regions(pdev);
 disable_dev:

@@ -778,7 +778,7 @@ static void mana_gd_remove(struct pci_dev *pdev)
 
 	pci_iounmap(pdev, gc->bar0_va);
 
-	vfree(gc);
+	mana_gd_free_context(gc);
 
 	pci_release_regions(pdev);
 	pci_disable_device(pdev);

index c610fc1067e0..1c182cd83387 100644
@@ -580,6 +580,9 @@ int mana_gd_create_mana_wq_cq(struct gdma_dev *gd,
 
 void mana_gd_destroy_queue(struct gdma_context *gc, struct gdma_queue *queue);
 
+/* Remove an EQ from interrupt dispatch without freeing its queue memory. */
+void mana_gd_fence_eq(struct gdma_context *gc, struct gdma_queue *queue);
+
 int mana_gd_poll_cq(struct gdma_queue *cq, struct gdma_comp *comp, int num_cqe);
 
 void mana_gd_ring_cq(struct gdma_queue *cq, u8 arm_bit);

@@ -1121,6 +1124,7 @@ int mana_gd_verify_vf_version(struct gdma_context *gc);
 int mana_gd_query_max_resources(struct gdma_context *gc);
 int mana_gd_setup(struct gdma_context *gc);
 void mana_gd_cleanup(struct gdma_context *gc);
+void mana_gd_free_context(struct gdma_context *gc);
 int mana_gd_detect_devices(struct gdma_context *gc);
 int mana_gd_init_registers(struct gdma_context *gc);
 irqreturn_t mana_gd_intr(int irq, void *arg);

index 1ab2d66c891e..1320bdfa9d2a 100644
@@ -185,6 +185,8 @@ struct hw_channel_context {
 	u16 hwc_init_q_depth_max;
 	u32 hwc_init_max_req_msg_size;
 	u32 hwc_init_max_resp_msg_size;
+	u32 hwc_init_max_num_cqs;
+	u32 hwc_init_cq_id;
 
 	struct completion hwc_init_eqe_comp;
 

@@ -199,11 +201,14 @@ struct hw_channel_context {
 	u32 dest_vrcq_id;
 	u32 hwc_timeout;
 
+	/* The PF may own the HWC queues while this is true. */
+	bool setup_active;
 	struct hwc_caller_ctx *caller_ctx;
 };
 
 int mana_hwc_create_channel(struct gdma_context *gc);
 void mana_hwc_destroy_channel(struct gdma_context *gc);
+void mana_hwc_free_retained_channel(struct gdma_context *gc);
 
 int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len,
 			  const void *req, u32 resp_len, void *resp);

index dbabcfb95daf..c8daa61fa759 100644
@@ -4,6 +4,8 @@
 #ifndef _SHM_CHANNEL_H
 #define _SHM_CHANNEL_H
 
+#include <linux/mutex.h>
+
 #define SMC_APERTURE_BITS 256
 #define SMC_BASIC_UNIT (sizeof(u32))
 #define SMC_APERTURE_DWORDS (SMC_APERTURE_BITS / (SMC_BASIC_UNIT * 8))

@@ -13,6 +15,9 @@
 struct shm_channel {
 	struct device *dev;
 	void __iomem *base;
+	/* Protects base and complete aperture request/response transactions. */
+	struct mutex transaction_lock;
+	bool transaction_lock_initialized;
 };
 
 void mana_smc_init(struct shm_channel *sc, struct device *dev,

@@ -20,8 +25,9 @@ void mana_smc_init(struct shm_channel *sc, struct device *dev,
 
 int mana_smc_setup_hwc(struct shm_channel *sc, bool reset_vf, u64 eq_addr,
 		       u64 cq_addr, u64 rq_addr, u64 sq_addr,
-		       u32 eq_msix_index);
+		       u32 eq_msix_index, bool *submitted);
 
-int mana_smc_teardown_hwc(struct shm_channel *sc, bool reset_vf);
+int mana_smc_teardown_hwc(struct shm_channel *sc, bool reset_vf,
+			  bool *setup_active);
 
 #endif /* _SHM_CHANNEL_H */

2.43.0

From: Long Li <[email protected]>

Concurrent requests require the response handler and the waiting sender to agree on the lifetime and result state of each message slot.

Protect the caller's output buffer, Linux error and device status with a per-slot lock. Publish the output buffer only while the sender is waiting and withdraw it before every return. A late response cannot copy through a pointer to caller storage whose lifetime has ended.

Reinitialize completion and result state under both the allocation and per-slot locks when acquiring the slot. A response handler can still hold the per-slot lock after the previous sender has returned its admission permit, so an unpublished bitmap bit alone is not sufficient protection. If the response records its result while the timed wait expires, return that result instead of reporting a false timeout.

Keep the baseline's post-timeout admission behavior at this step rather than introducing a temporary channel-wide latch that suppresses subsequent teardown commands. This does not solve late-response correlation after ID reuse. The next patch adds response ownership, per-slot quarantine and the associated bounded admission and teardown rules together.

Split response-status handling and request cleanup into helpers so the send path has structured returns rather than phase-jumping gotos.

---
 .../net/ethernet/microsoft/mana/hw_channel.c  | 124 +++++++++++++-----
 include/net/mana/hw_channel.h                 |   4 +-
 2 files changed, 94 insertions(+), 34 deletions(-)

index a1c1c6f32caf..03a3a0e9d3f8 100644
@@ -9,6 +9,7 @@
 static int mana_hwc_get_msg_index(struct hw_channel_context *hwc, u16 *msg_id)
 {
 	struct gdma_resource *r = &hwc->inflight_msg_res;
+	struct hwc_caller_ctx *ctx;
 	unsigned long flags;
 	u32 index;
 

@@ -19,8 +20,17 @@ static int mana_hwc_get_msg_index(struct hw_channel_context *hwc, u16 *msg_id)
 	index = find_first_zero_bit(hwc->inflight_msg_res.map,
 				    hwc->inflight_msg_res.size);
 
+	ctx = &hwc->caller_ctx[index];
+	spin_lock(&ctx->lock);
+	reinit_completion(&ctx->comp_event);
+	ctx->output_buf = NULL;
+	ctx->output_buflen = 0;
+	ctx->error = -EINPROGRESS;
+	ctx->status_code = 0;
+
 	bitmap_set(hwc->inflight_msg_res.map, index, 1);
 
+	spin_unlock(&ctx->lock);
 	spin_unlock_irqrestore(&r->lock, flags);
 
 	*msg_id = index;

@@ -90,14 +100,21 @@ static void mana_hwc_handle_resp(struct hw_channel_context *hwc, u32 resp_len,
 	}
 
 	ctx = hwc->caller_ctx + msg_id;
-	err = mana_hwc_verify_resp_msg(ctx, resp_msg, resp_len);
-	if (err)
-		goto out;
 
-	ctx->status_code = resp_msg->status;
+	spin_lock(&ctx->lock);
+	if (!ctx->output_buf) {
+		spin_unlock(&ctx->lock);
+		mana_hwc_post_rx_wqe(hwc->rxq, rx_req);
+		return;
+	}
 
-	memcpy(ctx->output_buf, resp_msg, resp_len);
-out:
+	err = mana_hwc_verify_resp_msg(ctx, resp_msg, resp_len);
+	if (!err) {
+		ctx->status_code = resp_msg->status;
+		memcpy(ctx->output_buf, resp_msg, resp_len);
+	}
+
+	ctx->output_buf = NULL;
 	ctx->error = err;
 
 	/* Must post rx wqe before complete(), otherwise the next rx may

@@ -106,6 +123,7 @@ static void mana_hwc_handle_resp(struct hw_channel_context *hwc, u32 resp_len,
 	mana_hwc_post_rx_wqe(hwc->rxq, rx_req);
 
 	complete(&ctx->comp_event);
+	spin_unlock(&ctx->lock);
 }
 
 static void mana_hwc_init_event_handler(void *ctx, struct gdma_queue *q_self,

@@ -694,8 +712,10 @@ static int mana_hwc_test_channel(struct hw_channel_context *hwc, u16 q_depth,
 	if (!ctx)
 		return -ENOMEM;
 
-	for (i = 0; i < q_depth; ++i)
+	for (i = 0; i < q_depth; ++i) {
 		init_completion(&ctx[i].comp_event);
+		spin_lock_init(&ctx[i].lock);
+	}
 
 	hwc->caller_ctx = ctx;
 

@@ -998,6 +1018,37 @@ void mana_hwc_free_retained_channel(struct gdma_context *gc)
 	gc->hwc.gdma_context = NULL;
 }
 
+static int mana_hwc_response_status(struct hw_channel_context *hwc,
+				    u32 command, int error, u32 status)
+{
+	if (error)
+		return error;
+
+	if (!status || status == GDMA_STATUS_MORE_ENTRIES)
+		return 0;
+
+	if (status == GDMA_STATUS_CMD_UNSUPPORTED)
+		return -EOPNOTSUPP;
+
+	if (command != MANA_QUERY_PHY_STAT)
+		dev_err(hwc->dev, "Command 0x%x failed with status: 0x%x\n",
+			command, status);
+
+	return -EPROTO;
+}
+
+static void mana_hwc_finish_request(struct hw_channel_context *hwc,
+				    struct hwc_caller_ctx *ctx, u16 msg_id)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&ctx->lock, flags);
+	ctx->output_buf = NULL;
+	spin_unlock_irqrestore(&ctx->lock, flags);
+
+	mana_hwc_put_msg_index(hwc, msg_id);
+}
+
 int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len,
 			  const void *req, u32 resp_len, void *resp)
 {

@@ -1005,26 +1056,32 @@ int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len,
 	struct hwc_wq *txq = hwc->txq;
 	struct gdma_req_hdr *req_msg;
 	struct hwc_caller_ctx *ctx;
+	unsigned long flags;
 	u32 dest_vrcq;
 	u32 dest_vrq;
 	u32 command;
+	u32 status;
 	u16 msg_id;
 	int err;
 
-	mana_hwc_get_msg_index(hwc, &msg_id);
+	err = mana_hwc_get_msg_index(hwc, &msg_id);
+	if (err)
+		return err;
 
 	tx_wr = &txq->msg_buf->reqs[msg_id];
+	ctx = hwc->caller_ctx + msg_id;
 
 	if (req_len > tx_wr->buf_len) {
 		dev_err(hwc->dev, "HWC: req msg size: %d > %d\n", req_len,
 			tx_wr->buf_len);
-		err = -EINVAL;
-		goto out;
+		mana_hwc_finish_request(hwc, ctx, msg_id);
+		return -EINVAL;
 	}
 
-	ctx = hwc->caller_ctx + msg_id;
+	spin_lock_irqsave(&ctx->lock, flags);
 	ctx->output_buf = resp;
 	ctx->output_buflen = resp_len;
+	spin_unlock_irqrestore(&ctx->lock, flags);
 
 	req_msg = (struct gdma_req_hdr *)tx_wr->buf_va;
 	if (req)

@@ -1046,11 +1103,23 @@ int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len,
 	err = mana_hwc_post_tx_wqe(txq, tx_wr, dest_vrq, dest_vrcq, false);
 	if (err) {
 		dev_err(hwc->dev, "HWC: Failed to post send WQE: %d\n", err);
-		goto out;
+		mana_hwc_finish_request(hwc, ctx, msg_id);
+		return err;
 	}
 
 	if (!wait_for_completion_timeout(&ctx->comp_event,
-					 (msecs_to_jiffies(hwc->hwc_timeout)))) {
+					 msecs_to_jiffies(hwc->hwc_timeout))) {
+		spin_lock_irqsave(&ctx->lock, flags);
+		ctx->output_buf = NULL;
+		err = ctx->error;
+		status = ctx->status_code;
+		spin_unlock_irqrestore(&ctx->lock, flags);
+
+		if (err != -EINPROGRESS) {
+			mana_hwc_finish_request(hwc, ctx, msg_id);
+			return mana_hwc_response_status(hwc, command, err, status);
+		}
+
 		if (hwc->hwc_timeout != 0)
 			dev_err(hwc->dev, "Command 0x%x timed out: %u ms\n",
 				command, hwc->hwc_timeout);

@@ -1059,27 +1128,16 @@ int mana_hwc_send_request(struct hw_channel_context *hwc, u32 req_len,
 		if (hwc->hwc_timeout > 1)
 			hwc->hwc_timeout = 1;
 
-		err = -ETIMEDOUT;
-		goto out;
+		mana_hwc_finish_request(hwc, ctx, msg_id);
+		return -ETIMEDOUT;
 	}
 
-	if (ctx->error) {
-		err = ctx->error;
-		goto out;
-	}
+	spin_lock_irqsave(&ctx->lock, flags);
+	ctx->output_buf = NULL;
+	err = ctx->error;
+	status = ctx->status_code;
+	spin_unlock_irqrestore(&ctx->lock, flags);
 
-	if (ctx->status_code && ctx->status_code != GDMA_STATUS_MORE_ENTRIES) {
-		if (ctx->status_code == GDMA_STATUS_CMD_UNSUPPORTED) {
-			err = -EOPNOTSUPP;
-			goto out;
-		}
-		if (command != MANA_QUERY_PHY_STAT)
-			dev_err(hwc->dev, "Command 0x%x failed with status: 0x%x\n",
-				command, ctx->status_code);
-		err = -EPROTO;
-		goto out;
-	}
-out:
-	mana_hwc_put_msg_index(hwc, msg_id);
-	return err;
+	mana_hwc_finish_request(hwc, ctx, msg_id);
+	return mana_hwc_response_status(hwc, command, err, status);
 }

index 1320bdfa9d2a..b732c7cc8d8f 100644
@@ -168,10 +168,12 @@ struct hwc_wq {
 
 struct hwc_caller_ctx {
 	struct completion comp_event;
+	/* Protects slot initialization, the output buffer and response state. */
+	spinlock_t lock;
 	void *output_buf;
 	u32 output_buflen;
 
-	u32 error; /* Linux error code */
+	int error; /* Linux error code */
 	u32 status_code;
 };
 

2.43.0

From: Long Li <[email protected]>

Give HWC message slots separate sender and response ownership. Initialize each slot under the inflight resource and per-slot locks, publish its bitmap bit last, and return its admission permit after both references are released. The per-slot lock also excludes an older duplicate callback from resetting or consuming the next caller's state.

Serialize SQ posting so concurrent senders cannot corrupt the producer state. A posted request which times out keeps its response reference and admission permit until its late response arrives or teardown cancels it. The bootstrap channel remains depth one here; the next patch negotiates and allocates larger queues.

Return -EBUSY for healthy admission pressure, but preserve -ETIMEDOUT when admission expires with outstanding abandoned responses. Otherwise a timeout in a non-statistics caller can strand the statistics worker in an endless busy/retry cycle instead of reaching its existing reset path. Retire the abandoned-response count with the response reference, exactly once, on a late response or teardown. Keep this state separate from device status.

Synchronize hwc_timeout updates with atomic access helpers. Teardown cancellation is terminal for a channel instance, firmware and query updates may replace only a live timeout, and fail-fast reduction may only lower a live value to one millisecond.

Keep setup private until the queues, admission state, caller contexts and direct EQ test are ready. Runtime teardown unpublishes the channel, rejects new admissions, wakes waiters and drains active senders before releasing or retaining PF-owned mappings. Apply the same unpublication and sender-drain rules to final software-only disposal of retained state.

Split admission, submission, waiting, response status and unwind into helpers so mana_hwc_send_request() uses structured returns instead of phase-jumping gotos.

---
 .../net/ethernet/microsoft/mana/gdma_cdx.c    |   1 +
 .../net/ethernet/microsoft/mana/gdma_main.c   |  91 ++-
 .../net/ethernet/microsoft/mana/gdma_pci.c    |  11 +-
 .../net/ethernet/microsoft/mana/hw_channel.c  | 594 +++++++++++++-----
 include/net/mana/gdma.h                       |  11 +
 include/net/mana/hw_channel.h                 |  46 +-
 6 files changed, 573 insertions(+), 181 deletions(-)

index 2e0dcb902d7e..fa10681fd793 100644
@@ -221,6 +221,7 @@ static int mana_cdx_gd_probe(struct cdx_device *cdx_dev)
 
 	mutex_init(&gc->eq_test_event_mutex);
 	mutex_init(&gc->gic_mutex);
+	spin_lock_init(&gc->hwc_lock);
 	dev_set_drvdata(&cdx_dev->dev, gc);
 	gc->bar0_pa = cdx_resource_start(cdx_dev, 0);
 	gc->bar0_size = cdx_resource_len(cdx_dev, 0);

index 7935cb274e36..636d9b0dd581 100644
@@ -14,7 +14,6 @@
 #include <linux/xarray.h>
 
 #include <net/mana/mana.h>
-#include <net/mana/hw_channel.h>
 
 struct mana_dev_recovery {
 	struct list_head list;

@@ -85,7 +84,6 @@ static void mana_serv_fpga(struct pci_dev *pdev)
 static void mana_serv_reset(struct pci_dev *pdev)
 {
 	struct gdma_context *gc = pci_get_drvdata(pdev);
-	struct hw_channel_context *hwc;
 	int ret;
 
 	if (!gc) {

@@ -95,15 +93,13 @@ static void mana_serv_reset(struct pci_dev *pdev)
 		return;
 	}
 
-	hwc = gc->hwc.driver_data;
-	if (!hwc) {
+	/* HWC is not responding in this case, so don't wait */
+	ret = mana_gd_cancel_hwc_timeout(gc);
+	if (ret) {
 		dev_err(&pdev->dev, "MANA service: no HWC\n");
 		goto out;
 	}
 
-	/* HWC is not responding in this case, so don't wait */
-	hwc->hwc_timeout = 0;
-
 	dev_info(&pdev->dev, "MANA reset cycle start\n");
 
 	mana_gd_suspend(pdev, PMSG_SUSPEND);

@@ -674,6 +670,7 @@ static int mana_gd_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	mutex_init(&gc->eq_test_event_mutex);
 	mutex_init(&gc->gic_mutex);
+	spin_lock_init(&gc->hwc_lock);
 	pci_set_drvdata(pdev, gc);
 	gc->bar0_pa = pci_resource_start(pdev, 0);
 	gc->bar0_size = pci_resource_len(pdev, 0);

index 1c182cd83387..67430faecc44 100644
@@ -528,6 +528,15 @@ struct gdma_context {
 	/* Hardware communication channel (HWC) */
 	struct gdma_dev		hwc;
 
+	/* Sender drain; the final wakeup runs under hwc_lock. */
+	wait_queue_head_t	hwc_drain_waitq;
+
+	/* Protects runtime HWC publication and active sender references.
+	 * Setup owns an unpublished HWC directly; timeout updates use atomic
+	 * access helpers and do not require this lock.
+	 */
+	spinlock_t		hwc_lock;
+
 	/* Azure network adapter */
 	struct gdma_dev		mana;
 

@@ -1165,6 +1174,8 @@ void mana_gd_free_memory(struct gdma_mem_info *gmi);
 int mana_gd_send_request(struct gdma_context *gc, u32 req_len, const void *req,
 			 u32 resp_len, void *resp);
 
+int mana_gd_cancel_hwc_timeout(struct gdma_context *gc);
+
 int mana_gd_destroy_dma_region(struct gdma_context *gc, u64 dma_region_handle);
 void mana_register_debugfs(void);
 void mana_unregister_debugfs(void);

2.43.0

From: Long Li <[email protected]>

The HWC starts with depth-one bootstrap queues. Treat the PF-reported depth as a maximum and rebuild the channel exactly for reports from 2 through
128. Keep the untouched bootstrap queues for depth one and reports above 128, and require a rebuilt channel to repeat the selected depth.

Treat the reported request and response sizes as directional maxima. Rebuild only when both reports equal the 4096-byte bootstrap allocation. Keep allocation strides separate from these limits, and share the effective host-to-PF request budget with the GDMA and RDMA request builders. Bound RDMA CREATE and ADD_PAGES batches by that budget, checking that the headers and at least one page address fit before subtracting their sizes.

Reject oversized messages before acquiring a slot or publishing caller storage. A retained, stopped channel still returns -ENODEV rather than reporting a size error against uninitialized negotiated limits.

Keep each candidate private through queue creation, establishment and the direct EQ test. Publish only the final tested channel. If a rebuild fails, including its functional test, tear it down and restore fresh, tested bootstrap queues. Stop if teardown is not acknowledged; never reuse PF-owned mappings. Restore the previous live timeout budget before the fallback so a failed test's fail-fast reduction does not poison the retry, without reviving terminal cancellation.

Invalidate the previous HWC doorbell before every ESTABLISH_HWC. Validate the BAR0 page geometry and reported index before accepting a new doorbell, and suppress MMIO while it is invalid. This also hardens initial setup, which previously accepted an unchecked device-supplied index. A malformed doorbell completes initialization with -EPROTO without relying on later INIT_DONE events that might require re-arming through that doorbell.

Use the initial PF report to select the queue geometry. Driver-version advertisement occurs after HWC establishment and cannot negotiate behavior needed while the channel is being created.

---
 drivers/infiniband/hw/mana/main.c             |  19 +-
 .../net/ethernet/microsoft/mana/gdma_main.c   |  26 +-
 .../net/ethernet/microsoft/mana/hw_channel.c  | 313 +++++++++++++++---
 include/net/mana/gdma.h                       |   2 +
 include/net/mana/hw_channel.h                 |  18 +-
 5 files changed, 313 insertions(+), 65 deletions(-)

index e4414d208d4a..ce840cee42ca 100644
@@ -406,28 +406,38 @@ static int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem
 	unsigned int create_req_msg_size;
 	struct hw_channel_context *hwc;
 	struct ib_block_iter biter;
-	size_t max_pgs_add_cmd = 0;
+	size_t max_pgs_add_cmd;
 	size_t max_pgs_create_cmd;
 	struct gdma_context *gc;
 	size_t num_pages_total;
 	unsigned int tail = 0;
+	u32 req_limit;
 	u64 *page_addr_list;
 	void *request_buf;
 	int err = 0;
 
 	gc = mdev_to_gc(dev);
 	hwc = gc->hwc.driver_data;
+	req_limit = mana_hwc_get_max_req_size(hwc);
+	if (req_limit < struct_size(create_req, page_addr_list, 1) ||
+	    req_limit < struct_size(add_req, page_addr_list, 1)) {
+		ibdev_dbg(&dev->ib_dev,
+			  "HWC request budget %u is too small for DMA region commands\n",
+			  req_limit);
+		return -EMSGSIZE;
+	}
 
 	num_pages_total = ib_umem_num_dma_blocks(umem, page_sz);
 
 	max_pgs_create_cmd =
-		(hwc->max_req_msg_size - sizeof(*create_req)) / sizeof(u64);
+		(req_limit - sizeof(*create_req)) / sizeof(u64);
+	max_pgs_add_cmd = (req_limit - sizeof(*add_req)) / sizeof(u64);
 	num_pages_to_handle =
 		min_t(size_t, num_pages_total, max_pgs_create_cmd);
 	create_req_msg_size =
 		struct_size(create_req, page_addr_list, num_pages_to_handle);
 
-	request_buf = kzalloc(hwc->max_req_msg_size, GFP_KERNEL);
+	request_buf = kzalloc(req_limit, GFP_KERNEL);
 	if (!request_buf)
 		return -ENOMEM;
 

@@ -470,9 +480,6 @@ static int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem
 			if (err)
 				goto out;
 
-			max_pgs_add_cmd = (hwc->max_req_msg_size -
-				sizeof(*add_req)) / sizeof(u64);
-
 			add_req = request_buf;
 			add_req->dma_region_handle = *gdma_region;
 			add_req->reserved3 = 0;

index 429cf2b8e65c..8b08f28bfdb8 100644
@@ -589,12 +589,26 @@ static int mana_gd_disable_queue(struct gdma_queue *queue)
 #define DOORBELL_OFFSET_EQ	0xFF8
 #define DOORBELL_OFFSET_DIM	0x820
 
+bool mana_gd_is_valid_doorbell(struct gdma_context *gc, u32 db_id)
+{
+	if (!gc->db_page_size || gc->db_page_off >= gc->bar0_size)
+		return false;
+
+	return gc->db_page_off +
+	       gc->db_page_size * ((u64)db_id + 1) <= gc->bar0_size;
+}
+
 static void mana_gd_ring_doorbell(struct gdma_context *gc, u32 db_index,
 				  enum gdma_queue_type q_type, u32 qid,
 				  u32 tail_ptr, u8 num_req)
 {
-	void __iomem *addr = gc->db_page_base + gc->db_page_size * db_index;
 	union gdma_doorbell_entry e = {};
+	void __iomem *addr;
+
+	if (unlikely(db_index == INVALID_DOORBELL))
+		return;
+
+	addr = gc->db_page_base + gc->db_page_size * db_index;
 
 	switch (q_type) {
 	case GDMA_EQ:

@@ -1254,7 +1268,7 @@ static int mana_gd_create_dma_region(struct gdma_dev *gd,
 	/* The caller must keep the HWC alive throughout queue creation. */
 	hwc = gc->hwc.driver_data;
 	req_msg_size = struct_size(req, page_addr_list, num_page);
-	if (req_msg_size > hwc->max_req_msg_size)
+	if (req_msg_size > mana_hwc_get_max_req_size(hwc))
 		return -EINVAL;
 
 	req = kzalloc(req_msg_size, GFP_KERNEL);

@@ -1534,13 +1548,7 @@ int mana_gd_register_device(struct gdma_dev *gd)
 		return err ? err : -EPROTO;
 	}
 
-	/* Validate that doorbell page for db_id is within the BAR0 region.
-	 * In mana_gd_ring_doorbell(), the address is calculated as:
-	 *   addr = db_page_base + db_page_size * db_id
-	 *        = (bar0_va + db_page_off) + (db_page_size * db_id)
-	 * So we need: db_page_off + db_page_size * (db_id + 1) <= bar0_size
-	 */
-	if (gc->db_page_off + gc->db_page_size * ((u64)resp.db_id + 1) > gc->bar0_size) {
+	if (!mana_gd_is_valid_doorbell(gc, resp.db_id)) {
 		dev_err(gc->dev, "Doorbell ID %u out of range\n", resp.db_id);
 		return -EPROTO;
 	}

index 67430faecc44..a2e5ad3543ae 100644
@@ -594,6 +594,8 @@ void mana_gd_fence_eq(struct gdma_context *gc, struct gdma_queue *queue);
 
 int mana_gd_poll_cq(struct gdma_queue *cq, struct gdma_comp *comp, int num_cqe);
 
+bool mana_gd_is_valid_doorbell(struct gdma_context *gc, u32 db_id);
+
 void mana_gd_ring_cq(struct gdma_queue *cq, u8 arm_bit);
 
 ssize_t mana_gd_read_ring(struct gdma_queue *q, char __user *buf,

index 9d6a0a535e7a..73315aca2e00 100644
@@ -5,6 +5,7 @@
 #define _HW_CHANNEL_H
 
 #include <linux/atomic.h>
+#include <linux/minmax.h>
 
 #define DEFAULT_LOG2_THROTTLING_FOR_ERROR_EQ  4
 

@@ -13,6 +14,9 @@
 
 #define HW_CHANNEL_VF_BOOTSTRAP_QUEUE_DEPTH 1
 
+/* Largest supported rebuild depth; larger reports retain bootstrap queues. */
+#define HW_CHANNEL_MAX_QUEUE_DEPTH 128
+
 #define HWC_INIT_DATA_CQID		1
 #define HWC_INIT_DATA_RQID		2
 #define HWC_INIT_DATA_SQID		3

@@ -200,12 +204,18 @@ struct hw_channel_context {
 
 	u16 num_inflight_msg;
 	u32 max_req_msg_size;
+	u32 max_resp_msg_size;
+	/* Immutable reported maxima for the final RQ and SQ directions. */
+	u32 rx_msg_size_limit;
+	u32 tx_msg_size_limit;
 
-	u16 hwc_init_q_depth_max;
+	u32 hwc_init_q_depth_max;
 	u32 hwc_init_max_req_msg_size;
 	u32 hwc_init_max_resp_msg_size;
 	u32 hwc_init_max_num_cqs;
 	u32 hwc_init_cq_id;
+	bool hwc_init_doorbell;
+	int hwc_init_error;
 
 	struct completion hwc_init_eqe_comp;
 

@@ -242,6 +252,12 @@ struct hw_channel_context {
 	struct hwc_caller_ctx *caller_ctx;
 };
 
+/* Host-to-PF requests are bounded by the negotiated TX limit and SQ stride. */
+static inline u32 mana_hwc_get_max_req_size(const struct hw_channel_context *hwc)
+{
+	return min(hwc->tx_msg_size_limit, hwc->max_resp_msg_size);
+}
+
 int mana_hwc_create_channel(struct gdma_context *gc);
 void mana_hwc_destroy_channel(struct gdma_context *gc);
 void mana_hwc_free_retained_channel(struct gdma_context *gc);

2.43.0

Thanks for the report. This is a pre-existing individual-CQ lifetime issue, rather than a regression introduced by this HWC series.

mana_gd_destroy_cq() clears the slot, but mana_gd_destroy_queue() can free the CQ while an EQ reader still holds its pointer. The later EQ-side RCU grace period does not protect the earlier CQ free.

I checked unmodified net-next at 98c2ad4f5d80 on October 17. Its MANA Ethernet, RDMA and header subtrees are identical to the series base, 05b5019b01fb. The Ethernet CQ reader/teardown ordering is also already present at upstream commit 02fed6d92bad ("net: mana: add msix index sharing between EQs"), from December 2023, well before this series.

The HWC-specific teardown addressed by this patch is different: after acknowledged HWC teardown, it unpublishes the CQ table and waits for existing IRQ-side RCU readers before freeing the HWC queues. Individual Ethernet and RDMA CQ lifetime handling remains separate, as stated in the patch description.

I agree that the reported issue should be addressed in a separate fix to the existing per-CQ teardown paths, independently of this HWC series.

Thanks,
Wei

Please respond to the netdev-bot AI review in a timely manner. (You can ignore the global review bot, it's not even CCed to netdev so maintainers don't actually see it :/)

Can't tell at a glance whether the Medium / High severity issues are indeed valid or nit picks, but since the merge window has already started let's revisit this after. Please repost in 2 weeks.

pw-bot: defer

Thanks for the review.

The individual Ethernet and RDMA CQ publication/lifetime issues are pre-existing. We will address separately.

Regards,
Wei

Thanks for the review.

This is pre-existing in the baseline driver; we will address separately. As noted in the review, patch 4 already bounds the requested response size by the RX slot capacity before admission in the completed series.

This is pre-existing in the baseline driver; we will address separately. As noted in the review, patch 3 already keeps a timed-out slot reserved until its pending response is consumed or teardown retires it. That protection remains in v9.

Regards,
Wei

Thanks for the review.

I will respin the series as v9 with the following changes.

Agreed. The independent admission deadline introduces a new failure before the command is submitted, including for destructive callers.

This is addressed in the prepared v9 patch 3. Admission uses a bitmap and waitqueue instead of down_timeout(). While the channel is healthy, a caller waits for a free slot rather than returning -EBUSY because of contention. This removes the congestion-only failure that could skip a teardown command. The response wait for a posted command remains bounded.

The remaining ordering, cancellation and logging cases are also addressed in v9 patch 3. There is no separate admission timer to expire before the holder records abandonment. Slot release, recorded response abandonment, cancellation and channel stop wake admission waiters. When no slot is available, abandonment or cancellation returns -ETIMEDOUT, preserving the existing recovery errno; a stopped channel returns -ENODEV. Healthy contention does not trigger recovery.

Available slots remain usable for best-effort requests after cancellation. mana_need_log() suppresses expected -ETIMEDOUT/-ENODEV after cancellation or channel unpublication, while keeping live-channel and other errors visible.

The missing RX repost on an out-of-range message ID is pre-existing. We will address separately.

Regards,
Wei

pw-bot: cr

Thanks for the review.

The PF specifies the HWC queue depth and corresponding ring sizes. It reports the required depth via HWC_INIT_DATA_QUEUE_DEPTH, and the VF rebuilds the rings to implement that PF requirement.

Regards,
Wei

Hi Jakub,

I just replied to netdev-bot about its review findings. Only couple issues in patch 3 are valid. Rest of them are either pre-existing or nit picks. I will send a v9 shortly.

Thanks,
Wei

PS: Thanks for letting me know that I can ignore the global review bot.