Skip to contentSkip to conversation

Extending the ena driver to support new features and enhance performance

PATCH In review [email protected] 6 participants 32 messages Jun 3, 2019 – Jun 7, 2019 1 message not in the archive

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v2 cover Extending the ena driver to support new features and enhance performance

From: Sameeh Jubran <[email protected]>

This patchset introduces the following:

* add support for changing the inline header size (max_header_size) for applications with overlay and nested headers
* enable automatic fallback to polling mode for admin queue when interrupt is not available or missed
* add good checksum counter for Rx ethtool statistics
* update ena.txt
* some minor code clean-up
* some performance enhancements with doorbell calculations

Differences from V1:

* net: ena: add handling of llq max tx burst size (1/11):
* fixed christmas tree issue

* net: ena: ethtool: add extra properties retrieval via get_priv_flags (2/11):
* replaced snprintf with strlcpy
* dropped confusing error message
* added more details to the commit message

Arthur Kiyanovski (1):
net: ena: ethtool: add extra properties retrieval via get_priv_flags

Sameeh Jubran (10):
net: ena: add handling of llq max tx burst size
net: ena: replace free_tx/rx_ids union with single free_ids field in
ena_ring
net: ena: arrange ena_probe() function variables in reverse christmas
tree
net: ena: add newline at the end of pr_err prints
net: ena: documentation: update ena.txt
net: ena: allow automatic fallback to polling mode
net: ena: add support for changing max_header_size in LLQ mode
net: ena: optimise calculations for CQ doorbell
net: ena: add good checksum counter
net: ena: use dev_info_once instead of static variable

 .../networking/device_drivers/amazon/ena.txt  |   5 +-
 .../net/ethernet/amazon/ena/ena_admin_defs.h  |  21 +++
 drivers/net/ethernet/amazon/ena/ena_com.c     | 123 ++++++++++++++----
 drivers/net/ethernet/amazon/ena/ena_com.h     |  48 +++++++
 drivers/net/ethernet/amazon/ena/ena_eth_com.c |  28 ++--
 drivers/net/ethernet/amazon/ena/ena_eth_com.h |  73 +++++++++--
 drivers/net/ethernet/amazon/ena/ena_ethtool.c |  78 +++++++++--
 drivers/net/ethernet/amazon/ena/ena_netdev.c  |  86 +++++++-----
 drivers/net/ethernet/amazon/ena/ena_netdev.h  |  16 +--
 9 files changed, 376 insertions(+), 102 deletions(-)
Signature · 1 line
2.17.1
#1 lore raw

PATCH v2 1/11 net: ena: add handling of llq max tx burst size

From: Sameeh Jubran <[email protected]>

There is a maximum TX burst size that the ENA device can handle. It is exposed by the device to the driver and the driver needs to comply with it to avoid bugs.

In this commit we:
1. Add ena_com_is_doorbell_needed(), which calculates the number of llq entries that will be used to hold a packet, and will return true if they exceed the number of allowed entries in a burst. If the function returns true, a doorbell needs to be invoked to send this packet in the next burst.

2. Follow the available entries in the current burst:
- Every doorbell a new burst begins
- With each write of an llq entry, the available entries in the current burst are decreased by 1.

---
 .../net/ethernet/amazon/ena/ena_admin_defs.h  |  5 ++
 drivers/net/ethernet/amazon/ena/ena_com.c     |  7 +++
 drivers/net/ethernet/amazon/ena/ena_com.h     |  2 +
 drivers/net/ethernet/amazon/ena/ena_eth_com.c | 28 ++++------
 drivers/net/ethernet/amazon/ena/ena_eth_com.h | 53 +++++++++++++++++++
 drivers/net/ethernet/amazon/ena/ena_netdev.c  |  7 +++
 6 files changed, 85 insertions(+), 17 deletions(-)
drivers/net/ethernet/amazon/ena/ena_admin_defs.h +5 -0
index 9f80b73f9..82cff1e12 100644
@@ -524,6 +524,11 @@ struct ena_admin_feature_llq_desc {
 
 	/* the stride control the driver selected to use */
 	u16 descriptors_stride_ctrl_enabled;
+
+	/* Maximum size in bytes taken by llq entries in a single tx burst.
+	 * Set to 0 when there is no such limit.
+	 */
+	u32 max_tx_burst_size;
 };
 
 struct ena_admin_queue_feature_desc {
drivers/net/ethernet/amazon/ena/ena_com.c +7 -0
index 7f8266b19..bd0d785b2 100644
@@ -396,6 +396,10 @@ static int ena_com_init_io_sq(struct ena_com_dev *ena_dev,
 		       0x0, io_sq->llq_info.desc_list_entry_size);
 		io_sq->llq_buf_ctrl.descs_left_in_line =
 			io_sq->llq_info.descs_num_before_header;
+
+		if (io_sq->llq_info.max_entries_in_tx_burst > 0)
+			io_sq->entries_in_tx_burst_left =
+				io_sq->llq_info.max_entries_in_tx_burst;
 	}
 
 	io_sq->tail = 0;
@@ -727,6 +731,9 @@ static int ena_com_config_llq_info(struct ena_com_dev *ena_dev,
 		       supported_feat, llq_info->descs_num_before_header);
 	}
 
+	llq_info->max_entries_in_tx_burst =
+		(u16)(llq_features->max_tx_burst_size /	llq_default_cfg->llq_ring_entry_size_value);
+
 	rc = ena_com_set_llq(ena_dev);
 	if (rc)
 		pr_err("Cannot set LLQ configuration: %d\n", rc);
drivers/net/ethernet/amazon/ena/ena_com.h +2 -0
index 078d6f2b4..f0cb043af 100644
@@ -159,6 +159,7 @@ struct ena_com_llq_info {
 	u16 desc_list_entry_size;
 	u16 descs_num_before_header;
 	u16 descs_per_entry;
+	u16 max_entries_in_tx_burst;
 };
 
 struct ena_com_io_cq {
@@ -238,6 +239,7 @@ struct ena_com_io_sq {
 	u8 phase;
 	u8 desc_entry_size;
 	u8 dma_addr_bits;
+	u16 entries_in_tx_burst_left;
 } ____cacheline_aligned;
 
 struct ena_com_admin_cq {
drivers/net/ethernet/amazon/ena/ena_eth_com.c +11 -17
index f6c2d3855..cad2b5728 100644
@@ -82,6 +82,17 @@ static inline int ena_com_write_bounce_buffer_to_dev(struct ena_com_io_sq *io_sq
 	dst_tail_mask = io_sq->tail & (io_sq->q_depth - 1);
 	dst_offset = dst_tail_mask * llq_info->desc_list_entry_size;
 
+	if (is_llq_max_tx_burst_exists(io_sq)) {
+		if (unlikely(!io_sq->entries_in_tx_burst_left)) {
+			pr_err("Error: trying to send more packets than tx burst allows\n");
+			return -ENOSPC;
+		}
+
+		io_sq->entries_in_tx_burst_left--;
+		pr_debug("decreasing entries_in_tx_burst_left of queue %d to %d\n",
+			 io_sq->qid, io_sq->entries_in_tx_burst_left);
+	}
+
 	/* Make sure everything was written into the bounce buffer before
 	 * writing the bounce buffer to the device
 	 */
@@ -274,23 +285,6 @@ static inline u16 ena_com_cdesc_rx_pkt_get(struct ena_com_io_cq *io_cq,
 	return count;
 }
 
-static inline bool ena_com_meta_desc_changed(struct ena_com_io_sq *io_sq,
-					     struct ena_com_tx_ctx *ena_tx_ctx)
-{
-	int rc;
-
-	if (ena_tx_ctx->meta_valid) {
-		rc = memcmp(&io_sq->cached_tx_meta,
-			    &ena_tx_ctx->ena_meta,
-			    sizeof(struct ena_com_tx_meta));
-
-		if (unlikely(rc != 0))
-			return true;
-	}
-
-	return false;
-}
-
 static inline int ena_com_create_and_store_tx_meta_desc(struct ena_com_io_sq *io_sq,
 							struct ena_com_tx_ctx *ena_tx_ctx)
 {
drivers/net/ethernet/amazon/ena/ena_eth_com.h +53 -0

67 lines not on this page · raw message

drivers/net/ethernet/amazon/ena/ena_netdev.c +7 -0
index 9c8364292..d2b82f917 100644
@@ -2172,6 +2172,13 @@ static netdev_tx_t ena_start_xmit(struct sk_buff *skb, struct net_device *dev)
 	/* set flags and meta data */
 	ena_tx_csum(&ena_tx_ctx, skb);
 
+	if (unlikely(ena_com_is_doorbell_needed(tx_ring->ena_com_io_sq, &ena_tx_ctx))) {
+		netif_dbg(adapter, tx_queued, dev,
+			  "llq tx max burst size of queue %d achieved, writing doorbell to send burst\n",
+			  qid);
+		ena_com_write_sq_doorbell(tx_ring->ena_com_io_sq);
+	}
+
 	/* prepare the packet's descriptors to dma engine */
 	rc = ena_com_prepare_tx(tx_ring->ena_com_io_sq, &ena_tx_ctx,
 				&nb_hw_desc);
Signature · 1 line
2.17.1
#4 lore raw

PATCH v2 4/11 net: ena: arrange ena_probe() function variables in reverse christmas tree

In reply to [email protected]

From: Sameeh Jubran <[email protected]>

Reverse christmas tree arrangement is when strings are written from longer to shorter with each line. Most of our functions are abiding this arrangement but this function does not.

In this commit we arrange the variables of ena_probe() in reverse christmas tree.

---
 drivers/net/ethernet/amazon/ena/ena_netdev.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/net/ethernet/amazon/ena/ena_netdev.c +6 -6
index b80b5eddc..399bd5453 100644
@@ -3281,17 +3281,17 @@ static int ena_calc_queue_size(struct pci_dev *pdev,
 static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 {
 	struct ena_com_dev_get_features_ctx get_feat_ctx;
-	static int version_printed;
-	struct net_device *netdev;
-	struct ena_adapter *adapter;
 	struct ena_llq_configurations llq_config;
 	struct ena_com_dev *ena_dev = NULL;
-	char *queue_type_str;
-	static int adapters_found;
+	struct ena_adapter *adapter;
+	static int version_printed;
 	int io_queue_num, bars, rc;
-	int queue_size;
+	struct net_device *netdev;
+	static int adapters_found;
+	char *queue_type_str;
 	u16 tx_sgl_size = 0;
 	u16 rx_sgl_size = 0;
+	int queue_size;
 	bool wd_state;
 
 	dev_dbg(&pdev->dev, "%s\n", __func__);
Signature · 1 line
2.17.1
#5 lore raw

PATCH v2 5/11 net: ena: add newline at the end of pr_err prints

In reply to [email protected]

From: Sameeh Jubran <[email protected]>

Some pr_err prints lacked '\n' in the end. Added where missing.

---
 drivers/net/ethernet/amazon/ena/ena_com.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
drivers/net/ethernet/amazon/ena/ena_com.c +8 -8
index 935e8fa8d..139b31549 100644
@@ -115,7 +115,7 @@ static int ena_com_admin_init_sq(struct ena_com_admin_queue *queue)
 					 GFP_KERNEL);
 
 	if (!sq->entries) {
-		pr_err("memory allocation failed");
+		pr_err("memory allocation failed\n");
 		return -ENOMEM;
 	}
 
@@ -137,7 +137,7 @@ static int ena_com_admin_init_cq(struct ena_com_admin_queue *queue)
 					 GFP_KERNEL);
 
 	if (!cq->entries) {
-		pr_err("memory allocation failed");
+		pr_err("memory allocation failed\n");
 		return -ENOMEM;
 	}
 
@@ -160,7 +160,7 @@ static int ena_com_admin_init_aenq(struct ena_com_dev *dev,
 					   GFP_KERNEL);
 
 	if (!aenq->entries) {
-		pr_err("memory allocation failed");
+		pr_err("memory allocation failed\n");
 		return -ENOMEM;
 	}
 
@@ -285,7 +285,7 @@ static inline int ena_com_init_comp_ctxt(struct ena_com_admin_queue *queue)
 
 	queue->comp_ctx = devm_kzalloc(queue->q_dmadev, size, GFP_KERNEL);
 	if (unlikely(!queue->comp_ctx)) {
-		pr_err("memory allocation failed");
+		pr_err("memory allocation failed\n");
 		return -ENOMEM;
 	}
 
@@ -356,7 +356,7 @@ static int ena_com_init_io_sq(struct ena_com_dev *ena_dev,
 		}
 
 		if (!io_sq->desc_addr.virt_addr) {
-			pr_err("memory allocation failed");
+			pr_err("memory allocation failed\n");
 			return -ENOMEM;
 		}
 	}
@@ -382,7 +382,7 @@ static int ena_com_init_io_sq(struct ena_com_dev *ena_dev,
 				devm_kzalloc(ena_dev->dmadev, size, GFP_KERNEL);
 
 		if (!io_sq->bounce_buf_ctrl.base_buffer) {
-			pr_err("bounce buffer memory allocation failed");
+			pr_err("bounce buffer memory allocation failed\n");
 			return -ENOMEM;
 		}
 
@@ -440,7 +440,7 @@ static int ena_com_init_io_cq(struct ena_com_dev *ena_dev,
 	}
 
 	if (!io_cq->cdesc_addr.virt_addr) {
-		pr_err("memory allocation failed");
+		pr_err("memory allocation failed\n");
 		return -ENOMEM;
 	}
 
@@ -829,7 +829,7 @@ static u32 ena_com_reg_bar_read32(struct ena_com_dev *ena_dev, u16 offset)
 	}
 
 	if (read_resp->reg_off != offset) {
-		pr_err("Read failure: wrong offset provided");
+		pr_err("Read failure: wrong offset provided\n");
 		ret = ENA_MMIO_READ_TIMEOUT;
 	} else {
 		ret = read_resp->reg_val;
Signature · 1 line
2.17.1
#6 lore raw

PATCH v2 6/11 net: ena: documentation: update ena.txt

In reply to [email protected]

From: Sameeh Jubran <[email protected]>

Small cosmetic changes to ena.txt

---
 Documentation/networking/device_drivers/amazon/ena.txt | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
Documentation/networking/device_drivers/amazon/ena.txt +4 -1
index 2b4b6f57e..1bb55c7b6 100644
@@ -73,7 +73,7 @@ operation.
 AQ is used for submitting management commands, and the
 results/responses are reported asynchronously through ACQ.
 
-ENA introduces a very small set of management commands with room for
+ENA introduces a small set of management commands with room for
 vendor-specific extensions. Most of the management operations are
 framed in a generic Get/Set feature command.
 
@@ -202,11 +202,14 @@ delay value to each level.
 The user can enable/disable adaptive moderation, modify the interrupt
 delay table and restore its default values through sysfs.
 
+RX copybreak:
+=============
 The rx_copybreak is initialized by default to ENA_DEFAULT_RX_COPYBREAK
 and can be configured by the ETHTOOL_STUNABLE command of the
 SIOCETHTOOL ioctl.
 
 SKB:
+====
 The driver-allocated SKB for frames received from Rx handling using
 NAPI context. The allocation method depends on the size of the packet.
 If the frame length is larger than rx_copybreak, napi_get_frags()
Signature · 1 line
2.17.1
#8 lore raw

PATCH v2 8/11 net: ena: add support for changing max_header_size in LLQ mode

In reply to [email protected]

From: Sameeh Jubran <[email protected]>

Up until now the driver always used a single setting for the sizes of the different parts of the llq entry - 128 for entry size, 2 for descriptors before header and 96 for maximum header size.

The current code makes sure that the parts of the llq entry are compatible with each other and with the initial llq entry size given by the device.

This commit changes this code to support any llq entry size

---
 drivers/net/ethernet/amazon/ena/ena_com.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
drivers/net/ethernet/amazon/ena/ena_com.c +4 -6
index 5e2abdda7..dbc12e383 100644
@@ -2992,8 +2992,8 @@ int ena_com_config_dev_mode(struct ena_com_dev *ena_dev,
 			    struct ena_admin_feature_llq_desc *llq_features,
 			    struct ena_llq_configurations *llq_default_cfg)
 {
+	struct ena_com_llq_info *llq_info = &ena_dev->llq_info;
 	int rc;
-	int size;
 
 	if (!llq_features->max_llq_num) {
 		ena_dev->tx_mem_queue_type = ENA_ADMIN_PLACEMENT_POLICY_HOST;
@@ -3004,12 +3004,10 @@ int ena_com_config_dev_mode(struct ena_com_dev *ena_dev,
 	if (rc)
 		return rc;
 
-	/* Validate the descriptor is not too big */
-	size = ena_dev->tx_max_header_size;
-	size += ena_dev->llq_info.descs_num_before_header *
-		sizeof(struct ena_eth_io_tx_desc);
+	ena_dev->tx_max_header_size = llq_info->desc_list_entry_size -
+		(llq_info->descs_num_before_header * sizeof(struct ena_eth_io_tx_desc));
 
-	if (unlikely(ena_dev->llq_info.desc_list_entry_size < size)) {
+	if (unlikely(ena_dev->tx_max_header_size == 0)) {
 		pr_err("the size of the LLQ entry is smaller than needed\n");
 		return -EINVAL;
 	}
Signature · 1 line
2.17.1
#7 lore raw

PATCH v2 7/11 net: ena: allow automatic fallback to polling mode

In reply to [email protected]

From: Sameeh Jubran <[email protected]>

Enable fallback to polling mode for Admin queue when identified a command response arrival without an accompanying MSI-X interrupt

---
 drivers/net/ethernet/amazon/ena/ena_com.c | 34 +++++++++++++++++------
 drivers/net/ethernet/amazon/ena/ena_com.h | 14 ++++++++++
 2 files changed, 39 insertions(+), 9 deletions(-)
drivers/net/ethernet/amazon/ena/ena_com.c +25 -9
index 139b31549..5e2abdda7 100644
@@ -762,16 +762,26 @@ static int ena_com_wait_and_process_admin_cq_interrupts(struct ena_comp_ctx *com
 		admin_queue->stats.no_completion++;
 		spin_unlock_irqrestore(&admin_queue->q_lock, flags);
 
-		if (comp_ctx->status == ENA_CMD_COMPLETED)
-			pr_err("The ena device have completion but the driver didn't receive any MSI-X interrupt (cmd %d)\n",
-			       comp_ctx->cmd_opcode);
-		else
-			pr_err("The ena device doesn't send any completion for the admin cmd %d status %d\n",
+		if (comp_ctx->status == ENA_CMD_COMPLETED) {
+			pr_err("The ena device sent a completion but the driver didn't receive a MSI-X interrupt (cmd %d), autopolling mode is %s\n",
+			       comp_ctx->cmd_opcode,
+			       admin_queue->auto_polling ? "ON" : "OFF");
+			/* Check if fallback to polling is enabled */
+			if (admin_queue->auto_polling)
+				admin_queue->polling = true;
+		} else {
+			pr_err("The ena device doesn't send a completion for the admin cmd %d status %d\n",
 			       comp_ctx->cmd_opcode, comp_ctx->status);
-
-		admin_queue->running_state = false;
-		ret = -ETIME;
-		goto err;
+		}
+		/* Check if shifted to polling mode.
+		 * This will happen if there is a completion without an interrupt
+		 * and autopolling mode is enabled. Continuing normal execution in such case
+		 */
+		if (!admin_queue->polling) {
+			admin_queue->running_state = false;
+			ret = -ETIME;
+			goto err;
+		}
 	}
 
 	ret = ena_com_comp_status_to_errno(comp_ctx->comp_status);
@@ -1650,6 +1660,12 @@ void ena_com_set_admin_polling_mode(struct ena_com_dev *ena_dev, bool polling)
 	ena_dev->admin_queue.polling = polling;
 }
 
+void ena_com_set_admin_auto_polling_mode(struct ena_com_dev *ena_dev,
+					 bool polling)
+{
+	ena_dev->admin_queue.auto_polling = polling;
+}
+
 int ena_com_mmio_reg_read_request_init(struct ena_com_dev *ena_dev)
 {
 	struct ena_com_mmio_read *mmio_read = &ena_dev->mmio_read;
drivers/net/ethernet/amazon/ena/ena_com.h +14 -0
index ce36444c3..6d356cb05 100644
@@ -283,6 +283,9 @@ struct ena_com_admin_queue {
 	/* Indicate if the admin queue should poll for completion */
 	bool polling;
 
+	/* Define if fallback to polling mode should occur */
+	bool auto_polling;
+
 	u16 curr_cmd_id;
 
 	/* Indicate that the ena was initialized and can
@@ -545,6 +548,17 @@ void ena_com_set_admin_polling_mode(struct ena_com_dev *ena_dev, bool polling);
  */
 bool ena_com_get_ena_admin_polling_mode(struct ena_com_dev *ena_dev);
 
+/* ena_com_set_admin_auto_polling_mode - Enable autoswitch to polling mode
+ * @ena_dev: ENA communication layer struct
+ * @polling: Enable/Disable polling mode
+ *
+ * Set the autopolling mode.
+ * If autopolling is on:
+ * In case of missing interrupt when data is available switch to polling.
+ */
+void ena_com_set_admin_auto_polling_mode(struct ena_com_dev *ena_dev,
+					 bool polling);
+
 /* ena_com_admin_q_comp_intr_handler - admin queue interrupt handler
  * @ena_dev: ENA communication layer struct
  *
Signature · 1 line
2.17.1
#9 lore raw

PATCH v2 9/11 net: ena: optimise calculations for CQ doorbell

In reply to [email protected]

From: Sameeh Jubran <[email protected]>

This patch initially checks if CQ doorbell is needed before proceeding with the calculations.

---
 drivers/net/ethernet/amazon/ena/ena_eth_com.h | 20 ++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)
drivers/net/ethernet/amazon/ena/ena_eth_com.h +11 -9
index 0a3d9180e..77986c0ea 100644
@@ -195,15 +195,17 @@ static inline int ena_com_update_dev_comp_head(struct ena_com_io_cq *io_cq)
 	u16 unreported_comp, head;
 	bool need_update;
 
-	head = io_cq->head;
-	unreported_comp = head - io_cq->last_head_update;
-	need_update = unreported_comp > (io_cq->q_depth / ENA_COMP_HEAD_THRESH);
-
-	if (io_cq->cq_head_db_reg && need_update) {
-		pr_debug("Write completion queue doorbell for queue %d: head: %d\n",
-			 io_cq->qid, head);
-		writel(head, io_cq->cq_head_db_reg);
-		io_cq->last_head_update = head;
+	if (unlikely(io_cq->cq_head_db_reg)) {
+		head = io_cq->head;
+		unreported_comp = head - io_cq->last_head_update;
+		need_update = unreported_comp > (io_cq->q_depth / ENA_COMP_HEAD_THRESH);
+
+		if (unlikely(need_update)) {
+			pr_debug("Write completion queue doorbell for queue %d: head: %d\n",
+				 io_cq->qid, head);
+			writel(head, io_cq->cq_head_db_reg);
+			io_cq->last_head_update = head;
+		}
 	}
 
 	return 0;
Signature · 1 line
2.17.1
#3 lore raw

PATCH v2 3/11 net: ena: replace free_tx/rx_ids union with single free_ids field in ena_ring

In reply to [email protected]

From: Sameeh Jubran <[email protected]>

struct ena_ring holds a union of free_rx_ids and free_tx_ids. Both of the above fields mean the exact same thing and are used exactly the same way.
Furthermore, these fields are always used with a prefix of the type of ring. So for tx it will be tx_ring->free_tx_ids, and for rx it will be rx_ring->free_rx_ids, which shows how redundant the "_tx" and "_rx" parts are.
Furthermore still, this may lead to confusing code like where tx_ring->free_rx_ids which works correctly but looks like a mess.

This commit removes the aforementioned redundancy by replacing the free_rx/tx_ids union with a single free_ids field.
It also changes a single goto label name from err_free_tx_ids: to
err_tx_free_ids: for consistency with the above new notation.

---
 drivers/net/ethernet/amazon/ena/ena_netdev.c | 48 ++++++++++----------
 drivers/net/ethernet/amazon/ena/ena_netdev.h | 11 ++---
 2 files changed, 28 insertions(+), 31 deletions(-)
drivers/net/ethernet/amazon/ena/ena_netdev.c +24 -24
index 33fab4f41..b80b5eddc 100644
@@ -228,11 +228,11 @@ static int ena_setup_tx_resources(struct ena_adapter *adapter, int qid)
 	}
 
 	size = sizeof(u16) * tx_ring->ring_size;
-	tx_ring->free_tx_ids = vzalloc_node(size, node);
-	if (!tx_ring->free_tx_ids) {
-		tx_ring->free_tx_ids = vzalloc(size);
-		if (!tx_ring->free_tx_ids)
-			goto err_free_tx_ids;
+	tx_ring->free_ids = vzalloc_node(size, node);
+	if (!tx_ring->free_ids) {
+		tx_ring->free_ids = vzalloc(size);
+		if (!tx_ring->free_ids)
+			goto err_tx_free_ids;
 	}
 
 	size = tx_ring->tx_max_header_size;
@@ -245,7 +245,7 @@ static int ena_setup_tx_resources(struct ena_adapter *adapter, int qid)
 
 	/* Req id ring for TX out of order completions */
 	for (i = 0; i < tx_ring->ring_size; i++)
-		tx_ring->free_tx_ids[i] = i;
+		tx_ring->free_ids[i] = i;
 
 	/* Reset tx statistics */
 	memset(&tx_ring->tx_stats, 0x0, sizeof(tx_ring->tx_stats));
@@ -256,9 +256,9 @@ static int ena_setup_tx_resources(struct ena_adapter *adapter, int qid)
 	return 0;
 
 err_push_buf_intermediate_buf:
-	vfree(tx_ring->free_tx_ids);
-	tx_ring->free_tx_ids = NULL;
-err_free_tx_ids:
+	vfree(tx_ring->free_ids);
+	tx_ring->free_ids = NULL;
+err_tx_free_ids:
 	vfree(tx_ring->tx_buffer_info);
 	tx_ring->tx_buffer_info = NULL;
 err_tx_buffer_info:
@@ -278,8 +278,8 @@ static void ena_free_tx_resources(struct ena_adapter *adapter, int qid)
 	vfree(tx_ring->tx_buffer_info);
 	tx_ring->tx_buffer_info = NULL;
 
-	vfree(tx_ring->free_tx_ids);
-	tx_ring->free_tx_ids = NULL;
+	vfree(tx_ring->free_ids);
+	tx_ring->free_ids = NULL;
 
 	vfree(tx_ring->push_buf_intermediate_buf);
 	tx_ring->push_buf_intermediate_buf = NULL;
@@ -377,10 +377,10 @@ static int ena_setup_rx_resources(struct ena_adapter *adapter,
 	}
 
 	size = sizeof(u16) * rx_ring->ring_size;
-	rx_ring->free_rx_ids = vzalloc_node(size, node);
-	if (!rx_ring->free_rx_ids) {
-		rx_ring->free_rx_ids = vzalloc(size);
-		if (!rx_ring->free_rx_ids) {
+	rx_ring->free_ids = vzalloc_node(size, node);
+	if (!rx_ring->free_ids) {
+		rx_ring->free_ids = vzalloc(size);
+		if (!rx_ring->free_ids) {
 			vfree(rx_ring->rx_buffer_info);
 			rx_ring->rx_buffer_info = NULL;
 			return -ENOMEM;
@@ -389,7 +389,7 @@ static int ena_setup_rx_resources(struct ena_adapter *adapter,
 
 	/* Req id ring for receiving RX pkts out of order */
 	for (i = 0; i < rx_ring->ring_size; i++)
-		rx_ring->free_rx_ids[i] = i;
+		rx_ring->free_ids[i] = i;
 
 	/* Reset rx statistics */
 	memset(&rx_ring->rx_stats, 0x0, sizeof(rx_ring->rx_stats));
@@ -415,8 +415,8 @@ static void ena_free_rx_resources(struct ena_adapter *adapter,
 	vfree(rx_ring->rx_buffer_info);
 	rx_ring->rx_buffer_info = NULL;
 
-	vfree(rx_ring->free_rx_ids);
-	rx_ring->free_rx_ids = NULL;
+	vfree(rx_ring->free_ids);
+	rx_ring->free_ids = NULL;
 }
 
 /* ena_setup_all_rx_resources - allocate I/O Rx queues resources for all queues
@@ -531,7 +531,7 @@ static int ena_refill_rx_bufs(struct ena_ring *rx_ring, u32 num)
 	for (i = 0; i < num; i++) {
 		struct ena_rx_buffer *rx_info;
 
-		req_id = rx_ring->free_rx_ids[next_to_use];
+		req_id = rx_ring->free_ids[next_to_use];
 		rc = validate_rx_req_id(rx_ring, req_id);
 		if (unlikely(rc < 0))
 			break;
@@ -797,7 +797,7 @@ static int ena_clean_tx_irq(struct ena_ring *tx_ring, u32 budget)
 		tx_pkts++;
 		total_done += tx_info->tx_descs;
 
-		tx_ring->free_tx_ids[next_to_clean] = req_id;
+		tx_ring->free_ids[next_to_clean] = req_id;
 		next_to_clean = ENA_TX_RING_IDX_NEXT(next_to_clean,
 						     tx_ring->ring_size);
 	}
@@ -911,7 +911,7 @@ static struct sk_buff *ena_rx_skb(struct ena_ring *rx_ring,
 
 		skb_put(skb, len);
 		skb->protocol = eth_type_trans(skb, rx_ring->netdev);
-		rx_ring->free_rx_ids[*next_to_clean] = req_id;
+		rx_ring->free_ids[*next_to_clean] = req_id;
 		*next_to_clean = ENA_RX_RING_IDX_ADD(*next_to_clean, descs,
 						     rx_ring->ring_size);
 		return skb;
@@ -935,7 +935,7 @@ static struct sk_buff *ena_rx_skb(struct ena_ring *rx_ring,
 
 		rx_info->page = NULL;
 
-		rx_ring->free_rx_ids[*next_to_clean] = req_id;
+		rx_ring->free_ids[*next_to_clean] = req_id;
 		*next_to_clean =
 			ENA_RX_RING_IDX_NEXT(*next_to_clean,
 					     rx_ring->ring_size);
@@ -1088,7 +1088,7 @@ static int ena_clean_rx_irq(struct ena_ring *rx_ring, struct napi_struct *napi,
 		/* exit if we failed to retrieve a buffer */
 		if (unlikely(!skb)) {
 			for (i = 0; i < ena_rx_ctx.descs; i++) {
-				rx_ring->free_tx_ids[next_to_clean] =
+				rx_ring->free_ids[next_to_clean] =
 					rx_ring->ena_bufs[i].req_id;
 				next_to_clean =
 					ENA_RX_RING_IDX_NEXT(next_to_clean,
@@ -2152,7 +2152,7 @@ static netdev_tx_t ena_start_xmit(struct sk_buff *skb, struct net_device *dev)
 	skb_tx_timestamp(skb);
 
 	next_to_use = tx_ring->next_to_use;
-	req_id = tx_ring->free_tx_ids[next_to_use];
+	req_id = tx_ring->free_ids[next_to_use];
 	tx_info = &tx_ring->tx_buffer_info[req_id];
 	tx_info->num_of_bufs = 0;
 
drivers/net/ethernet/amazon/ena/ena_netdev.h +4 -7
index 0681e18b0..74c316081 100644
@@ -221,13 +221,10 @@ struct ena_stats_rx {
 };
 
 struct ena_ring {
-	union {
-		/* Holds the empty requests for TX/RX
-		 * out of order completions
-		 */
-		u16 *free_tx_ids;
-		u16 *free_rx_ids;
-	};
+	/* Holds the empty requests for TX/RX
+	 * out of order completions
+	 */
+	u16 *free_ids;
 
 	union {
 		struct ena_tx_buffer *tx_buffer_info;
Signature · 1 line
2.17.1
#2 lore raw

PATCH v2 2/11 net: ena: ethtool: add extra properties retrieval via get_priv_flags

In reply to [email protected]

From: Arthur Kiyanovski <[email protected]>

This commit adds a mechanism for exposing different device properties via ethtool's priv_flags. The strings are provided by the device and copied to user space through the driver.

In this commit we:

Add commands, structs and defines necessary for handling extra properties

Add functions for:
Allocation/destruction of a buffer for extra properties strings. Retreival of extra properties strings and flags from the network device.

Handle the allocation of a buffer for extra properties strings.

* Initialize buffer with extra properties strings from the network device at driver startup.

Use ethtool's get_priv_flags to expose extra properties of the ENA device

---
 .../net/ethernet/amazon/ena/ena_admin_defs.h  | 16 ++++
 drivers/net/ethernet/amazon/ena/ena_com.c     | 56 ++++++++++++++
 drivers/net/ethernet/amazon/ena/ena_com.h     | 32 ++++++++
 drivers/net/ethernet/amazon/ena/ena_ethtool.c | 75 ++++++++++++++++---
 drivers/net/ethernet/amazon/ena/ena_netdev.c  | 14 ++++
 drivers/net/ethernet/amazon/ena/ena_netdev.h  |  2 +
 6 files changed, 184 insertions(+), 11 deletions(-)
drivers/net/ethernet/amazon/ena/ena_admin_defs.h +16 -0
index 82cff1e12..414bae989 100644
@@ -32,6 +32,8 @@
 #ifndef _ENA_ADMIN_H_
 #define _ENA_ADMIN_H_
 
+#define ENA_ADMIN_EXTRA_PROPERTIES_STRING_LEN 32
+#define ENA_ADMIN_EXTRA_PROPERTIES_COUNT     32
 
 enum ena_admin_aq_opcode {
 	ENA_ADMIN_CREATE_SQ                         = 1,
@@ -60,6 +62,8 @@ enum ena_admin_aq_feature_id {
 	ENA_ADMIN_MAX_QUEUES_NUM                    = 2,
 	ENA_ADMIN_HW_HINTS                          = 3,
 	ENA_ADMIN_LLQ                               = 4,
+	ENA_ADMIN_EXTRA_PROPERTIES_STRINGS          = 5,
+	ENA_ADMIN_EXTRA_PROPERTIES_FLAGS            = 6,
 	ENA_ADMIN_RSS_HASH_FUNCTION                 = 10,
 	ENA_ADMIN_STATELESS_OFFLOAD_CONFIG          = 11,
 	ENA_ADMIN_RSS_REDIRECTION_TABLE_CONFIG      = 12,
@@ -560,6 +564,14 @@ struct ena_admin_set_feature_mtu_desc {
 	u32 mtu;
 };
 
+struct ena_admin_get_extra_properties_strings_desc {
+	u32 count;
+};
+
+struct ena_admin_get_extra_properties_flags_desc {
+	u32 flags;
+};
+
 struct ena_admin_set_feature_host_attr_desc {
 	/* host OS info base address in OS memory. host info is 4KB of
 	 * physically contiguous
@@ -864,6 +876,10 @@ struct ena_admin_get_feat_resp {
 		struct ena_admin_feature_intr_moder_desc intr_moderation;
 
 		struct ena_admin_ena_hw_hints hw_hints;
+
+		struct ena_admin_get_extra_properties_strings_desc extra_properties_strings;
+
+		struct ena_admin_get_extra_properties_flags_desc extra_properties_flags;
 	} u;
 };
 
drivers/net/ethernet/amazon/ena/ena_com.c +56 -0

62 lines not on this page · raw message

drivers/net/ethernet/amazon/ena/ena_com.h +32 -0
index f0cb043af..ce36444c3 100644
@@ -347,6 +347,12 @@ struct ena_host_attribute {
 	dma_addr_t host_info_dma_addr;
 };
 
+struct ena_extra_properties_strings {
+	u8 *virt_addr;
+	dma_addr_t dma_addr;
+	u32 size;
+};
+
 /* Each ena_dev is a PCI function. */
 struct ena_com_dev {
 	struct ena_com_admin_queue admin_queue;
@@ -375,6 +381,7 @@ struct ena_com_dev {
 	struct ena_intr_moder_entry *intr_moder_tbl;
 
 	struct ena_com_llq_info llq_info;
+	struct ena_extra_properties_strings extra_properties_strings;
 };
 
 struct ena_com_dev_get_features_ctx {
@@ -596,6 +603,31 @@ int ena_com_validate_version(struct ena_com_dev *ena_dev);
 int ena_com_get_link_params(struct ena_com_dev *ena_dev,
 			    struct ena_admin_get_feat_resp *resp);
 
+/* ena_com_extra_properties_strings_init - Initialize the extra properties strings buffer.
+ * @ena_dev: ENA communication layer struct
+ *
+ * Initialize the extra properties strings buffer.
+ */
+int ena_com_extra_properties_strings_init(struct ena_com_dev *ena_dev);
+
+/* ena_com_delete_extra_properties_strings - Free the extra properties strings buffer.
+ * @ena_dev: ENA communication layer struct
+ *
+ * Free the allocated extra properties strings buffer.
+ */
+void ena_com_delete_extra_properties_strings(struct ena_com_dev *ena_dev);
+
+/* ena_com_get_extra_properties_flags - Retrieve extra properties flags.
+ * @ena_dev: ENA communication layer struct
+ * @resp: Extra properties flags.
+ *
+ * Retrieve the extra properties flags.
+ *
+ * @return - 0 on Success negative value otherwise.
+ */
+int ena_com_get_extra_properties_flags(struct ena_com_dev *ena_dev,
+				       struct ena_admin_get_feat_resp *resp);
+
 /* ena_com_get_dma_width - Retrieve physical dma address width the device
  * supports.
  * @ena_dev: ENA communication layer struct
drivers/net/ethernet/amazon/ena/ena_ethtool.c +64 -11

123 lines not on this page · raw message

drivers/net/ethernet/amazon/ena/ena_netdev.c +14 -0
index d2b82f917..33fab4f41 100644
@@ -2369,6 +2369,14 @@ err:
 	ena_com_delete_debug_area(adapter->ena_dev);
 }
 
+static void ena_extra_properties_strings_destroy(struct net_device *netdev)
+{
+	struct ena_adapter *adapter = netdev_priv(netdev);
+
+	ena_com_delete_extra_properties_strings(adapter->ena_dev);
+	adapter->ena_extra_properties_count = 0;
+}
+
 static void ena_get_stats64(struct net_device *netdev,
 			    struct rtnl_link_stats64 *stats)
 {
@@ -3424,6 +3432,9 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	ena_config_debug_area(adapter);
 
+	adapter->ena_extra_properties_count =
+		ena_com_extra_properties_strings_init(ena_dev);
+
 	memcpy(adapter->netdev->perm_addr, adapter->mac_addr, netdev->addr_len);
 
 	netif_carrier_off(netdev);
@@ -3463,6 +3474,7 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	return 0;
 
 err_rss:
+	ena_extra_properties_strings_destroy(netdev);
 	ena_com_delete_debug_area(ena_dev);
 	ena_com_rss_destroy(ena_dev);
 err_free_msix:
@@ -3529,6 +3541,8 @@ static void ena_remove(struct pci_dev *pdev)
 
 	ena_com_delete_host_info(ena_dev);
 
+	ena_extra_properties_strings_destroy(netdev);
+
 	ena_release_bars(ena_dev, pdev);
 
 	pci_disable_device(pdev);
drivers/net/ethernet/amazon/ena/ena_netdev.h +2 -0
index 63870072c..0681e18b0 100644
@@ -364,6 +364,8 @@ struct ena_adapter {
 	u32 last_monitored_tx_qid;
 
 	enum ena_regs_reset_reason_types reset_reason;
+
+	u8 ena_extra_properties_count;
 };
 
 void ena_set_ethtool_ops(struct net_device *netdev);
Signature · 1 line
2.17.1
#10 lore raw

PATCH v2 10/11 net: ena: add good checksum counter

In reply to [email protected]

From: Sameeh Jubran <[email protected]>

Add a new statistics to ETHTOOL to specify if the device calculated and validated the Rx csum.

---
 drivers/net/ethernet/amazon/ena/ena_ethtool.c | 3 ++-
 drivers/net/ethernet/amazon/ena/ena_netdev.c  | 3 +++
 drivers/net/ethernet/amazon/ena/ena_netdev.h  | 3 ++-
 3 files changed, 7 insertions(+), 2 deletions(-)
drivers/net/ethernet/amazon/ena/ena_ethtool.c +2 -1
index 28fe4c6ef..5687a2860 100644
@@ -88,13 +88,14 @@ static const struct ena_stats ena_stats_tx_strings[] = {
 static const struct ena_stats ena_stats_rx_strings[] = {
 	ENA_STAT_RX_ENTRY(cnt),
 	ENA_STAT_RX_ENTRY(bytes),
+	ENA_STAT_RX_ENTRY(rx_copybreak_pkt),
+	ENA_STAT_RX_ENTRY(csum_good),
 	ENA_STAT_RX_ENTRY(refil_partial),
 	ENA_STAT_RX_ENTRY(bad_csum),
 	ENA_STAT_RX_ENTRY(page_alloc_fail),
 	ENA_STAT_RX_ENTRY(skb_alloc_fail),
 	ENA_STAT_RX_ENTRY(dma_mapping_err),
 	ENA_STAT_RX_ENTRY(bad_desc_num),
-	ENA_STAT_RX_ENTRY(rx_copybreak_pkt),
 	ENA_STAT_RX_ENTRY(bad_req_id),
 	ENA_STAT_RX_ENTRY(empty_rx_ring),
 	ENA_STAT_RX_ENTRY(csum_unchecked),
drivers/net/ethernet/amazon/ena/ena_netdev.c +3 -0
index 399bd5453..cff297e19 100644
@@ -1001,6 +1001,9 @@ static inline void ena_rx_checksum(struct ena_ring *rx_ring,
 
 		if (likely(ena_rx_ctx->l4_csum_checked)) {
 			skb->ip_summed = CHECKSUM_UNNECESSARY;
+			u64_stats_update_begin(&rx_ring->syncp);
+			rx_ring->rx_stats.csum_good++;
+			u64_stats_update_end(&rx_ring->syncp);
 		} else {
 			u64_stats_update_begin(&rx_ring->syncp);
 			rx_ring->rx_stats.csum_unchecked++;
drivers/net/ethernet/amazon/ena/ena_netdev.h +2 -1
index 74c316081..ec111cfc5 100644
@@ -208,13 +208,14 @@ struct ena_stats_tx {
 struct ena_stats_rx {
 	u64 cnt;
 	u64 bytes;
+	u64 rx_copybreak_pkt;
+	u64 csum_good;
 	u64 refil_partial;
 	u64 bad_csum;
 	u64 page_alloc_fail;
 	u64 skb_alloc_fail;
 	u64 dma_mapping_err;
 	u64 bad_desc_num;
-	u64 rx_copybreak_pkt;
 	u64 bad_req_id;
 	u64 empty_rx_ring;
 	u64 csum_unchecked;
Signature · 1 line
2.17.1
#11 lore raw

PATCH v2 11/11 net: ena: use dev_info_once instead of static variable

In reply to [email protected]

From: Sameeh Jubran <[email protected]>

---
 drivers/net/ethernet/amazon/ena/ena_netdev.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
drivers/net/ethernet/amazon/ena/ena_netdev.c +1 -3
index cff297e19..68bed2417 100644
@@ -3287,7 +3287,6 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	struct ena_llq_configurations llq_config;
 	struct ena_com_dev *ena_dev = NULL;
 	struct ena_adapter *adapter;
-	static int version_printed;
 	int io_queue_num, bars, rc;
 	struct net_device *netdev;
 	static int adapters_found;
@@ -3299,8 +3298,7 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	dev_dbg(&pdev->dev, "%s\n", __func__);
 
-	if (version_printed++ == 0)
-		dev_info(&pdev->dev, "%s", version);
+	dev_info_once(&pdev->dev, "%s", version);
 
 	rc = pci_enable_device_mem(pdev);
 	if (rc) {
Signature · 1 line
2.17.1
maintainer
#12 lore raw
In reply to [email protected]

From: <[email protected]>
Date: Mon, 3 Jun 2019 17:43:18 +0300

Quoted 15 lines

From: Sameeh Jubran <[email protected]>

This patchset introduces the following:

* add support for changing the inline header size (max_header_size) for applications with overlay and nested headers
* enable automatic fallback to polling mode for admin queue when interrupt is not available or missed
* add good checksum counter for Rx ethtool statistics
* update ena.txt
* some minor code clean-up
* some performance enhancements with doorbell calculations

Differences from V1:

Quoted 7 lines

* net: ena: add handling of llq max tx burst size (1/11):
* fixed christmas tree issue

* net: ena: ethtool: add extra properties retrieval via get_priv_flags (2/11):
* replaced snprintf with strlcpy
* dropped confusing error message
* added more details to the commit message

Series applied.

#13 lore raw
In reply to [email protected]
Quoted 4 lines · On Mon, 3 Jun 2019 17:43:18 +0300, [email protected] wrote:

* net: ena: ethtool: add extra properties retrieval via get_priv_flags (2/11):
* replaced snprintf with strlcpy
* dropped confusing error message
* added more details to the commit message

I asked you to clearly state that you are using the blindly passing this info from the FW to user space. Stating that you "retrieve" it is misleading.

IMHO it's a dangerous precedent, you're extending kernel's uAPI down to the proprietary FW of the device. Now we have no idea what the flags are, so we can't refactor and create common APIs among drivers, or even use the same names. We have no idea what you're exposing.

#15 lore raw
In reply to a message not in the archive
Quoted 30 lines · David
Quoted 9 lines · Jakub Kicinski
Quoted 4 lines · On Mon, 3 Jun 2019 17:43:18 +0300, [email protected] wrote:

* net: ena: ethtool: add extra properties retrieval via get_priv_flags (2/11):
* replaced snprintf with strlcpy
* dropped confusing error message
* added more details to the commit message

I asked you to clearly state that you are using the blindly passing this info from the FW to user space. Stating that you "retrieve" it is misleading.

Yes, we should be very clear about that.

Quoted 4 lines

IMHO it's a dangerous precedent, you're extending kernel's uAPI down to the proprietary FW of the device. Now we have no idea what the flags are, so we can't refactor and create common APIs among drivers, or even use the same names. We have no idea what you're exposing.

Yes, that should be documented very clearly too, and we should absolutely make sure that anything that makes sense for other devices is considered for making a common API.

However, the deployment environment for ENA is kind of weird — we get to upgrade the *firmware*, while guest kernels might get updated only once a decade. So the passthrough you're objecting to, actually gives us fairly much the only viable way to offer many users the tuning options they need — or at least to experiment and establish whether they *do* need them or not, and thus if we should turn them into a "real" generic property.

Quoted 4 lines

You do have a valid concern, but I think we can handle it, and I suspect that the approach taken here does make sense. Let's just make it explicitly clear, and also document the properties that we expect to be exposing, for visibility.

Thank you.

It's generally easier to push FW updates, also to enterprises, rather than rolling out full new kernels. People are less concerned with FW updates, because the only thing those can break is the NIC. Some customers also run their own modified kernels. I'd dispute the fact that Amazon is so very special.

I don't dispute that this is convenient for Amazon and your FW team. However, what's best for the vendors differs here from what's good for Open Source (the FW is proprietary) and IMHO health of Linux ecosystem in general.

Any "SmartNIC" vendor has temptation of uAPI-level hand off to the firmware (including my employer), we all run pretty beefy processors inside "the NIC" after all. The device centric ethtool configuration can be implemented by just forwarding the uAPI structures as they are to the FW. I'm sure Andrew and others who would like to see Linux takes more control over PHYs etc. would not like this scenario, either.

To address your specific points, let's be realistic, after initial submission it's unlikely the features will be updated in a timely manner anywhere else than perhaps some Amazon's own docs. Can you be more specific on what those tuning options are today? There are users who don't care to update their kernels for 10 years, yet they care about some minor tuning option of the NIC?

#16 lore raw
Quoted 6 lines

Any "SmartNIC" vendor has temptation of uAPI-level hand off to the firmware (including my employer), we all run pretty beefy processors inside "the NIC" after all. The device centric ethtool configuration can be implemented by just forwarding the uAPI structures as they are to the FW. I'm sure Andrew and others who would like to see Linux takes more control over PHYs etc. would not like this scenario, either.

No, i would not. There are a few good examples of both firmware and open drivers being used to control the same PHY, on different boards. The PHY driver was developed by the community, and has more features than the firmware driver. And it keeps gaining features. The firmware i stuck, no updates. The community driver can be debugged, the firmware is a black box, no chance of the community fixing any bugs in it.

And PHYs are commodity devices. I doubt there is any value add in the firmware for a PHY, any real IPR which makes the product better, magic sauce related to the PHY. So just save the cost of writing and maintaining firmware, export the MDIO bus, and let Linux control it. Concentrate the engineers on the interesting parts of the NIC, the Smart parts, where there can be real IPR.

And i would say this is true for any NIC. Let Linux control the PHY.

Andrew

#17 lore raw

Andrew,

Sent from my iPhone

Quoted 22 lines · Andrew Lunn
Quoted 6 lines

Any "SmartNIC" vendor has temptation of uAPI-level hand off to the firmware (including my employer), we all run pretty beefy processors inside "the NIC" after all. The device centric ethtool configuration can be implemented by just forwarding the uAPI structures as they are to the FW. I'm sure Andrew and others who would like to see Linux takes more control over PHYs etc. would not like this scenario, either.

No, i would not. There are a few good examples of both firmware and open drivers being used to control the same PHY, on different boards. The PHY driver was developed by the community, and has more features than the firmware driver. And it keeps gaining features. The firmware i stuck, no updates. The community driver can be debugged, the firmware is a black box, no chance of the community fixing any bugs in it.

And PHYs are commodity devices. I doubt there is any value add in the firmware for a PHY, any real IPR which makes the product better, magic sauce related to the PHY. So just save the cost of writing and maintaining firmware, export the MDIO bus, and let Linux control it. Concentrate the engineers on the interesting parts of the NIC, the Smart parts, where there can be real IPR.

Quoted 4 lines

And i would say this is true for any NIC. Let Linux control the PHY.

Andrew

It may be true for old GbE PHYs where it’s a discrete chip from the likes of Marvell or broadcom

But at 25/50/100G, the PHy is actually part of the nic. It’s a very complex SERDES. Cloud providers like us spend enormous amount of time testing the PHY across process and voltage variations, all cable types, length and manufacturing variations, and against all switches we use. Community drivers won’t be able to validate and tune all this.

Plus we would need exact same setting for Linux, including all distributions even 10year old like RHEL6, for all Windows, ESX, DPDK, FreeBSD, and support millions of different customers with different sets of Machine images.

In this case, there is no practical choice by have the firmware to manage the PHY

#18 lore raw
Quoted 43 lines · Nafea

Andrew,

Sent from my iPhone

Quoted 22 lines · Andrew Lunn
Quoted 6 lines

Any "SmartNIC" vendor has temptation of uAPI-level hand off to the firmware (including my employer), we all run pretty beefy processors inside "the NIC" after all. The device centric ethtool configuration can be implemented by just forwarding the uAPI structures as they are to the FW. I'm sure Andrew and others who would like to see Linux takes more control over PHYs etc. would not like this scenario, either.

No, i would not. There are a few good examples of both firmware and open drivers being used to control the same PHY, on different boards. The PHY driver was developed by the community, and has more features than the firmware driver. And it keeps gaining features. The firmware i stuck, no updates. The community driver can be debugged, the firmware is a black box, no chance of the community fixing any bugs in it.

And PHYs are commodity devices. I doubt there is any value add in the firmware for a PHY, any real IPR which makes the product better, magic sauce related to the PHY. So just save the cost of writing and maintaining firmware, export the MDIO bus, and let Linux control it. Concentrate the engineers on the interesting parts of the NIC, the Smart parts, where there can be real IPR.

Quoted 4 lines

And i would say this is true for any NIC. Let Linux control the PHY.

Andrew

It may be true for old GbE PHYs where it’s a discrete chip from the likes of Marvell or broadcom

But at 25/50/100G, the PHy is actually part of the nic. It’s a very complex SERDES. Cloud providers like us spend enormous amount of time testing the PHY across process and voltage variations, all cable types, length and manufacturing variations, and against all switches we use. Community drivers won’t be able to validate and tune all this.

Quoted 7 lines

Plus we would need exact same setting for Linux, including all distributions even 10year old like RHEL6, for all Windows, ESX, DPDK, FreeBSD, and support millions of different customers with different sets of Machine images.

In this case, there is no practical choice by have the firmware to manage the PHY

I don't quite know why we're talking about PHYs in this context. ENA is basically a virtio NIC. It has no PHY.

#19 lore raw
Quoted 41 lines · David Woodhouse
Quoted 40 lines · Nafea
Quoted 15 lines · Andrew Lunn
Quoted 6 lines

Any "SmartNIC" vendor has temptation of uAPI-level hand off to the firmware (including my employer), we all run pretty beefy processors inside "the NIC" after all. The device centric ethtool configuration can be implemented by just forwarding the uAPI structures as they are to the FW. I'm sure Andrew and others who would like to see Linux takes more control over PHYs etc. would not like this scenario, either.

No, i would not. There are a few good examples of both firmware and open drivers being used to control the same PHY, on different boards. The PHY driver was developed by the community, and has more features than the firmware driver. And it keeps gaining features. The firmware i stuck, no updates. The community driver can be debugged, the firmware is a black box, no chance of the community fixing any bugs in it.

Quoted 8 lines

And PHYs are commodity devices. I doubt there is any value add in the firmware for a PHY, any real IPR which makes the product better, magic sauce related to the PHY. So just save the cost of writing and maintaining firmware, export the MDIO bus, and let Linux control it. Concentrate the engineers on the interesting parts of the NIC, the Smart parts, where there can be real IPR.

And i would say this is true for any NIC. Let Linux control the PHY.

It may be true for old GbE PHYs where it’s a discrete chip from the likes of Marvell or broadcom

But at 25/50/100G, the PHy is actually part of the nic. It’s a very complex SERDES. Cloud providers like us spend enormous amount of time testing the PHY across process and voltage variations, all cable types, length and manufacturing variations, and against all switches we use. Community drivers won’t be able to validate and tune all this.

Plus we would need exact same setting for Linux, including all distributions even 10year old like RHEL6, for all Windows, ESX, DPDK, FreeBSD, and support millions of different customers with different sets of Machine images.

Quoted 5 lines
Quoted 2 lines

In this case, there is no practical choice by have the firmware to manage the PHY

I don't quite know why we're talking about PHYs in this context. ENA is basically a virtio NIC. It has no PHY.

I brought it up as an example, to illustrate that we'd rather see less trust and control being blindly handed over to the firmware.

Would you mind answering what are the examples of very important flags you need to expose to users with 10yo kernels? Or any examples for that matter..

#20 lore raw

As of today there are no flags exposed by ENA NIC device, however, we are planning to use them in the near future.
We want to provide customers with extra methods to identify (and differentiate) multiple network interfaces that can be attached to a single VM. Currently, customers can identify a specific network interface (ENI) only by MAC address, and later look up this MAC among other multiple ENIs that they have. In some cases it might not be convenient. Using these flags will let us inform the customer about ENI`s specific properties. It's not finalized, but tentatively it can look like this:
primary-eni: on /* Differentiate between primary and secondary ENIs */
has-associated-efa: on /* Will specify ENA device has another associated EFA device */

Quoted 72 lines

-----Original Message-----
From: Jakub Kicinski <[email protected]>
Sent: Tuesday, June 4, 2019 8:24 PM
To: David Woodhouse <[email protected]>
Cc: Bshara, Nafea <[email protected]>; Andrew Lunn <[email protected]>; Jubran, Samih <[email protected]>; Kiyanovski, Arthur <[email protected]>; Bshara, Saeed <[email protected]>; Tzalik, Guy <[email protected]>; Matushevsky, Alexander <[email protected]>; Liguori, Anthony <[email protected]>; Saidi, Ali <[email protected]>; Machulsky, Zorik <[email protected]>; [email protected]; Wilson, Matt <[email protected]>; [email protected]; Belgazal, Netanel <[email protected]>; Herrenschmidt, Benjamin <[email protected]>
Subject: Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

Quoted 8 lines · David Woodhouse
Quoted 7 lines · Nafea
Quoted 6 lines · Andrew Lunn
Quoted 6 lines

Any "SmartNIC" vendor has temptation of uAPI-level hand off to the firmware (including my employer), we all run pretty beefy processors inside "the NIC" after all. The device centric ethtool configuration can be implemented by just forwarding the uAPI structures as they are to the FW. I'm sure Andrew and others who would like to see Linux takes more control over PHYs etc.

would not like this scenario, either.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

No, i would not. There are a few good examples of both firmware and open drivers being used to control the same PHY, on different boards. The PHY driver was developed by the community, and has more features than the firmware driver. And it keeps gaining features. The firmware i stuck, no updates. The community driver can be debugged, the firmware is a black box, no chance of the community fixing any bugs in it.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

And PHYs are commodity devices. I doubt there is any value add in the firmware for a PHY, any real IPR which makes the product better, magic sauce related to the PHY. So just save the cost of writing and maintaining firmware, export the MDIO bus, and let Linux

control it.

Quoted 20 lines
Quoted 20 lines
Quoted 4 lines

Concentrate the engineers on the interesting parts of the NIC, the Smart parts, where there can be real IPR.

And i would say this is true for any NIC. Let Linux control the PHY.

It may be true for old GbE PHYs where it’s a discrete chip from the likes of Marvell or broadcom

But at 25/50/100G, the PHy is actually part of the nic. It’s a very complex SERDES. Cloud providers like us spend enormous amount of time testing the PHY across process and voltage variations, all cable types, length and manufacturing variations, and against all switches we use. Community drivers won’t be able to validate and tune all this.

Plus we would need exact same setting for Linux, including all distributions even 10year old like RHEL6, for all Windows, ESX, DPDK, FreeBSD, and support millions of different customers with different sets of Machine images.

Quoted 5 lines
Quoted 2 lines

In this case, there is no practical choice by have the firmware to manage the PHY

I don't quite know why we're talking about PHYs in this context. ENA is basically a virtio NIC. It has no PHY.

I brought it up as an example, to illustrate that we'd rather see less trust and control being blindly handed over to the firmware.

Would you mind answering what are the examples of very important flags you need to expose to users with 10yo kernels? Or any examples for that matter..

#21 lore raw

Hi Samih!

Please don't top post on Linux kernel mailing lists.

Quoted 9 lines · Samih

As of today there are no flags exposed by ENA NIC device, however, we are planning to use them in the near future. We want to provide customers with extra methods to identify (and differentiate) multiple network interfaces that can be attached to a single VM. Currently, customers can identify a specific network interface (ENI) only by MAC address, and later look up this MAC among other multiple ENIs that they have. In some cases it might not be convenient. Using these flags will let us inform the customer about ENI`s specific properties.

Oh no :( You're using private _feature_ flags as labels or tags.

Quoted 2 lines

It's not finalized, but tentatively it can look like this:
primary-eni: on /* Differentiate between primary and secondary ENIs */

Did you consider using phys_port_name for this use case?

If the intent is to have those interfaces bonded, even better would be to extend the net_failover module or work on user space ABI for VM failover. That might be a bigger effort, though.

Quoted 1 line

has-associated-efa: on /* Will specify ENA device has another associated EFA device */

IDK how your ENA/EFA thing works, but sounds like something that should be solved in the device model.

#22 lore raw

On 6/6/19, 10:16 AM, "Jakub Kicinski" <[email protected]> wrote:

Hi Samih!

Please don't top post on Linux kernel mailing lists.

On Thu, 6 Jun 2019 10:23:40 +0000, Jubran, Samih wrote:
> As of today there are no flags exposed by ENA NIC device, however, we > are planning to use them in the near future. We want to provide > customers with extra methods to identify (and differentiate) multiple > network interfaces that can be attached to a single VM. Currently, > customers can identify a specific network interface (ENI) only by MAC > address, and later look up this MAC among other multiple ENIs that > they have. In some cases it might not be convenient. Using these > flags will let us inform the customer about ENI`s specific > properties.

Oh no :( You're using private _feature_ flags as labels or tags.

> It's not finalized, but tentatively it can look like this:
> primary-eni: on /* Differentiate between primary and secondary ENIs */

Did you consider using phys_port_name for this use case?

If the intent is to have those interfaces bonded, even better would be to extend the net_failover module or work on user space ABI for VM failover. That might be a bigger effort, though.

> has-associated-efa: on /* Will specify ENA device has another associated EFA device */

IDK how your ENA/EFA thing works, but sounds like something that should be solved in the device model.

[NB] ENA is a netDev, EFA is an ib_dev. Both share the same physical network
All previous attempt to make them share at device level leads to a lot of complexity at the OS level, with inter-dependencies that are pronged to error
Not to mention that OS distribution that backport from mainline would backport different subset of each driver, let alone they belong to two subtrees with two different maintainers and we don’t want to be in a place where we have to coordinate releases between two subgroups

As such, we selected a much safer path, that operational, development and deployment of two different drivers (netdev vs ib_dev) much easier but exposing the nic as two different Physical functions/devices

Only cost we have is that we need this extra propriety to help user identify which two devices are related hence Samih's comments

#23 lore raw
Quoted 19 lines · Nafea

On 6/6/19, 10:16 AM, "Jakub Kicinski" <[email protected]> wrote:

Hi Samih!

Please don't top post on Linux kernel mailing lists.

On Thu, 6 Jun 2019 10:23:40 +0000, Jubran, Samih wrote:
> As of today there are no flags exposed by ENA NIC device, however, we > are planning to use them in the near future. We want to provide > customers with extra methods to identify (and differentiate) multiple > network interfaces that can be attached to a single VM. Currently, > customers can identify a specific network interface (ENI) only by MAC > address, and later look up this MAC among other multiple ENIs that > they have. In some cases it might not be convenient. Using these > flags will let us inform the customer about ENI`s specific > properties.

Oh no :( You're using private _feature_ flags as labels or tags.

Quoted 8 lines

> It's not finalized, but tentatively it can look like this:
> primary-eni: on /* Differentiate between primary and secondary ENIs */

Did you consider using phys_port_name for this use case?

If the intent is to have those interfaces bonded, even better would be to extend the net_failover module or work on user space ABI for VM failover. That might be a bigger effort, though.

Someone else will address this point?

Quoted 14 lines
Quoted 1 line

has-associated-efa: on /* Will specify ENA device has another associated EFA device */

IDK how your ENA/EFA thing works, but sounds like something that should be solved in the device model.

[NB] ENA is a netDev, EFA is an ib_dev. Both share the same physical network
All previous attempt to make them share at device level leads to a lot of complexity at the OS level, with inter-dependencies that are pronged to error Not to mention that OS distribution that backport from mainline would backport different subset of each driver, let alone they belong to two subtrees with two different maintainers and we don’t want to be in a place where we have to coordinate releases between two subgroups

Quoted 3 lines

As such, we selected a much safer path, that operational, development and deployment of two different drivers (netdev vs ib_dev) much easier but exposing the nic as two different Physical functions/devices

Only cost we have is that we need this extra propriety to help user identify which two devices are related hence Samih's comments

I think you're arguing with a different point than the one I made :) Do you think I said they should use the same PCI device? I haven't.

IIUC you are exposing a "tag" through the feature API on the netdev to indicate that there is another PCI device present on the system?

What I said is if there is a relation between PCI devices the best level to expose this relation at is at the device model level. IOW have some form on "link" in sysfs, not in a random place on a netdev.

Having said that, it's entirely unclear to me what the user scenario is here. You say "which two devices related", yet you only have one bit, so it can indicate that there is another device, not _which_ device is related. Information you can full well get from running lspci 🤷 Do the devices have the same PCI ID/vendor:model?

#24 lore raw

Sent from my iPhone

Quoted 61 lines · Jakub Kicinski
Quoted 21 lines · Nafea

On 6/6/19, 10:16 AM, "Jakub Kicinski" <[email protected]> wrote:

Hi Samih!

Please don't top post on Linux kernel mailing lists.

Quoted 10 lines · Samih

As of today there are no flags exposed by ENA NIC device, however, we are planning to use them in the near future. We want to provide customers with extra methods to identify (and differentiate) multiple network interfaces that can be attached to a single VM. Currently, customers can identify a specific network interface (ENI) only by MAC address, and later look up this MAC among other multiple ENIs that they have. In some cases it might not be convenient. Using these flags will let us inform the customer about ENI`s specific properties.

Oh no :( You're using private _feature_ flags as labels or tags.

Quoted 8 lines
Quoted 2 lines

It's not finalized, but tentatively it can look like this:
primary-eni: on /* Differentiate between primary and secondary ENIs */

Did you consider using phys_port_name for this use case?

If the intent is to have those interfaces bonded, even better would be to extend the net_failover module or work on user space ABI for VM failover. That might be a bigger effort, though.

Someone else will address this point?

Quoted 14 lines
Quoted 1 line

has-associated-efa: on /* Will specify ENA device has another associated EFA device */

IDK how your ENA/EFA thing works, but sounds like something that should be solved in the device model.

[NB] ENA is a netDev, EFA is an ib_dev. Both share the same physical network
All previous attempt to make them share at device level leads to a lot of complexity at the OS level, with inter-dependencies that are pronged to error Not to mention that OS distribution that backport from mainline would backport different subset of each driver, let alone they belong to two subtrees with two different maintainers and we don’t want to be in a place where we have to coordinate releases between two subgroups

Quoted 3 lines

As such, we selected a much safer path, that operational, development and deployment of two different drivers (netdev vs ib_dev) much easier but exposing the nic as two different Physical functions/devices

Only cost we have is that we need this extra propriety to help user identify which two devices are related hence Samih's comments

I think you're arguing with a different point than the one I made :) Do you think I said they should use the same PCI device? I haven't.

IIUC you are exposing a "tag" through the feature API on the netdev to indicate that there is another PCI device present on the system?

What I said is if there is a relation between PCI devices the best level to expose this relation at is at the device model level. IOW have some form on "link" in sysfs, not in a random place on a netdev.

Quoted 5 lines

Having said that, it's entirely unclear to me what the user scenario is here. You say "which two devices related", yet you only have one bit, so it can indicate that there is another device, not _which_ device is related. Information you can full well get from running lspci 🤷 Do the devices have the same PCI ID/vendor:model?

Different model id
Will look into sysfs

#25 lore raw
Quoted 7 lines · Nafea
Quoted 5 lines

Having said that, it's entirely unclear to me what the user scenario is here. You say "which two devices related", yet you only have one bit, so it can indicate that there is another device, not _which_ device is related. Information you can full well get from running lspci 🤷 Do the devices have the same PCI ID/vendor:model?

Different model id

Okay, then you know which one is which. Are there multiple ENAs but one EFA?

Quoted 1 line

Will look into sysfs

I still don't understand what is the problem you're trying to solve, perhaps phys_port_id is the way to go...

The larger point here is that we can't guide you to the right API unless we know what you're trying to achieve. And we don't have the slightest clue of what're trying to achieve if uAPI is forwarded to the device.

Honestly this is worse, and way more basic than I thought, I think 315c28d2b714 ("net: ena: ethtool: add extra properties retrieval via get_priv_flags") needs to be reverted.

#26 lore raw

Sent from my iPhone

Quoted 9 lines · Jakub Kicinski
Quoted 6 lines · Nafea
Quoted 5 lines

Having said that, it's entirely unclear to me what the user scenario is here. You say "which two devices related", yet you only have one bit, so it can indicate that there is another device, not _which_ device is related. Information you can full well get from running lspci 🤷 Do the devices have the same PCI ID/vendor:model?

Quoted 5 lines
Quoted 1 line

Different model id

Okay, then you know which one is which. Are there multiple ENAs but one EFA?

Yes, very possible. Very common

Typical use case that instances have one ena for control plane, one for internet facing , and one 100G ena that also have efa capabilities

Quoted 6 lines
Quoted 1 line

Will look into sysfs

I still don't understand what is the problem you're trying to solve, perhaps phys_port_id is the way to go...

Quoted 8 lines

The larger point here is that we can't guide you to the right API unless we know what you're trying to achieve. And we don't have the slightest clue of what're trying to achieve if uAPI is forwarded to the device.

Honestly this is worse, and way more basic than I thought, I think 315c28d2b714 ("net: ena: ethtool: add extra properties retrieval via get_priv_flags") needs to be reverted.

Let’s not do that until we finish this discussion and explain the various use cases

#27 lore raw
Quoted 10 lines · Nafea
Quoted 10 lines · Jakub Kicinski
Quoted 7 lines · Nafea
Quoted 5 lines

Having said that, it's entirely unclear to me what the user scenario is here. You say "which two devices related", yet you only have one bit, so it can indicate that there is another device, not _which_ device is related. Information you can full well get from running lspci 🤷 Do the devices have the same PCI ID/vendor:model?

Different model id

Quoted 7 lines
Quoted 2 lines

Okay, then you know which one is which. Are there multiple ENAs but one EFA?

Yes, very possible. Very common

Typical use case that instances have one ena for control plane, one for internet facing , and one 100G ena that also have efa capabilities

I see, and those are PCI devices.. Some form of platform data would seem like the best fit to me. There is something called:

/sys/bus/pci/${dbdf}/label

It seems to come from some ACPI table - DSM maybe? I think you can put whatever string you want there 🤔

Quoted 11 lines
Quoted 11 lines
Quoted 1 line

Will look into sysfs

I still don't understand what is the problem you're trying to solve, perhaps phys_port_id is the way to go...

The larger point here is that we can't guide you to the right API unless we know what you're trying to achieve. And we don't have the slightest clue of what're trying to achieve if uAPI is forwarded to the device.

Quoted 5 lines
Quoted 3 lines

Honestly this is worse, and way more basic than I thought, I think 315c28d2b714 ("net: ena: ethtool: add extra properties retrieval via get_priv_flags") needs to be reverted.

Let’s not do that until we finish this discussion and explain the various use cases

Whatever we decide is the right API for tagging interfaces in a virtual environment, it's definitely not going to be private feature flags.

#28 lore raw

Sent from my iPhone

Quoted 8 lines · Jakub Kicinski
Quoted 7 lines
Quoted 2 lines

Okay, then you know which one is which. Are there multiple ENAs but one EFA?

Yes, very possible. Very common

Typical use case that instances have one ena for control plane, one for internet facing , and one 100G ena that also have efa capabilities

Quoted 7 lines

I see, and those are PCI devices.. Some form of platform data would seem like the best fit to me. There is something called:

/sys/bus/pci/${dbdf}/label

It seems to come from some ACPI table - DSM maybe? I think you can put whatever string you want there 🤔

Acpi path won’t work, much of thee interface are hot attached, using native pcie hot plug and acpi won’t be involved.

#29 lore raw
Quoted 12 lines · Nafea
Quoted 11 lines · Jakub Kicinski
Quoted 7 lines
Quoted 2 lines

Okay, then you know which one is which. Are there multiple ENAs but one EFA?

Yes, very possible. Very common

Typical use case that instances have one ena for control plane, one for internet facing , and one 100G ena that also have efa capabilities

I see, and those are PCI devices.. Some form of platform data would seem like the best fit to me. There is something called:

Quoted 7 lines
Quoted 4 lines

/sys/bus/pci/${dbdf}/label

It seems to come from some ACPI table - DSM maybe? I think you can put whatever string you want there 🤔

Acpi path won’t work, much of thee interface are hot attached, using native pcie hot plug and acpi won’t be involved.

Perhaps hotplug break DSM, I won't pretend to be an ACPI expert. So you can find a way to stuff the label into that file from another source. There's also VPD, or custom PCI caps, but platform data generally seems like a better idea.

#30 lore raw
Quoted 18 lines · Jakub Kicinski
Quoted 17 lines · Nafea
Quoted 15 lines · Jakub Kicinski
Quoted 7 lines
Quoted 2 lines

Okay, then you know which one is which. Are there multiple ENAs but one EFA?

Yes, very possible. Very common

Typical use case that instances have one ena for control plane, one for internet facing , and one 100G ena that also have efa capabilities

I see, and those are PCI devices.. Some form of platform data would seem like the best fit to me. There is something called:

/sys/bus/pci/${dbdf}/label

It seems to come from some ACPI table - DSM maybe? I think you can put whatever string you want there 🤔

Quoted 7 lines
Quoted 2 lines

Acpi path won’t work, much of thee interface are hot attached, using native pcie hot plug and acpi won’t be involved.

Perhaps hotplug break DSM, I won't pretend to be an ACPI expert. So you can find a way to stuff the label into that file from another source. There's also VPD, or custom PCI caps, but platform data generally seems like a better idea.

Jiri, do you have any thoughts about using phys_port_name for exposing topology in virtual environments. Perhaps that's the best fit on the netdev side. It's not like we want any other port name in such case, and having it automatically appended to the netdev name may be useful.

#31 lore raw

On 6/7/19, 2:27 PM, "Jakub Kicinski" <[email protected]> wrote:

On Thu, 6 Jun 2019 18:14:16 -0700, Jakub Kicinski wrote:
> On Fri, 7 Jun 2019 01:04:14 +0000, Bshara, Nafea wrote:
> > On Jun 6, 2019, at 4:43 PM, Jakub Kicinski wrote:
> > >>> Okay, then you know which one is which. Are there multiple ENAs but > > >>> one EFA?
> > >>
> > >> Yes, very possible. Very common
> > >>
> > >> Typical use case that instances have one ena for control plane, one > > >> for internet facing , and one 100G ena that also have efa capabilities > > >
> > > I see, and those are PCI devices.. Some form of platform data would > > > seem like the best fit to me. There is something called:
> > >
> > > /sys/bus/pci/${dbdf}/label
> > >
> > > It seems to come from some ACPI table - DSM maybe? I think you can put > > > whatever string you want there 🤔
> >
> > Acpi path won’t work, much of thee interface are hot attached, using > > native pcie hot plug and acpi won’t be involved.
>
> Perhaps hotplug break DSM, I won't pretend to be an ACPI expert. So you > can find a way to stuff the label into that file from another source. > There's also VPD, or custom PCI caps, but platform data generally seems > like a better idea.

Jiri, do you have any thoughts about using phys_port_name for exposing topology in virtual environments. Perhaps that's the best fit on the netdev side. It's not like we want any other port name in such case, and having it automatically appended to the netdev name may be useful.

any preference for that vs /sysfs ?

#32 lore raw
Quoted 29 lines · Nafea

On 6/7/19, 2:27 PM, "Jakub Kicinski" <[email protected]> wrote:

On Thu, 6 Jun 2019 18:14:16 -0700, Jakub Kicinski wrote:
> On Fri, 7 Jun 2019 01:04:14 +0000, Bshara, Nafea wrote:
> > On Jun 6, 2019, at 4:43 PM, Jakub Kicinski wrote:
> > >>> Okay, then you know which one is which. Are there multiple ENAs but > > >>> one EFA?
> > >>
> > >> Yes, very possible. Very common
> > >>
> > >> Typical use case that instances have one ena for control plane, one > > >> for internet facing , and one 100G ena that also have efa capabilities > > >
> > > I see, and those are PCI devices.. Some form of platform data would > > > seem like the best fit to me. There is something called:
> > >
> > > /sys/bus/pci/${dbdf}/label
> > >
> > > It seems to come from some ACPI table - DSM maybe? I think you can put > > > whatever string you want there 🤔
> >
> > Acpi path won’t work, much of thee interface are hot attached, using > > native pcie hot plug and acpi won’t be involved.
>
> Perhaps hotplug break DSM, I won't pretend to be an ACPI expert. So you > can find a way to stuff the label into that file from another source. > There's also VPD, or custom PCI caps, but platform data generally seems > like a better idea.

Quoted 6 lines

Jiri, do you have any thoughts about using phys_port_name for exposing topology in virtual environments. Perhaps that's the best fit on the netdev side. It's not like we want any other port name in such case, and having it automatically appended to the netdev name may be useful.

any preference for that vs /sysfs ?

I think so. For the topology (control, internal, external) I feel like its a reasonable fit, and well supported by udev/systemd. We use it for appending port names of the switch today (p0, p1, p2, etc). Jiri was working on making the core kernel generate those automatically so lets give him a chance to speak up.

For the ENA<>EFA link I think you may still need to resort to sysfs, unless it can be somehow implied by the topology label?