Skip to contentSkip to conversation

Power-optimized RX for Ethernet devices

RFC In review Anatoly Burakov 20 participants 421 messages May 27, 2020 – Jan 29, 2021 Page 4 of 4

Subsystems (3)
lore
Share & tools
lore
#349 lore raw

PATCH v9 4/10 ethdev: add simple power management API

In reply to Liang Ma

Add a simple API to allow getting address of next RX descriptor from the PMD, as well as release notes information.

---

Notes:
v8:
- Rename version map file name.

v7:
- Fixed queue ID validation
- Fixed documentation

v6:
- Rebase on top of latest main
- Ensure the API checks queue ID (Konstantin)
- Removed accidental inclusion of unrelated release notes
v5:
- Bring function format in line with other functions in the file
- Ensure the API is supported by the driver before calling it (Konstantin)

---
 doc/guides/rel_notes/release_20_11.rst |  5 +++++
 lib/librte_ethdev/rte_ethdev.c         | 23 +++++++++++++++++++++
 lib/librte_ethdev/rte_ethdev.h         | 28 ++++++++++++++++++++++++++
 lib/librte_ethdev/rte_ethdev_driver.h  | 28 ++++++++++++++++++++++++++
 lib/librte_ethdev/version.map          |  1 +
 5 files changed, 85 insertions(+)
doc/guides/rel_notes/release_20_11.rst +5 -0
index d8ac359e51..2827a000db 100644
@@ -139,6 +139,11 @@ New Features
   Hairpin Tx part flow rules can be inserted explicitly.
   New API is added to get the hairpin peer ports list.
 
+* **ethdev: add 1 new EXPERIMENTAL API for PMD power management.**
+
+  * ``rte_eth_get_wake_addr()``
+  * add new eth_dev_ops ``get_wake_addr``
+
 * **Updated Broadcom bnxt driver.**
 
   Updated the Broadcom bnxt driver with new features and improvements, including:
lib/librte_ethdev/rte_ethdev.c +23 -0
index b12bb3854d..4f3115fe8e 100644
@@ -5138,6 +5138,29 @@ rte_eth_tx_burst_mode_get(uint16_t port_id, uint16_t queue_id,
 		       dev->dev_ops->tx_burst_mode_get(dev, queue_id, mode));
 }
 
+int
+rte_eth_get_wake_addr(uint16_t port_id, uint16_t queue_id,
+		volatile void **wake_addr, uint64_t *expected, uint64_t *mask,
+		uint8_t *data_sz)
+{
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	dev = &rte_eth_devices[port_id];
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->get_wake_addr, -ENOTSUP);
+
+	if (queue_id >= dev->data->nb_rx_queues) {
+		RTE_ETHDEV_LOG(ERR, "Invalid RX queue_id=%u\n", queue_id);
+		return -EINVAL;
+	}
+
+	return eth_err(port_id,
+		dev->dev_ops->get_wake_addr(dev->data->rx_queues[queue_id],
+			wake_addr, expected, mask, data_sz));
+}
+
 int
 rte_eth_dev_set_mc_addr_list(uint16_t port_id,
 			     struct rte_ether_addr *mc_addr_set,
lib/librte_ethdev/rte_ethdev.h +28 -0
index e341a08817..11559e7bc8 100644
@@ -4364,6 +4364,34 @@ __rte_experimental
 int rte_eth_tx_burst_mode_get(uint16_t port_id, uint16_t queue_id,
 	struct rte_eth_burst_mode *mode);
 
+/**
+ * Retrieve the wake up address for the receive queue.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param queue_id
+ *   The Rx queue on the Ethernet device for which information will be
+ *   retrieved.
+ * @param wake_addr
+ *   The pointer to the address which will be monitored.
+ * @param expected
+ *   The pointer to value to be expected when descriptor is set.
+ * @param mask
+ *   The pointer to comparison bitmask for the expected value.
+ * @param data_sz
+ *   The pointer to data size for the expected value and comparison bitmask.
+ *
+ * @return
+ *   - 0: Success.
+ *   -ENOTSUP: Operation not supported.
+ *   -EINVAL: Invalid parameters.
+ *   -ENODEV: Invalid port ID.
+ */
+__rte_experimental
+int rte_eth_get_wake_addr(uint16_t port_id, uint16_t queue_id,
+		volatile void **wake_addr, uint64_t *expected, uint64_t *mask,
+		uint8_t *data_sz);
+
 /**
  * Retrieve device registers and register attributes (number of registers and
  * register size)
lib/librte_ethdev/rte_ethdev_driver.h +28 -0
index c63b9f7eb7..d7548dfe74 100644
@@ -752,6 +752,32 @@ typedef int (*eth_hairpin_queue_peer_unbind_t)
 	(struct rte_eth_dev *dev, uint16_t cur_queue, uint32_t direction);
 /**< @internal Unbind peer queue from the current queue. */
 
+/**
+ * @internal
+ * Get address of memory location whose contents will change whenever there is
+ * new data to be received on an RX queue.
+ *
+ * @param rxq
+ *   Ethdev queue pointer.
+ * @param tail_desc_addr
+ *   The pointer point to where the address will be stored.
+ * @param expected
+ *   The pointer point to value to be expected when descriptor is set.
+ * @param mask
+ *   The pointer point to comparison bitmask for the expected value.
+ * @param data_sz
+ *   Data size for the expected value (can be 1, 2, 4, or 8 bytes)
+ * @return
+ *   Negative errno value on error, 0 on success.
+ *
+ * @retval 0
+ *   Success
+ * @retval -EINVAL
+ *   Invalid parameters
+ */
+typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *mask, uint8_t *data_sz);
+
 /**
  * @internal A structure containing the functions exported by an Ethernet driver.
  */
@@ -910,6 +936,8 @@ struct eth_dev_ops {
 	/**< Set up the connection between the pair of hairpin queues. */
 	eth_hairpin_queue_peer_unbind_t hairpin_queue_peer_unbind;
 	/**< Disconnect the hairpin queues of a pair from each other. */
+	eth_get_wake_addr_t get_wake_addr;
+	/**< Get next RX queue ring entry address. */
 };
 
 /**
lib/librte_ethdev/version.map +1 -0
index 8ddda2547f..392c273712 100644
@@ -244,6 +244,7 @@ EXPERIMENTAL {
 	rte_flow_get_restore_info;
 	rte_flow_tunnel_action_decap_release;
 	rte_flow_tunnel_item_release;
+	rte_eth_get_wake_addr;
 };
 
 INTERNAL {
Signature · 1 line
2.17.1
#350 lore raw

Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API

24/10/2020 01:06, Liang Ma:

Quoted 6 lines

Add a simple API to allow getting address of next RX descriptor from the PMD, as well as release notes information.

You keep forgetting Cc ethdev maintainers
(it is automatic when using --cc-cmd devtools/get-maintainer.sh). As a result we still don't have any feedback from Ferruh and Andrew. And I believe such API requires having feedback from maintainers of other NIC drivers. I tried to explain this concern already in email and community meetings, but I see no progress.

Below are my comments.

doc/guides/rel_notes/release_20_11.rst New Features · line 142

Quoted 7 lines
doc/guides/rel_notes/release_20_11.rst +1 -0
@@ -139,6 +139,11 @@ New Features
   Hairpin Tx part flow rules can be inserted explicitly.
   New API is added to get the hairpin peer ports list.
 
+* **ethdev: add 1 new EXPERIMENTAL API for PMD power management.**

The guidelines at the top of the file say using past tense. No need to mention it is experimental as any new API.

doc/guides/rel_notes/release_20_11.rst New Features · line 145

Quoted 3 lines
doc/guides/rel_notes/release_20_11.rst +3 -0
+
+  * ``rte_eth_get_wake_addr()``
+  * add new eth_dev_ops ``get_wake_addr``

No need to mention the dev_ops in the release notes. Better to explain what it brings from an user perspective.

lib/librte_ethdev/rte_ethdev.h __rte_experimental · line 4368

Quoted 4 lines
lib/librte_ethdev/rte_ethdev.h +2 -0
+/**
+ * Retrieve the wake up address for the receive queue.

I guess how this function should be used,
but a bit more explanations would not hurt here.

Quoted 8 lines

+ *
+ * @param port_id
+ * The port identifier of the Ethernet device.
+ * @param queue_id
+ * The Rx queue on the Ethernet device for which information will be
+ * retrieved.
+ * @param wake_addr
+ * The pointer to the address which will be monitored.

This function does not make the address monitored, right?

Quoted 2 lines

+ * @param expected
+ * The pointer to value to be expected when descriptor is set.

Not sure we should restrict it to a "descriptor".

Expecting a value or some bits looks too much restrictive. I understand it probably fits well for Intel NICs, but in the general case, we can imagine that any change in a byte array could be a wake up signal.

Quoted 4 lines

+ * @param mask
+ * The pointer to comparison bitmask for the expected value.
+ * @param data_sz
+ * The pointer to data size for the expected value and comparison bitmask.

It is not clear that above 4 parameters are filled by the driver.

Quoted 11 lines

+ *
+ * @return

+ *   - 0: Success.
+ *   -ENOTSUP: Operation not supported.
+ *   -EINVAL: Invalid parameters.
+ *   -ENODEV: Invalid port ID.

+ */

lib/librte_ethdev/rte_ethdev.h +4 -0
+__rte_experimental
+int rte_eth_get_wake_addr(uint16_t port_id, uint16_t queue_id,
+		volatile void **wake_addr, uint64_t *expected, uint64_t *mask,
+		uint8_t *data_sz);

[...]

lib/librte_ethdev/version.map EXPERIMENTAL { · line 248

Quoted 8 lines
lib/librte_ethdev/version.map +1 -0
@@ -244,6 +244,7 @@ EXPERIMENTAL {
 	rte_flow_get_restore_info;
 	rte_flow_tunnel_action_decap_release;
 	rte_flow_tunnel_item_release;
+	rte_eth_get_wake_addr;
 };

Please sort in alphabetical order.

#351 lore raw

Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API

<snip>
thanks for your information.
Sorry for that.
All related maintainer(include other NIC PMD) will be Cced in v10.

Quoted 15 lines

You keep forgetting Cc ethdev maintainers
(it is automatic when using --cc-cmd devtools/get-maintainer.sh). As a result we still don't have any feedback from Ferruh and Andrew. And I believe such API requires having feedback from maintainers of other NIC drivers. I tried to explain this concern already in email and community meetings, but I see no progress.

Below are my comments.

Quoted 6 lines
doc/guides/rel_notes/release_20_11.rst +0 -0
@@ -139,6 +139,11 @@ New Features
   Hairpin Tx part flow rules can be inserted explicitly.
   New API is added to get the hairpin peer ports list.
 
Quoted 4 lines

doc/guides/rel_notes/release_20_11.rst New Features · line 142

Quoted 1 line
doc/guides/rel_notes/release_20_11.rst +1 -0
+* **ethdev: add 1 new EXPERIMENTAL API for PMD power management.**

The guidelines at the top of the file say using past tense. No need to mention it is experimental as any new API.

agree

Quoted 7 lines

doc/guides/rel_notes/release_20_11.rst New Features · line 145

Quoted 4 lines
doc/guides/rel_notes/release_20_11.rst +3 -0
+
+  * ``rte_eth_get_wake_addr()``
+  * add new eth_dev_ops ``get_wake_addr``

No need to mention the dev_ops in the release notes. Better to explain what it brings from an user perspective.

agree

Quoted 8 lines

lib/librte_ethdev/rte_ethdev.h __rte_experimental · line 4368

Quoted 5 lines
lib/librte_ethdev/rte_ethdev.h +2 -0
+/**
+ * Retrieve the wake up address for the receive queue.

I guess how this function should be used,
but a bit more explanations would not hurt here.

agree

Quoted 10 lines
Quoted 8 lines

+ *
+ * @param port_id
+ * The port identifier of the Ethernet device.
+ * @param queue_id
+ * The Rx queue on the Ethernet device for which information will be
+ * retrieved.
+ * @param wake_addr
+ * The pointer to the address which will be monitored.

This function does not make the address monitored, right?

This function only get the target wakeup address. that does not monitor this address.

Quoted 5 lines
Quoted 3 lines

+ * @param expected
+ * The pointer to value to be expected when descriptor is set.

Not sure we should restrict it to a "descriptor".

actully that is not limited to a descriptor, any writeback content should work.

Quoted 5 lines

Expecting a value or some bits looks too much restrictive. I understand it probably fits well for Intel NICs, but in the general case, we can imagine that any change in a byte array could be a wake up signal.

this parameter doesn not limited user how to use it.
In fact, current design can support any bits change within 64 bits content.

Quoted 20 lines
Quoted 5 lines

+ * @param mask
+ * The pointer to comparison bitmask for the expected value.
+ * @param data_sz
+ * The pointer to data size for the expected value and comparison bitmask.

It is not clear that above 4 parameters are filled by the driver.

Quoted 11 lines

+ *
+ * @return

+ *   - 0: Success.
+ *   -ENOTSUP: Operation not supported.
+ *   -EINVAL: Invalid parameters.
+ *   -ENODEV: Invalid port ID.

+ */

lib/librte_ethdev/rte_ethdev.h +4 -0
+__rte_experimental
+int rte_eth_get_wake_addr(uint16_t port_id, uint16_t queue_id,
+		volatile void **wake_addr, uint64_t *expected, uint64_t *mask,
+		uint8_t *data_sz);

[...]

Quoted 10 lines

lib/librte_ethdev/version.map EXPERIMENTAL { · line 248

Quoted 8 lines
lib/librte_ethdev/version.map +1 -0
@@ -244,6 +244,7 @@ EXPERIMENTAL {
 	rte_flow_get_restore_info;
 	rte_flow_tunnel_action_decap_release;
 	rte_flow_tunnel_item_release;
+	rte_eth_get_wake_addr;
 };

Please sort in alphabetical order.

agree will do

Quoted 2 lines
#352 lore raw

Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API

27/10/2020 12:15, Liang, Ma:

Quoted 26 lines
Quoted 7 lines

lib/librte_ethdev/rte_ethdev.h __rte_experimental · line 4368

Quoted 4 lines
lib/librte_ethdev/rte_ethdev.h +2 -0
+/**
+ * Retrieve the wake up address for the receive queue.

I guess how this function should be used,
but a bit more explanations would not hurt here.

agree

Quoted 10 lines
Quoted 8 lines

+ *
+ * @param port_id
+ * The port identifier of the Ethernet device.
+ * @param queue_id
+ * The Rx queue on the Ethernet device for which information will be
+ * retrieved.
+ * @param wake_addr
+ * The pointer to the address which will be monitored.

This function does not make the address monitored, right?

This function only get the target wakeup address. that does not monitor this address.

Quoted 5 lines
Quoted 3 lines

+ * @param expected
+ * The pointer to value to be expected when descriptor is set.

Not sure we should restrict it to a "descriptor".

actully that is not limited to a descriptor, any writeback content should work.

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

Expecting a value or some bits looks too much restrictive. I understand it probably fits well for Intel NICs, but in the general case, we can imagine that any change in a byte array could be a wake up signal.

this parameter doesn not limited user how to use it.
In fact, current design can support any bits change within 64 bits content.

How the driver can specify that any value change should be monitored? I understand that it is only a value/mask pair,
it does not give room for "any value".

#353 lore raw

Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API

Quoted 34 lines

27/10/2020 12:15, Liang, Ma:

Quoted 31 lines
Quoted 7 lines

lib/librte_ethdev/rte_ethdev.h __rte_experimental · line 4368

Quoted 4 lines
lib/librte_ethdev/rte_ethdev.h +2 -0
+/**
+ * Retrieve the wake up address for the receive queue.

I guess how this function should be used,
but a bit more explanations would not hurt here.

agree

Quoted 10 lines
Quoted 8 lines

+ *
+ * @param port_id
+ * The port identifier of the Ethernet device.
+ * @param queue_id
+ * The Rx queue on the Ethernet device for which information will be
+ * retrieved.
+ * @param wake_addr
+ * The pointer to the address which will be monitored.

This function does not make the address monitored, right?

This function only get the target wakeup address. that does not monitor this address.

Quoted 5 lines
Quoted 3 lines

+ * @param expected
+ * The pointer to value to be expected when descriptor is set.

Not sure we should restrict it to a "descriptor".

actully that is not limited to a descriptor, any writeback content should work.

Quoted 5 lines

Expecting a value or some bits looks too much restrictive. I understand it probably fits well for Intel NICs, but in the general case, we can imagine that any change in a byte array could be a wake up signal.

Quoted 6 lines
Quoted 2 lines

this parameter doesn not limited user how to use it.
In fact, current design can support any bits change within 64 bits content.

How the driver can specify that any value change should be monitored? I understand that it is only a value/mask pair,
it does not give room for "any value".

As I can read the code, value=0, mask=0 will provide you with 'any value'. Though it would mean that rte_power_monitor() will *always* go into sleep, so not sure what will be there any practical usage for such case. Konstantin

#354 lore raw

Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API

27/10/2020 18:43, Ananyev, Konstantin:

Quoted 35 lines
Quoted 35 lines

27/10/2020 12:15, Liang, Ma:

Quoted 33 lines
Quoted 7 lines

lib/librte_ethdev/rte_ethdev.h __rte_experimental · line 4368

Quoted 4 lines
lib/librte_ethdev/rte_ethdev.h +2 -0
+/**
+ * Retrieve the wake up address for the receive queue.

I guess how this function should be used,
but a bit more explanations would not hurt here.

agree

Quoted 10 lines
Quoted 8 lines

+ *
+ * @param port_id
+ * The port identifier of the Ethernet device.
+ * @param queue_id
+ * The Rx queue on the Ethernet device for which information will be
+ * retrieved.
+ * @param wake_addr
+ * The pointer to the address which will be monitored.

This function does not make the address monitored, right?

This function only get the target wakeup address. that does not monitor this address.

Quoted 5 lines
Quoted 3 lines

+ * @param expected
+ * The pointer to value to be expected when descriptor is set.

Not sure we should restrict it to a "descriptor".

actully that is not limited to a descriptor, any writeback content should work.

Quoted 5 lines

Expecting a value or some bits looks too much restrictive. I understand it probably fits well for Intel NICs, but in the general case, we can imagine that any change in a byte array could be a wake up signal.

this parameter doesn not limited user how to use it.
In fact, current design can support any bits change within 64 bits content.

Quoted 7 lines
Quoted 3 lines

How the driver can specify that any value change should be monitored? I understand that it is only a value/mask pair,
it does not give room for "any value".

As I can read the code, value=0, mask=0 will provide you with 'any value'. Though it would mean that rte_power_monitor() will *always* go into sleep, so not sure what will be there any practical usage for such case.

I think what is missing is to allow waking up when the value of a byte array is changing, without specifiying any value.

#355 lore raw

Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API

Quoted 56 lines

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Tuesday, October 27, 2020 6:31 PM
To: Ma, Liang J <[email protected]>; Ananyev, Konstantin <[email protected]>
Cc: [email protected]; Burakov, Anatoly <[email protected]>; [email protected]; Zhang, Qi Z <[email protected]>; [email protected]; Xing, Beilei <[email protected]>; Guo, Jia <[email protected]>; Yang, Qiming <[email protected]>; Wang, Haiyue <[email protected]>; Richardson, Bruce <[email protected]>; Hunt, David <[email protected]>; [email protected]; [email protected]; McDaniel, Timothy <[email protected]>; Eads, Gage <[email protected]>; [email protected]; Andrew Rybchenko <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API

27/10/2020 18:43, Ananyev, Konstantin:

Quoted 39 lines
Quoted 38 lines

27/10/2020 12:15, Liang, Ma:

Quoted 26 lines
Quoted 7 lines

lib/librte_ethdev/rte_ethdev.h __rte_experimental · line 4368

Quoted 4 lines
lib/librte_ethdev/rte_ethdev.h +2 -0
+/**
+ * Retrieve the wake up address for the receive queue.

I guess how this function should be used,
but a bit more explanations would not hurt here.

agree

Quoted 10 lines
Quoted 8 lines

+ *
+ * @param port_id
+ * The port identifier of the Ethernet device.
+ * @param queue_id
+ * The Rx queue on the Ethernet device for which information will be
+ * retrieved.
+ * @param wake_addr
+ * The pointer to the address which will be monitored.

This function does not make the address monitored, right?

This function only get the target wakeup address. that does not monitor this address.

Quoted 5 lines
Quoted 3 lines

+ * @param expected
+ * The pointer to value to be expected when descriptor is set.

Not sure we should restrict it to a "descriptor".

actully that is not limited to a descriptor, any writeback content should work.

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

Expecting a value or some bits looks too much restrictive. I understand it probably fits well for Intel NICs, but in the general case, we can imagine that any change in a byte array could be a wake up signal.

this parameter doesn not limited user how to use it.
In fact, current design can support any bits change within 64 bits content.

How the driver can specify that any value change should be monitored? I understand that it is only a value/mask pair,
it does not give room for "any value".

Quoted 6 lines
Quoted 3 lines

As I can read the code, value=0, mask=0 will provide you with 'any value'. Though it would mean that rte_power_monitor() will *always* go into sleep, so not sure what will be there any practical usage for such case.

I think what is missing is to allow waking up when the value of a byte array is changing, without specifiying any value.

I think it will always wakeup on any write to wait_addr.
What you control with value/mask pair - when we should go to sleep. In other words:
ret = rte_eth_get_wake_addr(port, queue, &wait_addr, &value, &mask, ....);

mask==0: always go to sleep, wakeup at any store to wait_addr.
mask!=0: go to sleep only if (*wait_addr & mask) == value, wakeup at any store to wait_addr.

Liang, Anatoly - feel free to correct me here, if I missed something.

#356 lore raw

Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API

Quoted 65 lines · Konstantin
Quoted 57 lines

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Tuesday, October 27, 2020 6:31 PM
To: Ma, Liang J <[email protected]>; Ananyev, Konstantin <[email protected]>
Cc: [email protected]; Burakov, Anatoly <[email protected]>; [email protected]; Zhang, Qi Z <[email protected]>; [email protected]; Xing, Beilei <[email protected]>; Guo, Jia <[email protected]>; Yang, Qiming <[email protected]>; Wang, Haiyue <[email protected]>; Richardson, Bruce <[email protected]>; Hunt, David <[email protected]>; [email protected]; [email protected]; McDaniel, Timothy <[email protected]>; Eads, Gage <[email protected]>; [email protected]; Andrew Rybchenko <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API

27/10/2020 18:43, Ananyev, Konstantin:

Quoted 39 lines
Quoted 38 lines

27/10/2020 12:15, Liang, Ma:

Quoted 26 lines
Quoted 7 lines

lib/librte_ethdev/rte_ethdev.h

Quoted 4 lines
lib/librte_ethdev/rte_ethdev.h +2 -0
+/**
+ * Retrieve the wake up address for the receive queue.

I guess how this function should be used,
but a bit more explanations would not hurt here.

agree

Quoted 10 lines
Quoted 8 lines

+ *
+ * @param port_id
+ * The port identifier of the Ethernet device.
+ * @param queue_id
+ * The Rx queue on the Ethernet device for which information will be
+ * retrieved.
+ * @param wake_addr
+ * The pointer to the address which will be monitored.

This function does not make the address monitored, right?

This function only get the target wakeup address. that does not monitor this address.

Quoted 5 lines
Quoted 3 lines

+ * @param expected
+ * The pointer to value to be expected when descriptor is set.

Not sure we should restrict it to a "descriptor".

actully that is not limited to a descriptor, any writeback content should work.

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

Expecting a value or some bits looks too much restrictive. I understand it probably fits well for Intel NICs, but in the general case, we can imagine that any change in a byte array could be a wake up signal.

this parameter doesn not limited user how to use it.
In fact, current design can support any bits change within 64 bits content.

How the driver can specify that any value change should be monitored? I understand that it is only a value/mask pair,
it does not give room for "any value".

Quoted 6 lines
Quoted 3 lines

As I can read the code, value=0, mask=0 will provide you with 'any value'. Though it would mean that rte_power_monitor() will *always* go into sleep, so not sure what will be there any practical usage for such case.

I think what is missing is to allow waking up when the value of a byte array is changing, without specifiying any value.

Quoted 7 lines

I think it will always wakeup on any write to wait_addr.
What you control with value/mask pair - when we should go to sleep. In other words:
ret = rte_eth_get_wake_addr(port, queue, &wait_addr, &value, &mask, ....);

mask==0: always go to sleep, wakeup at any store to wait_addr.
mask!=0: go to sleep only if (*wait_addr & mask) == value, wakeup at any store to wait_addr.

I did not get this impression on reading it first time. Till you put it this way.
The comment "if the masked value is already matching, abort" stumped me.

Quoted 3 lines

Liang, Anatoly - feel free to correct me here, if I missed something.

#357 lore raw

Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API

In reply to Ananyev, Konstantin
Quoted 69 lines · Konstantin
Quoted 56 lines

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Tuesday, October 27, 2020 6:31 PM
To: Ma, Liang J <[email protected]>; Ananyev, Konstantin <[email protected]>
Cc: [email protected]; Burakov, Anatoly <[email protected]>; [email protected]; Zhang, Qi Z <[email protected]>; [email protected]; Xing, Beilei <[email protected]>; Guo, Jia <[email protected]>; Yang, Qiming <[email protected]>; Wang, Haiyue <[email protected]>; Richardson, Bruce <[email protected]>; Hunt, David <[email protected]>; [email protected]; [email protected]; McDaniel, Timothy <[email protected]>; Eads, Gage <[email protected]>; [email protected]; Andrew Rybchenko <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API

27/10/2020 18:43, Ananyev, Konstantin:

Quoted 39 lines
Quoted 38 lines

27/10/2020 12:15, Liang, Ma:

Quoted 26 lines
Quoted 7 lines

lib/librte_ethdev/rte_ethdev.h

Quoted 4 lines
lib/librte_ethdev/rte_ethdev.h +2 -0
+/**
+ * Retrieve the wake up address for the receive queue.

I guess how this function should be used,
but a bit more explanations would not hurt here.

agree

Quoted 10 lines
Quoted 8 lines

+ *
+ * @param port_id
+ * The port identifier of the Ethernet device.
+ * @param queue_id
+ * The Rx queue on the Ethernet device for which information will be
+ * retrieved.
+ * @param wake_addr
+ * The pointer to the address which will be monitored.

This function does not make the address monitored, right?

This function only get the target wakeup address. that does not monitor this address.

Quoted 5 lines
Quoted 3 lines

+ * @param expected
+ * The pointer to value to be expected when descriptor is set.

Not sure we should restrict it to a "descriptor".

actully that is not limited to a descriptor, any writeback content should work.

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

Expecting a value or some bits looks too much restrictive. I understand it probably fits well for Intel NICs, but in the general case, we can imagine that any change in a byte array could be a wake up signal.

this parameter doesn not limited user how to use it.
In fact, current design can support any bits change within 64 bits content.

How the driver can specify that any value change should be monitored? I understand that it is only a value/mask pair,
it does not give room for "any value".

Quoted 6 lines
Quoted 3 lines

As I can read the code, value=0, mask=0 will provide you with 'any value'. Though it would mean that rte_power_monitor() will *always* go into sleep, so not sure what will be there any practical usage for such case.

I think what is missing is to allow waking up when the value of a byte array is changing, without specifiying any value.

I think it will always wakeup on any write to wait_addr.
What you control with value/mask pair - when we should go to sleep. In other words:
ret = rte_eth_get_wake_addr(port, queue, &wait_addr, &value, &mask, ....);

Quoted 4 lines

mask==0: always go to sleep, wakeup at any store to wait_addr.
mask!=0: go to sleep only if (*wait_addr & mask) == value, wakeup at any store to wait_addr.

Liang, Anatoly - feel free to correct me here, if I missed something.

mask and expected value is used to prevent race condition with NIC. there is a interval between code get target address and issue umwait action. therefore, the mask is a selector of the bits, if the current value & mask already == the expected value just before issue the umwait, that mean that address is no longer useful for umwait because of DMA has happend.
then the code path is abort and back to normal logic.

#358 lore raw

PATCH v9 5/10 power: add PMD power management API and callback

In reply to Liang Ma

Add a simple on/off switch that will enable saving power when no packets are arriving. It is based on counting the number of empty polls and, when the number reaches a certain threshold, entering an architecture-defined optimized power state that will either wait until a TSC timestamp expires, or when packets arrive.

This API mandates a core-to-single-queue mapping (that is, multiple queued per device are supported, but they have to be polled on different cores).

This design is using PMD RX callbacks.

1. UMWAIT/UMONITOR:

When a certain threshold of empty polls is reached, the core will go into a power optimized sleep while waiting on an address of next RX descriptor to be written to.

2. Pause instruction

Instead of move the core into deeper C state, this method uses the pause instruction to avoid busy polling.

3. Frequency scaling
Reuse existing DPDK power library to scale up/down core frequency depending on traffic volume.

---

Notes:
v8:
- Rename version map file name

v7:
- Fixed race condition (Konstantin)
- Slight rework of the structure of monitor code
- Added missing inline for wakeup

v6:
- Added wakeup mechanism for UMWAIT
- Removed memory allocation (everything is now allocated statically)
- Fixed various typos and comments
- Check for invalid queue ID
- Moved release notes to this patch

v5:
- Make error checking more robust
- Prevent initializing scaling if ACPI or PSTATE env wasn't set
- Prevent initializing UMWAIT path if PMD doesn't support get_wake_addr
- Add some debug logging
- Replace x86-specific code path to generic path using the intrinsic check

---
 doc/guides/rel_notes/release_20_11.rst |  11 +
 lib/librte_power/meson.build           |   5 +-
 lib/librte_power/rte_power_pmd_mgmt.c  | 320 +++++++++++++++++++++++++
 lib/librte_power/rte_power_pmd_mgmt.h  |  92 +++++++
 lib/librte_power/version.map           |   4 +
 5 files changed, 430 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_power/rte_power_pmd_mgmt.c
 create mode 100644 lib/librte_power/rte_power_pmd_mgmt.h
doc/guides/rel_notes/release_20_11.rst +11 -0
index 2827a000db..5f32a5da1d 100644
@@ -350,6 +350,17 @@ New Features
   * Replaced ``--scalar`` command-line option with ``--alg=<value>``, to allow
     the user to select the desired classify method.
 
+* **Add PMD power management mechanism**
+
+  3 new Ethernet PMD power management mechanism is added through existing
+  RX callback infrastructure.
+
+  * Add power saving scheme based on UMWAIT instruction (x86 only)
+  * Add power saving scheme based on ``rte_pause()``
+  * Add power saving scheme based on frequency scaling through the power library
+  * Add new EXPERIMENTAL API ``rte_power_pmd_mgmt_queue_enable()``
+  * Add new EXPERIMENTAL API ``rte_power_pmd_mgmt_queue_disable()``
+
 
 Removed Items
 -------------
lib/librte_power/meson.build +3 -2
index 78c031c943..cc3c7a8646 100644
@@ -9,6 +9,7 @@ sources = files('rte_power.c', 'power_acpi_cpufreq.c',
 		'power_kvm_vm.c', 'guest_channel.c',
 		'rte_power_empty_poll.c',
 		'power_pstate_cpufreq.c',
+		'rte_power_pmd_mgmt.c',
 		'power_common.c')
-headers = files('rte_power.h','rte_power_empty_poll.h')
-deps += ['timer']
+headers = files('rte_power.h','rte_power_empty_poll.h','rte_power_pmd_mgmt.h')
+deps += ['timer' ,'ethdev']
lib/librte_power/rte_power_pmd_mgmt.c new file +320 -0

320 lines not on this page · raw message

lib/librte_power/rte_power_pmd_mgmt.h new file +92 -0

92 lines not on this page · raw message

lib/librte_power/version.map +4 -0
index 69ca9af616..3f2f6cd6f6 100644
@@ -34,4 +34,8 @@ EXPERIMENTAL {
 	rte_power_guest_channel_receive_msg;
 	rte_power_poll_stat_fetch;
 	rte_power_poll_stat_update;
+	# added in 20.11
+	rte_power_pmd_mgmt_queue_enable;
+	rte_power_pmd_mgmt_queue_disable;
+
 };
Signature · 1 line
2.17.1
#359 lore raw

PATCH v9 6/10 net/ixgbe: implement power management API

In reply to Liang Ma

Implement support for the power management API by implementing a get_wake_addr function that will return an address of an RX ring's status bit.

---
 drivers/net/ixgbe/ixgbe_ethdev.c |  1 +
 drivers/net/ixgbe/ixgbe_rxtx.c   | 25 +++++++++++++++++++++++++
 drivers/net/ixgbe/ixgbe_rxtx.h   |  2 ++
 3 files changed, 28 insertions(+)
drivers/net/ixgbe/ixgbe_ethdev.c +1 -0
index 00101c2eec..fcc4026372 100644
@@ -588,6 +588,7 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = {
 	.udp_tunnel_port_del  = ixgbe_dev_udp_tunnel_port_del,
 	.tm_ops_get           = ixgbe_tm_ops_get,
 	.tx_done_cleanup      = ixgbe_dev_tx_done_cleanup,
+	.get_wake_addr        = ixgbe_get_wake_addr,
 };
 
 /*
drivers/net/ixgbe/ixgbe_rxtx.c +25 -0
index d1d3baff90..096dff37ba 100644
@@ -1367,6 +1367,31 @@ const uint32_t
 		RTE_PTYPE_INNER_L3_IPV4_EXT | RTE_PTYPE_INNER_L4_UDP,
 };
 
+int ixgbe_get_wake_addr(void *rx_queue, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *mask, uint8_t *data_sz)
+{
+	volatile union ixgbe_adv_rx_desc *rxdp;
+	struct ixgbe_rx_queue *rxq = rx_queue;
+	uint16_t desc;
+
+	desc = rxq->rx_tail;
+	rxdp = &rxq->rx_ring[desc];
+	/* watch for changes in status bit */
+	*tail_desc_addr = &rxdp->wb.upper.status_error;
+
+	/*
+	 * we expect the DD bit to be set to 1 if this descriptor was already
+	 * written to.
+	 */
+	*expected = rte_cpu_to_le_32(IXGBE_RXDADV_STAT_DD);
+	*mask = rte_cpu_to_le_32(IXGBE_RXDADV_STAT_DD);
+
+	/* the registers are 32-bit */
+	*data_sz = 4;
+
+	return 0;
+}
+
 /* @note: fix ixgbe_dev_supported_ptypes_get() if any change here. */
 static inline uint32_t
 ixgbe_rxd_pkt_info_to_pkt_type(uint32_t pkt_info, uint16_t ptype_mask)
drivers/net/ixgbe/ixgbe_rxtx.h +2 -0
index 6d2f7c9da3..1ef0b05e66 100644
@@ -299,5 +299,7 @@ uint64_t ixgbe_get_tx_port_offloads(struct rte_eth_dev *dev);
 uint64_t ixgbe_get_rx_queue_offloads(struct rte_eth_dev *dev);
 uint64_t ixgbe_get_rx_port_offloads(struct rte_eth_dev *dev);
 uint64_t ixgbe_get_tx_queue_offloads(struct rte_eth_dev *dev);
+int ixgbe_get_wake_addr(void *rx_queue, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *mask, uint8_t *data_sz);
 
 #endif /* _IXGBE_RXTX_H_ */
Signature · 1 line
2.17.1
#360 lore raw

PATCH v9 7/10 net/i40e: implement power management API

In reply to Liang Ma

Implement support for the power management API by implementing a get_wake_addr function that will return an address of an RX ring's status bit.

---
 drivers/net/i40e/i40e_ethdev.c |  1 +
 drivers/net/i40e/i40e_rxtx.c   | 26 ++++++++++++++++++++++++++
 drivers/net/i40e/i40e_rxtx.h   |  2 ++
 3 files changed, 29 insertions(+)
drivers/net/i40e/i40e_ethdev.c +1 -0
index 4778aaf299..358a38232b 100644
@@ -513,6 +513,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
 	.mtu_set                      = i40e_dev_mtu_set,
 	.tm_ops_get                   = i40e_tm_ops_get,
 	.tx_done_cleanup              = i40e_tx_done_cleanup,
+	.get_wake_addr	              = i40e_get_wake_addr,
 };
 
 /* store statistics names and its offset in stats structure */
drivers/net/i40e/i40e_rxtx.c +26 -0
index 5df9a9df56..78862fe3a2 100644
@@ -72,6 +72,32 @@
 #define I40E_TX_OFFLOAD_NOTSUP_MASK \
 		(PKT_TX_OFFLOAD_MASK ^ I40E_TX_OFFLOAD_MASK)
 
+int
+i40e_get_wake_addr(void *rx_queue, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *mask, uint8_t *data_sz)
+{
+	struct i40e_rx_queue *rxq = rx_queue;
+	volatile union i40e_rx_desc *rxdp;
+	uint16_t desc;
+
+	desc = rxq->rx_tail;
+	rxdp = &rxq->rx_ring[desc];
+	/* watch for changes in status bit */
+	*tail_desc_addr = &rxdp->wb.qword1.status_error_len;
+
+	/*
+	 * we expect the DD bit to be set to 1 if this descriptor was already
+	 * written to.
+	 */
+	*expected = rte_cpu_to_le_64(1 << I40E_RX_DESC_STATUS_DD_SHIFT);
+	*mask = rte_cpu_to_le_64(1 << I40E_RX_DESC_STATUS_DD_SHIFT);
+
+	/* registers are 64-bit */
+	*data_sz = 8;
+
+	return 0;
+}
+
 static inline void
 i40e_rxd_to_vlan_tci(struct rte_mbuf *mb, volatile union i40e_rx_desc *rxdp)
 {
drivers/net/i40e/i40e_rxtx.h +2 -0
index 57d7b4160b..5826cf1099 100644
@@ -248,6 +248,8 @@ uint16_t i40e_recv_scattered_pkts_vec_avx2(void *rx_queue,
 	struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
 uint16_t i40e_xmit_pkts_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
 	uint16_t nb_pkts);
+int i40e_get_wake_addr(void *rx_queue, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *value, uint8_t *data_sz);
 
 /* For each value it means, datasheet of hardware can tell more details
  *
Signature · 1 line
2.17.1
#361 lore raw

PATCH v9 8/10 net/ice: implement power management API

In reply to Liang Ma

Implement support for the power management API by implementing a get_wake_addr function that will return an address of an RX ring's status bit.

---
 drivers/net/ice/ice_ethdev.c |  1 +
 drivers/net/ice/ice_rxtx.c   | 26 ++++++++++++++++++++++++++
 drivers/net/ice/ice_rxtx.h   |  2 ++
 3 files changed, 29 insertions(+)
drivers/net/ice/ice_ethdev.c +1 -0
index c65125ff32..54f185ad4d 100644
@@ -216,6 +216,7 @@ static const struct eth_dev_ops ice_eth_dev_ops = {
 	.udp_tunnel_port_add          = ice_dev_udp_tunnel_port_add,
 	.udp_tunnel_port_del          = ice_dev_udp_tunnel_port_del,
 	.tx_done_cleanup              = ice_tx_done_cleanup,
+	.get_wake_addr	              = ice_get_wake_addr,
 };
 
 /* store statistics names and its offset in stats structure */
drivers/net/ice/ice_rxtx.c +26 -0
index ee576c362a..fafd6ada62 100644
@@ -26,6 +26,32 @@ uint64_t rte_net_ice_dynflag_proto_xtr_ipv6_flow_mask;
 uint64_t rte_net_ice_dynflag_proto_xtr_tcp_mask;
 uint64_t rte_net_ice_dynflag_proto_xtr_ip_offset_mask;
 
+int ice_get_wake_addr(void *rx_queue, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *mask, uint8_t *data_sz)
+{
+	volatile union ice_rx_flex_desc *rxdp;
+	struct ice_rx_queue *rxq = rx_queue;
+	uint16_t desc;
+
+	desc = rxq->rx_tail;
+	rxdp = &rxq->rx_ring[desc];
+	/* watch for changes in status bit */
+	*tail_desc_addr = &rxdp->wb.status_error0;
+
+	/*
+	 * we expect the DD bit to be set to 1 if this descriptor was already
+	 * written to.
+	 */
+	*expected = rte_cpu_to_le_16(1 << ICE_RX_FLEX_DESC_STATUS0_DD_S);
+	*mask = rte_cpu_to_le_16(1 << ICE_RX_FLEX_DESC_STATUS0_DD_S);
+
+	/* register is 16-bit */
+	*data_sz = 2;
+
+	return 0;
+}
+
+
 static inline uint8_t
 ice_proto_xtr_type_to_rxdid(uint8_t xtr_type)
 {
drivers/net/ice/ice_rxtx.h +2 -0
index 1c23c7541e..7eeb8d467e 100644
@@ -250,6 +250,8 @@ uint16_t ice_xmit_pkts_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
 				uint16_t nb_pkts);
 int ice_fdir_programming(struct ice_pf *pf, struct ice_fltr_desc *fdir_desc);
 int ice_tx_done_cleanup(void *txq, uint32_t free_cnt);
+int ice_get_wake_addr(void *rx_queue, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *mask, uint8_t *data_sz);
 
 #define FDIR_PARSING_ENABLE_PER_QUEUE(ad, on) do { \
 	int i; \
Signature · 1 line
2.17.1
#362 lore raw

PATCH v9 9/10 examples/l3fwd-power: enable PMD power mgmt

In reply to Liang Ma

Add PMD power management feature support to l3fwd-power sample app.

---

Notes:
v8:
- Add return status check for queue enable

v6:
- Fixed typos in documentation

---
 .../sample_app_ug/l3_forward_power_man.rst    | 13 ++++++
 examples/l3fwd-power/main.c                   | 46 ++++++++++++++++++-
 2 files changed, 58 insertions(+), 1 deletion(-)
doc/guides/sample_app_ug/l3_forward_power_man.rst +13 -0
index d7e1dc5813..b10ebe662e 100644
@@ -109,6 +109,8 @@ where,
 
 *   --telemetry:  Telemetry mode.
 
+*   --pmd-mgmt: PMD power management mode.
+
 See :doc:`l3_forward` for details.
 The L3fwd-power example reuses the L3fwd command line options.
 
@@ -455,3 +457,14 @@ reference cycles and accordingly busy rate is set  to either 0% or
 
 The new stats ``empty_poll`` , ``full_poll`` and ``busy_percent`` can be viewed by running the script
 ``/usertools/dpdk-telemetry-client.py`` and selecting the menu option ``Send for global Metrics``.
+
+PMD power management Mode
+-------------------------
+
+The PMD power management  mode support for ``l3fwd-power`` is a standalone mode, in this mode
+``l3fwd-power`` does simple l3fwding along with enable the power saving scheme on specific
+port/queue/lcore. Main purpose for this mode is to demonstrate how to use the PMD power management API.
+
+.. code-block:: console
+
+        ./build/examples/dpdk-l3fwd-power -l 1-3 --  --pmd-mgmt -p 0x0f --config="(0,0,2),(0,1,3)"
examples/l3fwd-power/main.c +45 -1

100 lines not on this page · raw message

Signature · 1 line
2.17.1
#363 lore raw

PATCH v9 10/10 doc: update programmer's guide for power library

In reply to Liang Ma

Update programmer's guide to document PMD power management usage.

---
 doc/guides/prog_guide/power_man.rst | 42 +++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)
doc/guides/prog_guide/power_man.rst +42 -0
index 0a3755a901..38c64d31e4 100644
@@ -192,6 +192,45 @@ User Cases
 ----------
 The mechanism can applied to any device which is based on polling. e.g. NIC, FPGA.
 
+PMD Power Management API
+------------------------
+
+Abstract
+~~~~~~~~
+Existing power management mechanisms require developers to change application
+design or change code to make use of it. The PMD power management API provides a
+convenient alternative by utilizing Ethernet PMD RX callbacks, and triggering
+power saving whenever empty poll count reaches a certain number.
+
+  * UMWAIT/UMONITOR
+
+   This power saving scheme will put the CPU into optimized power state and use
+   the UMWAIT/UMONITOR instructions to monitor the Ethernet PMD RX descriptor
+   address, and wake the CPU up whenever there's new traffic.
+
+  * Pause
+
+   This power saving scheme will use the `rte_pause` function to avoid busy
+   polling.
+
+  * Frequency scaling
+
+   This power saving scheme will use existing power library functionality to
+   scale the core frequency up/down depending on traffic volume.
+
+
+.. note::
+
+   Currently, this power management API is limited to mandatory mapping of 1
+   queue to 1 core (multiple queues are supported, but they must be polled from
+   different cores).
+
+API Overview for PMD Power Management
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* **Queue Enable**: Enable specific power scheme for certain queue/port/core
+
+* **Queue Disable**: Disable power scheme for certain queue/port/core
+
 References
 ----------
 
@@ -200,3 +239,6 @@ References
 
 *   The :doc:`../sample_app_ug/vm_power_management`
     chapter in the :doc:`../sample_app_ug/index` section.
+
+*   The :doc:`../sample_app_ug/rxtx_callbacks`
+    chapter in the :doc:`../sample_app_ug/index` section.
Signature · 1 line
2.17.1
#364 lore raw

Re: [dpdk-dev] [PATCH v9 10/10] doc: update programmer's guide for power library

24/10/2020 01:06, Liang Ma:

Quoted 8 lines

Update programmer's guide to document PMD power management usage.

---
 doc/guides/prog_guide/power_man.rst | 42 +++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)

Why don't you update this doc in the patch 5
adding the code in the power library?

doc/guides/prog_guide/power_man.rst User Cases · line 197

Quoted 9 lines
doc/guides/prog_guide/power_man.rst +3 -0
@@ -192,6 +192,45 @@ User Cases
 ----------
 The mechanism can applied to any device which is based on polling. e.g. NIC, FPGA.
 
+PMD Power Management API
+------------------------
+

Blank line missing below:

doc/guides/prog_guide/power_man.rst User Cases · line 204

Quoted 7 lines
doc/guides/prog_guide/power_man.rst +7 -0
+Abstract
+~~~~~~~~
+Existing power management mechanisms require developers to change application
+design or change code to make use of it. The PMD power management API provides a
+convenient alternative by utilizing Ethernet PMD RX callbacks, and triggering
+power saving whenever empty poll count reaches a certain number.
+

Here you start a list of schemes, without introducing it.

Quoted 15 lines
doc/guides/prog_guide/power_man.rst +15 -0
+  * UMWAIT/UMONITOR
+
+   This power saving scheme will put the CPU into optimized power state and use
+   the UMWAIT/UMONITOR instructions to monitor the Ethernet PMD RX descriptor
+   address, and wake the CPU up whenever there's new traffic.
+
+  * Pause
+
+   This power saving scheme will use the `rte_pause` function to avoid busy
+   polling.
+
+  * Frequency scaling
+
+   This power saving scheme will use existing power library functionality to
+   scale the core frequency up/down depending on traffic volume.

doc/guides/prog_guide/power_man.rst User Cases · line 229

Quoted 10 lines
doc/guides/prog_guide/power_man.rst +10 -0
+
+
+.. note::
+
+   Currently, this power management API is limited to mandatory mapping of 1
+   queue to 1 core (multiple queues are supported, but they must be polled from
+   different cores).
+
+API Overview for PMD Power Management
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Underlining is long enough...
... but a blank line is missing after the title.

Quoted 3 lines
doc/guides/prog_guide/power_man.rst +3 -0
+* **Queue Enable**: Enable specific power scheme for certain queue/port/core
+
+* **Queue Disable**: Disable power scheme for certain queue/port/core
#365 lore raw

Re: [dpdk-dev] [PATCH v9 10/10] doc: update programmer's guide for power library

Hi Thomas,
all is fixed in v10.
Regards
Liang

Quoted 71 lines · Thomas Monjalon

24/10/2020 01:06, Liang Ma:

Quoted 8 lines

Update programmer's guide to document PMD power management usage.

---
 doc/guides/prog_guide/power_man.rst | 42 +++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)

Why don't you update this doc in the patch 5
adding the code in the power library?

doc/guides/prog_guide/power_man.rst User Cases · line 197

Quoted 9 lines
doc/guides/prog_guide/power_man.rst +3 -0
@@ -192,6 +192,45 @@ User Cases
 ----------
 The mechanism can applied to any device which is based on polling. e.g. NIC, FPGA.
 
+PMD Power Management API
+------------------------
+

Blank line missing below:

doc/guides/prog_guide/power_man.rst User Cases · line 204

Quoted 7 lines
doc/guides/prog_guide/power_man.rst +7 -0
+Abstract
+~~~~~~~~
+Existing power management mechanisms require developers to change application
+design or change code to make use of it. The PMD power management API provides a
+convenient alternative by utilizing Ethernet PMD RX callbacks, and triggering
+power saving whenever empty poll count reaches a certain number.
+

Here you start a list of schemes, without introducing it.

Quoted 15 lines
doc/guides/prog_guide/power_man.rst +15 -0
+  * UMWAIT/UMONITOR
+
+   This power saving scheme will put the CPU into optimized power state and use
+   the UMWAIT/UMONITOR instructions to monitor the Ethernet PMD RX descriptor
+   address, and wake the CPU up whenever there's new traffic.
+
+  * Pause
+
+   This power saving scheme will use the `rte_pause` function to avoid busy
+   polling.
+
+  * Frequency scaling
+
+   This power saving scheme will use existing power library functionality to
+   scale the core frequency up/down depending on traffic volume.

doc/guides/prog_guide/power_man.rst User Cases · line 229

Quoted 10 lines
doc/guides/prog_guide/power_man.rst +10 -0
+
+
+.. note::
+
+   Currently, this power management API is limited to mandatory mapping of 1
+   queue to 1 core (multiple queues are supported, but they must be polled from
+   different cores).
+
+API Overview for PMD Power Management
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Underlining is long enough...
... but a blank line is missing after the title.

Quoted 3 lines
doc/guides/prog_guide/power_man.rst +3 -0
+* **Queue Enable**: Enable specific power scheme for certain queue/port/core
+
+* **Queue Disable**: Disable power scheme for certain queue/port/core
#366 lore raw

v10 cover Add PMD power mgmt

In reply to Liang Ma

This patchset proposes a simple API for Ethernet drivers to cause the CPU to enter a power-optimized state while waiting for packets to arrive, along with a set of generic intrinsics that facilitate that. This is achieved through cooperation with the NIC driver that will allow us to know address of wake up event, and wait for writes on it.

On IA, this is achieved through using UMONITOR/UMWAIT instructions. They are used in their raw opcode form because there is no widespread compiler support for them yet. Still, the API is made generic enough to hopefully support other architectures, if they happen to implement similar instructions.

To achieve power savings, there is a very simple mechanism
used: we're counting empty polls, and if a certain threshold is reached, we get the address of next RX ring descriptor from the NIC driver, arm the monitoring hardware, and enter a power-optimized state. We will then wake up when either a timeout happens, or a write happens (or generally whenever CPU feels like waking up - this is platform- specific), and proceed as normal. The empty poll counter is reset whenever we actually get packets, so we only go to sleep when we know nothing is going on. The mechanism is generic which can be used for any write back descriptor.

Why are we putting it into ethdev as opposed to leaving this up to the application? Our customers specifically requested a way to do it wit minimal changes to the application code. The current approach allows to just flip a switch and automatically have power savings.

- Only 1:1 core to queue mapping is supported, meaning that each lcore must at most handle RX on a single queue
- Support 3 type policies. UMWAIT/PAUSE/Frequency_Scale
- Power management is enabled per-queue
- The API doesn't extend to other device types

Liang Ma (9):
eal: add new x86 cpuid support for WAITPKG
eal: add power management intrinsics
eal: add intrinsics support check infrastructure
ethdev: add simple power management API
power: add PMD power management API and callback net/ixgbe: implement power management API net/i40e: implement power management API net/ice: implement power management API examples/l3fwd-power: enable PMD power mgmt

 doc/guides/prog_guide/power_man.rst           |  48 +++
 doc/guides/rel_notes/release_20_11.rst        |  15 +
 .../sample_app_ug/l3_forward_power_man.rst    |  13 +
 drivers/net/i40e/i40e_ethdev.c                |   1 +
 drivers/net/i40e/i40e_rxtx.c                  |  26 ++
 drivers/net/i40e/i40e_rxtx.h                  |   2 +
 drivers/net/ice/ice_ethdev.c                  |   1 +
 drivers/net/ice/ice_rxtx.c                    |  26 ++
 drivers/net/ice/ice_rxtx.h                    |   2 +
 drivers/net/ixgbe/ixgbe_ethdev.c              |   1 +
 drivers/net/ixgbe/ixgbe_rxtx.c                |  25 ++
 drivers/net/ixgbe/ixgbe_rxtx.h                |   2 +
 examples/l3fwd-power/main.c                   |  46 ++-
 lib/librte_eal/arm/include/meson.build        |   1 +
 .../arm/include/rte_power_intrinsics.h        |  60 ++++
 lib/librte_eal/arm/rte_cpuflags.c             |   6 +
 lib/librte_eal/include/generic/rte_cpuflags.h |  26 ++
 .../include/generic/rte_power_intrinsics.h    | 123 +++++++
 lib/librte_eal/include/meson.build            |   1 +
 lib/librte_eal/ppc/include/meson.build        |   1 +
 .../ppc/include/rte_power_intrinsics.h        |  60 ++++
 lib/librte_eal/ppc/rte_cpuflags.c             |   7 +
 lib/librte_eal/version.map                    |   1 +
 lib/librte_eal/x86/include/meson.build        |   1 +
 lib/librte_eal/x86/include/rte_cpuflags.h     |   1 +
 .../x86/include/rte_power_intrinsics.h        | 135 ++++++++
 lib/librte_eal/x86/rte_cpuflags.c             |  14 +
 lib/librte_ethdev/rte_ethdev.c                |  23 ++
 lib/librte_ethdev/rte_ethdev.h                |  33 ++
 lib/librte_ethdev/rte_ethdev_driver.h         |  28 ++
 lib/librte_ethdev/version.map                 |   1 +
 lib/librte_power/meson.build                  |   5 +-
 lib/librte_power/rte_power_pmd_mgmt.c         | 320 ++++++++++++++++++
 lib/librte_power/rte_power_pmd_mgmt.h         |  92 +++++
 lib/librte_power/version.map                  |   4 +
 35 files changed, 1148 insertions(+), 3 deletions(-)
 create mode 100644 lib/librte_eal/arm/include/rte_power_intrinsics.h
 create mode 100644 lib/librte_eal/include/generic/rte_power_intrinsics.h
 create mode 100644 lib/librte_eal/ppc/include/rte_power_intrinsics.h
 create mode 100644 lib/librte_eal/x86/include/rte_power_intrinsics.h
 create mode 100644 lib/librte_power/rte_power_pmd_mgmt.c
 create mode 100644 lib/librte_power/rte_power_pmd_mgmt.h
Signature · 1 line
2.17.1
#367 lore raw

PATCH v10 1/9 eal: add new x86 cpuid support for WAITPKG

Add a new CPUID flag indicating processor support for UMONITOR/UMWAIT and TPAUSE instructions instruction.

---
 lib/librte_eal/x86/include/rte_cpuflags.h | 1 +
 lib/librte_eal/x86/rte_cpuflags.c         | 2 ++
 2 files changed, 3 insertions(+)
lib/librte_eal/x86/include/rte_cpuflags.h +1 -0
index c1d20364d1..848ba9cbfb 100644
@@ -132,6 +132,7 @@ enum rte_cpu_flag_t {
 	RTE_CPUFLAG_MOVDIR64B,              /**< Direct Store Instructions 64B */
 	RTE_CPUFLAG_AVX512VP2INTERSECT,     /**< AVX512 Two Register Intersection */
 
+	RTE_CPUFLAG_WAITPKG,                /**< UMONITOR/UMWAIT/TPAUSE */
 	/* The last item */
 	RTE_CPUFLAG_NUMFLAGS,               /**< This should always be the last! */
 };
lib/librte_eal/x86/rte_cpuflags.c +2 -0
index 30439e7951..0325c4b93b 100644
@@ -110,6 +110,8 @@ const struct feature_entry rte_cpu_feature_table[] = {
 	FEAT_DEF(AVX512F, 0x00000007, 0, RTE_REG_EBX, 16)
 	FEAT_DEF(RDSEED, 0x00000007, 0, RTE_REG_EBX, 18)
 
+	FEAT_DEF(WAITPKG, 0x00000007, 0, RTE_REG_ECX, 5)
+
 	FEAT_DEF(LAHF_SAHF, 0x80000001, 0, RTE_REG_ECX,  0)
 	FEAT_DEF(LZCNT, 0x80000001, 0, RTE_REG_ECX,  4)
 
Signature · 1 line
2.17.1
#368 lore raw

PATCH v10 2/9 eal: add power management intrinsics

In reply to Liang Ma

Add two new power management intrinsics, and provide an implementation in eal/x86 based on UMONITOR/UMWAIT instructions. The instructions are implemented as raw byte opcodes because there is not yet widespread compiler support for these instructions.

The power management instructions provide an architecture-specific function to either wait until a specified TSC timestamp is reached, or optionally wait until either a TSC timestamp is reached or a memory location is written to. The monitor function also provides an optional comparison, to avoid sleeping when the expected write has already happened, and no more writes are expected.

For more details, please refer to Intel(R) 64 and IA-32 Architectures Software Developer's Manual, Volume 2.

---
 lib/librte_eal/arm/include/meson.build        |   1 +
 .../arm/include/rte_power_intrinsics.h        |  60 ++++++++
 .../include/generic/rte_power_intrinsics.h    | 111 ++++++++++++++
 lib/librte_eal/include/meson.build            |   1 +
 lib/librte_eal/ppc/include/meson.build        |   1 +
 .../ppc/include/rte_power_intrinsics.h        |  60 ++++++++
 lib/librte_eal/x86/include/meson.build        |   1 +
 .../x86/include/rte_power_intrinsics.h        | 135 ++++++++++++++++++
 8 files changed, 370 insertions(+)
 create mode 100644 lib/librte_eal/arm/include/rte_power_intrinsics.h
 create mode 100644 lib/librte_eal/include/generic/rte_power_intrinsics.h
 create mode 100644 lib/librte_eal/ppc/include/rte_power_intrinsics.h
 create mode 100644 lib/librte_eal/x86/include/rte_power_intrinsics.h
lib/librte_eal/arm/include/meson.build +1 -0
index 73b750a18f..c6a9f70d73 100644
@@ -20,6 +20,7 @@ arch_headers = files(
 	'rte_pause_32.h',
 	'rte_pause_64.h',
 	'rte_pause.h',
+	'rte_power_intrinsics.h',
 	'rte_prefetch_32.h',
 	'rte_prefetch_64.h',
 	'rte_prefetch.h',
lib/librte_eal/arm/include/rte_power_intrinsics.h new file +60 -0

60 lines not on this page · raw message

lib/librte_eal/include/generic/rte_power_intrinsics.h new file +111 -0

111 lines not on this page · raw message

lib/librte_eal/include/meson.build +1 -0
index cd09027958..3a12e87e19 100644
@@ -60,6 +60,7 @@ generic_headers = files(
 	'generic/rte_memcpy.h',
 	'generic/rte_pause.h',
 	'generic/rte_prefetch.h',
+	'generic/rte_power_intrinsics.h',
 	'generic/rte_rwlock.h',
 	'generic/rte_spinlock.h',
 	'generic/rte_ticketlock.h',
lib/librte_eal/ppc/include/meson.build +1 -0
index ab4bd28092..0873b2aecb 100644
@@ -10,6 +10,7 @@ arch_headers = files(
 	'rte_io.h',
 	'rte_memcpy.h',
 	'rte_pause.h',
+	'rte_power_intrinsics.h',
 	'rte_prefetch.h',
 	'rte_rwlock.h',
 	'rte_spinlock.h',
lib/librte_eal/ppc/include/rte_power_intrinsics.h new file +60 -0

60 lines not on this page · raw message

lib/librte_eal/x86/include/meson.build +1 -0
index f0e998c2fe..494a8142a2 100644
@@ -13,6 +13,7 @@ arch_headers = files(
 	'rte_io.h',
 	'rte_memcpy.h',
 	'rte_prefetch.h',
+	'rte_power_intrinsics.h',
 	'rte_pause.h',
 	'rte_rtm.h',
 	'rte_rwlock.h',
lib/librte_eal/x86/include/rte_power_intrinsics.h new file +135 -0

135 lines not on this page · raw message

Signature · 1 line
2.17.1
#369 lore raw

Re: [dpdk-dev] [PATCH v10 2/9] eal: add power management intrinsics

27/10/2020 15:59, Liang Ma:

Quoted 8 lines
lib/librte_eal/x86/include/rte_power_intrinsics.h new file +8 -0
+static inline uint64_t
+__get_umwait_val(const volatile void *p, const uint8_t sz)
+{
+       switch (sz) {
+       case sizeof(uint8_t):
+               return *(const volatile uint8_t *)p;
+       case sizeof(uint16_t):
+               return *(const volatile uint16_t *)p;

lib/librte_eal/x86/include/rte_power_intrinsics.h line 33

Quoted 10 lines
lib/librte_eal/x86/include/rte_power_intrinsics.h new file +10 -0
+       case sizeof(uint32_t):
+               return *(const volatile uint32_t *)p;
+       case sizeof(uint64_t):
+               return *(const volatile uint64_t *)p;
+       default:
+               /* this is an intrinsic, so we can't have any error handling */
+               RTE_ASSERT(0);
+               return 0;
+       }
+}

This function is going to pollute the namespace.
I will fix it to __rte_power_get_umwait_val().

#370 lore raw

PATCH v10 3/9 eal: add intrinsics support check infrastructure

In reply to Liang Ma

Currently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.

This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.

--

Notes:
v8:
- Rename eal version.map

v6:
- Fix the comments

---
 lib/librte_eal/arm/rte_cpuflags.c             |  6 +++++
 lib/librte_eal/include/generic/rte_cpuflags.h | 26 +++++++++++++++++++
 .../include/generic/rte_power_intrinsics.h    | 12 +++++++++
 lib/librte_eal/ppc/rte_cpuflags.c             |  7 +++++
 lib/librte_eal/version.map                    |  1 +
 lib/librte_eal/x86/rte_cpuflags.c             | 12 +++++++++
 6 files changed, 64 insertions(+)
lib/librte_eal/arm/rte_cpuflags.c +6 -0
index 7b257b7873..e3a53bcece 100644
@@ -151,3 +151,9 @@ rte_cpu_get_flag_name(enum rte_cpu_flag_t feature)
 		return NULL;
 	return rte_cpu_feature_table[feature].name;
 }
+
+void
+rte_cpu_get_intrinsics_support(struct rte_cpu_intrinsics *intrinsics)
+{
+	memset(intrinsics, 0, sizeof(*intrinsics));
+}
lib/librte_eal/include/generic/rte_cpuflags.h +26 -0
index 872f0ebe3e..28a5aecde8 100644
@@ -13,6 +13,32 @@
 #include "rte_common.h"
 #include <errno.h>
 
+#include <rte_compat.h>
+
+/**
+ * Structure used to describe platform-specific intrinsics that may or may not
+ * be supported at runtime.
+ */
+struct rte_cpu_intrinsics {
+	uint32_t power_monitor : 1;
+	/**< indicates support for rte_power_monitor function */
+	uint32_t power_pause : 1;
+	/**< indicates support for rte_power_pause function */
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Check CPU support for various intrinsics at runtime.
+ *
+ * @param intrinsics
+ *     Pointer to a structure to be filled.
+ */
+__rte_experimental
+void
+rte_cpu_get_intrinsics_support(struct rte_cpu_intrinsics *intrinsics);
+
 /**
  * Enumeration of all CPU features supported
  */
lib/librte_eal/include/generic/rte_power_intrinsics.h +12 -0
index fb897d9060..03a326f076 100644
@@ -32,6 +32,10 @@
  * checked against the expected value, and if they match, the entering of
  * optimized power state may be aborted.
  *
+ * @warning It is responsibility of the user to check if this function is
+ *   supported at runtime using `rte_cpu_get_features()` API call. Failing to do
+ *   so may result in an illegal CPU instruction error.
+ *
  * @param p
  *   Address to monitor for changes.
  * @param expected_value
@@ -69,6 +73,10 @@ static inline void rte_power_monitor(const volatile void *p,
  * This call will also lock a spinlock on entering sleep, and release it on
  * waking up the CPU.
  *
+ * @warning It is responsibility of the user to check if this function is
+ *   supported at runtime using `rte_cpu_get_features()` API call. Failing to do
+ *   so may result in an illegal CPU instruction error.
+ *
  * @param p
  *   Address to monitor for changes.
  * @param expected_value
@@ -101,6 +109,10 @@ static inline void rte_power_monitor_sync(const volatile void *p,
  * Enter an architecture-defined optimized power state until a certain TSC
  * timestamp is reached.
  *
+ * @warning It is responsibility of the user to check if this function is
+ *   supported at runtime using `rte_cpu_get_features()` API call. Failing to do
+ *   so may result in an illegal CPU instruction error.
+ *
  * @param tsc_timestamp
  *   Maximum TSC timestamp to wait for. Note that the wait behavior is
  *   architecture-dependent.
lib/librte_eal/ppc/rte_cpuflags.c +7 -0
index 3bb7563ce9..61db5c216d 100644
@@ -8,6 +8,7 @@
 #include <elf.h>
 #include <fcntl.h>
 #include <assert.h>
+#include <string.h>
 #include <unistd.h>
 
 /* Symbolic values for the entries in the auxiliary table */
@@ -108,3 +109,9 @@ rte_cpu_get_flag_name(enum rte_cpu_flag_t feature)
 		return NULL;
 	return rte_cpu_feature_table[feature].name;
 }
+
+void
+rte_cpu_get_intrinsics_support(struct rte_cpu_intrinsics *intrinsics)
+{
+	memset(intrinsics, 0, sizeof(*intrinsics));
+}
lib/librte_eal/version.map +1 -0
index c23ff57ce6..269cdccfd3 100644
@@ -402,6 +402,7 @@ EXPERIMENTAL {
 	rte_service_lcore_may_be_active;
 	rte_vect_get_max_simd_bitwidth;
 	rte_vect_set_max_simd_bitwidth;
+	rte_cpu_get_intrinsics_support;
 };
 
 INTERNAL {
lib/librte_eal/x86/rte_cpuflags.c +12 -0
index 0325c4b93b..a96312ff7f 100644
@@ -7,6 +7,7 @@
 #include <stdio.h>
 #include <errno.h>
 #include <stdint.h>
+#include <string.h>
 
 #include "rte_cpuid.h"
 
@@ -179,3 +180,14 @@ rte_cpu_get_flag_name(enum rte_cpu_flag_t feature)
 		return NULL;
 	return rte_cpu_feature_table[feature].name;
 }
+
+void
+rte_cpu_get_intrinsics_support(struct rte_cpu_intrinsics *intrinsics)
+{
+	memset(intrinsics, 0, sizeof(*intrinsics));
+
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_WAITPKG)) {
+		intrinsics->power_monitor = 1;
+		intrinsics->power_pause = 1;
+	}
+}
Signature · 1 line
2.17.1
#371 lore raw

Re: [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure

Quoted 11 lines · Liang Ma

Currently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.

This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.

Quoted 6 lines

Coming late to the party, it seems crowded...

Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +2 -0
index 872f0ebe3e..28a5aecde8 100644
@@ -13,6 +13,32 @@
 #include "rte_common.h"
 #include <errno.h>
 
+#include <rte_compat.h>
+

lib/librte_eal/include/generic/rte_cpuflags.h line 27

Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +10 -0
+/**
+ * Structure used to describe platform-specific intrinsics that may or may not
+ * be supported at runtime.
+ */
+struct rte_cpu_intrinsics {
+       uint32_t power_monitor : 1;
+       /**< indicates support for rte_power_monitor function */
+       uint32_t power_pause : 1;
+       /**< indicates support for rte_power_pause function */
+};

- The rte_power library is supposed to be built on top of cpuflags. Not the other way around.
Those capabilities should have been kept inside the rte_power_ API and not pollute the cpuflags API.

- All of this should have come as a single patch as the previously introduced API is unusable before.

Quoted 22 lines
lib/librte_eal/include/generic/rte_cpuflags.h +14 -0
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Check CPU support for various intrinsics at runtime.
+ *
+ * @param intrinsics
+ *     Pointer to a structure to be filled.
+ */
+__rte_experimental
+void
+rte_cpu_get_intrinsics_support(struct rte_cpu_intrinsics *intrinsics);
+
 /**
  * Enumeration of all CPU features supported
  */
lib/librte_eal/include/generic/rte_power_intrinsics.h +0 -0
index fb897d9060..03a326f076 100644

lib/librte_eal/include/generic/rte_power_intrinsics.h line 38

Quoted 7 lines
lib/librte_eal/include/generic/rte_power_intrinsics.h +4 -0
@@ -32,6 +32,10 @@
  * checked against the expected value, and if they match, the entering of
  * optimized power state may be aborted.
  *
+ * @warning It is responsibility of the user to check if this function is
+ *   supported at runtime using `rte_cpu_get_features()` API call. Failing to do
+ *   so may result in an illegal CPU instruction error.
+ *

- Reading this API description... what am I supposed to do in my application or driver who wants to use the new rte_power_monitor/rte_power_pause stuff?

I should call rte_cpu_get_features(TOTO) ?
This comment does not give a hint.

I suppose the intent was to refer to the rte_cpu_get_intrinsics_support() thing. This must be fixed.

- Again, I wonder why we are exposing all this stuff. This should be hidden in the rte_power API.

Signature · 1 line
David Marchand
#372 lore raw

Re: [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure

Quoted 90 lines · David Marchand
Quoted 11 lines · Liang Ma

Currently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.

This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.

Quoted 6 lines

Coming late to the party, it seems crowded...

Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +2 -0
index 872f0ebe3e..28a5aecde8 100644
@@ -13,6 +13,32 @@
  #include "rte_common.h"
  #include <errno.h>
 
+#include <rte_compat.h>
+

lib/librte_eal/include/generic/rte_cpuflags.h line 27

Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +10 -0
+/**
+ * Structure used to describe platform-specific intrinsics that may or may not
+ * be supported at runtime.
+ */
+struct rte_cpu_intrinsics {
+       uint32_t power_monitor : 1;
+       /**< indicates support for rte_power_monitor function */
+       uint32_t power_pause : 1;
+       /**< indicates support for rte_power_pause function */
+};

- The rte_power library is supposed to be built on top of cpuflags. Not the other way around.
Those capabilities should have been kept inside the rte_power_ API and not pollute the cpuflags API.

- All of this should have come as a single patch as the previously introduced API is unusable before.

Quoted 22 lines
lib/librte_eal/include/generic/rte_cpuflags.h +14 -0
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Check CPU support for various intrinsics at runtime.
+ *
+ * @param intrinsics
+ *     Pointer to a structure to be filled.
+ */
+__rte_experimental
+void
+rte_cpu_get_intrinsics_support(struct rte_cpu_intrinsics *intrinsics);
+
  /**
   * Enumeration of all CPU features supported
   */
lib/librte_eal/include/generic/rte_power_intrinsics.h +0 -0
index fb897d9060..03a326f076 100644

lib/librte_eal/include/generic/rte_power_intrinsics.h line 38

Quoted 7 lines
lib/librte_eal/include/generic/rte_power_intrinsics.h +4 -0
@@ -32,6 +32,10 @@
   * checked against the expected value, and if they match, the entering of
   * optimized power state may be aborted.
   *
+ * @warning It is responsibility of the user to check if this function is
+ *   supported at runtime using `rte_cpu_get_features()` API call. Failing to do
+ *   so may result in an illegal CPU instruction error.
+ *

- Reading this API description... what am I supposed to do in my application or driver who wants to use the new rte_power_monitor/rte_power_pause stuff?

I should call rte_cpu_get_features(TOTO) ?
This comment does not give a hint.

Quoted 8 lines

I suppose the intent was to refer to the rte_cpu_get_intrinsics_support() thing. This must be fixed.

- Again, I wonder why we are exposing all this stuff. This should be hidden in the rte_power API.

We're exposing all of this here because the intrinsics are *not* part of the power API but rather are generic headers within EAL. Therefore, any infrastructure checking for their support can *not* reside in the power library, but instead has to be in EAL.

The intended usage here is to call this function before calling rte_power_monitor(), such that:

	struct rte_cpu_intrinsics intrinsics;

	rte_cpu_get_intrinsics_support(&intrinsics);

	if (!intrinsics.power_monitor) {
		// rte_power_monitor not supported and cannot be used
		return;
	}
	// proceed with rte_power_monitor usage

Failing to do that will result in either -ENOTSUP on non-x86, or illegal instruction crash on x86 that doesn't have that instruction (because we encode raw opcode).

I've *not* added this to the previous patches because i wanted to get this part reviewed specifically, and not mix it with other IA-specific stuff. It seems that i've succeeded in that goal, as this patch has 4 likes^W acks :)

Signature · 2 lines
Thanks,
Anatoly
#373 lore raw

Re: [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure

30/10/2020 11:09, Burakov, Anatoly:

Quoted 107 lines · David Marchand
Quoted 90 lines · Liang Ma
Quoted 11 lines

Currently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.

This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.

Quoted 6 lines

Coming late to the party, it seems crowded...

Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +2 -0
index 872f0ebe3e..28a5aecde8 100644
@@ -13,6 +13,32 @@
  #include "rte_common.h"
  #include <errno.h>
 
+#include <rte_compat.h>
+

lib/librte_eal/include/generic/rte_cpuflags.h line 27

Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +10 -0
+/**
+ * Structure used to describe platform-specific intrinsics that may or may not
+ * be supported at runtime.
+ */
+struct rte_cpu_intrinsics {
+       uint32_t power_monitor : 1;
+       /**< indicates support for rte_power_monitor function */
+       uint32_t power_pause : 1;
+       /**< indicates support for rte_power_pause function */
+};

- The rte_power library is supposed to be built on top of cpuflags. Not the other way around.
Those capabilities should have been kept inside the rte_power_ API and not pollute the cpuflags API.

- All of this should have come as a single patch as the previously introduced API is unusable before.

Quoted 22 lines
lib/librte_eal/include/generic/rte_cpuflags.h +14 -0
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Check CPU support for various intrinsics at runtime.
+ *
+ * @param intrinsics
+ *     Pointer to a structure to be filled.
+ */
+__rte_experimental
+void
+rte_cpu_get_intrinsics_support(struct rte_cpu_intrinsics *intrinsics);
+
  /**
   * Enumeration of all CPU features supported
   */
lib/librte_eal/include/generic/rte_power_intrinsics.h +0 -0
index fb897d9060..03a326f076 100644

lib/librte_eal/include/generic/rte_power_intrinsics.h line 38

Quoted 7 lines
lib/librte_eal/include/generic/rte_power_intrinsics.h +4 -0
@@ -32,6 +32,10 @@
   * checked against the expected value, and if they match, the entering of
   * optimized power state may be aborted.
   *
+ * @warning It is responsibility of the user to check if this function is
+ *   supported at runtime using `rte_cpu_get_features()` API call. Failing to do
+ *   so may result in an illegal CPU instruction error.
+ *

- Reading this API description... what am I supposed to do in my application or driver who wants to use the new rte_power_monitor/rte_power_pause stuff?

I should call rte_cpu_get_features(TOTO) ?
This comment does not give a hint.

Quoted 7 lines

I suppose the intent was to refer to the rte_cpu_get_intrinsics_support() thing. This must be fixed.

- Again, I wonder why we are exposing all this stuff. This should be hidden in the rte_power API.

We're exposing all of this here because the intrinsics are *not* part of the power API but rather are generic headers within EAL. Therefore, any infrastructure checking for their support can *not* reside in the power library, but instead has to be in EAL.

The intended usage here is to call this function before calling rte_power_monitor(), such that:

Quoted 8 lines
	struct rte_cpu_intrinsics intrinsics;

	rte_cpu_get_intrinsics_support(&intrinsics);

	if (!intrinsics.power_monitor) {
		// rte_power_monitor not supported and cannot be used
		return;
	}

This check could be done inside the rte_power API.

Quoted 10 lines
	// proceed with rte_power_monitor usage

Failing to do that will result in either -ENOTSUP on non-x86, or illegal instruction crash on x86 that doesn't have that instruction (because we encode raw opcode).

I've *not* added this to the previous patches because i wanted to get this part reviewed specifically, and not mix it with other IA-specific stuff. It seems that i've succeeded in that goal, as this patch has 4 likes^W acks :)

You did not explain the need for rte_cpu_get_features() call.

#374 lore raw

Re: [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure

Quoted 110 lines · Thomas Monjalon

30/10/2020 11:09, Burakov, Anatoly:

Quoted 109 lines · David Marchand
Quoted 90 lines · Liang Ma
Quoted 11 lines

Currently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.

This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.

Quoted 6 lines

Coming late to the party, it seems crowded...

Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +2 -0
index 872f0ebe3e..28a5aecde8 100644
@@ -13,6 +13,32 @@
   #include "rte_common.h"
   #include <errno.h>
 
+#include <rte_compat.h>
+

lib/librte_eal/include/generic/rte_cpuflags.h line 27

Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +10 -0
+/**
+ * Structure used to describe platform-specific intrinsics that may or may not
+ * be supported at runtime.
+ */
+struct rte_cpu_intrinsics {
+       uint32_t power_monitor : 1;
+       /**< indicates support for rte_power_monitor function */
+       uint32_t power_pause : 1;
+       /**< indicates support for rte_power_pause function */
+};

- The rte_power library is supposed to be built on top of cpuflags. Not the other way around.
Those capabilities should have been kept inside the rte_power_ API and not pollute the cpuflags API.

- All of this should have come as a single patch as the previously introduced API is unusable before.

Quoted 22 lines
lib/librte_eal/include/generic/rte_cpuflags.h +14 -0
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Check CPU support for various intrinsics at runtime.
+ *
+ * @param intrinsics
+ *     Pointer to a structure to be filled.
+ */
+__rte_experimental
+void
+rte_cpu_get_intrinsics_support(struct rte_cpu_intrinsics *intrinsics);
+
   /**
    * Enumeration of all CPU features supported
    */
lib/librte_eal/include/generic/rte_power_intrinsics.h +0 -0
index fb897d9060..03a326f076 100644

lib/librte_eal/include/generic/rte_power_intrinsics.h line 38

Quoted 7 lines
lib/librte_eal/include/generic/rte_power_intrinsics.h +4 -0
@@ -32,6 +32,10 @@
    * checked against the expected value, and if they match, the entering of
    * optimized power state may be aborted.
    *
+ * @warning It is responsibility of the user to check if this function is
+ *   supported at runtime using `rte_cpu_get_features()` API call. Failing to do
+ *   so may result in an illegal CPU instruction error.
+ *

- Reading this API description... what am I supposed to do in my application or driver who wants to use the new rte_power_monitor/rte_power_pause stuff?

I should call rte_cpu_get_features(TOTO) ?
This comment does not give a hint.

Quoted 7 lines

I suppose the intent was to refer to the rte_cpu_get_intrinsics_support() thing. This must be fixed.

- Again, I wonder why we are exposing all this stuff. This should be hidden in the rte_power API.

We're exposing all of this here because the intrinsics are *not* part of the power API but rather are generic headers within EAL. Therefore, any infrastructure checking for their support can *not* reside in the power library, but instead has to be in EAL.

The intended usage here is to call this function before calling rte_power_monitor(), such that:

	struct rte_cpu_intrinsics intrinsics;
Quoted 8 lines
Quoted 6 lines
	rte_cpu_get_intrinsics_support(&intrinsics);

	if (!intrinsics.power_monitor) {
		// rte_power_monitor not supported and cannot be used
		return;
	}

This check could be done inside the rte_power API.

I'm not quite clear on exactly what you're asking here.

Do you mean the example code above? If so, code like that is already present in the power library, at the callback enable stage.

If you mean to say, i should put this check into the rte_power_monitor intrinsic, then no, i don't think it's a good idea to have this expensive check every time you call rte_power_monitor.

If you mean put this entire infrastructure into the power API - well, that kinda defeats the purpose of both having these intrinsics in generic headers and having a generic CPU feature check infrastructure that was requested of us during the review. We of course can move the intrinsic to the power library and outside of EAL, but then anything that requires UMWAIT will have to depend on the librte_power.

Please clarify exactly what changes you would like to see here, and what is your objection.

Quoted 7 lines
Quoted 7 lines
	// proceed with rte_power_monitor usage

Failing to do that will result in either -ENOTSUP on non-x86, or illegal instruction crash on x86 that doesn't have that instruction (because we encode raw opcode).

Quoted 7 lines
Quoted 4 lines

I've *not* added this to the previous patches because i wanted to get this part reviewed specifically, and not mix it with other IA-specific stuff. It seems that i've succeeded in that goal, as this patch has 4 likes^W acks :)

You did not explain the need for rte_cpu_get_features() call.

Did not explain *where*? Are you suggesting i put things about rte_power_monitor into documentation for rte_cpu_get_intrinsics_support? The documentation for rte_power_monitor already states that one should use rte_cpu_get_intrinsics_support API to check if the rte_power_monitor is supported on current machine. What else is missing?

Signature · 2 lines
Thanks,
Anatoly
#375 lore raw

Re: [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure

30/10/2020 14:37, Burakov, Anatoly:

Quoted 120 lines · Thomas Monjalon
Quoted 110 lines

30/10/2020 11:09, Burakov, Anatoly:

Quoted 109 lines · David Marchand
Quoted 90 lines · Liang Ma
Quoted 11 lines

Currently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.

This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.

Quoted 6 lines

Coming late to the party, it seems crowded...

Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +2 -0
index 872f0ebe3e..28a5aecde8 100644
@@ -13,6 +13,32 @@
   #include "rte_common.h"
   #include <errno.h>
 
+#include <rte_compat.h>
+

lib/librte_eal/include/generic/rte_cpuflags.h line 27

Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +10 -0
+/**
+ * Structure used to describe platform-specific intrinsics that may or may not
+ * be supported at runtime.
+ */
+struct rte_cpu_intrinsics {
+       uint32_t power_monitor : 1;
+       /**< indicates support for rte_power_monitor function */
+       uint32_t power_pause : 1;
+       /**< indicates support for rte_power_pause function */
+};

- The rte_power library is supposed to be built on top of cpuflags. Not the other way around.
Those capabilities should have been kept inside the rte_power_ API and not pollute the cpuflags API.

- All of this should have come as a single patch as the previously introduced API is unusable before.

Quoted 22 lines
lib/librte_eal/include/generic/rte_cpuflags.h +14 -0
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Check CPU support for various intrinsics at runtime.
+ *
+ * @param intrinsics
+ *     Pointer to a structure to be filled.
+ */
+__rte_experimental
+void
+rte_cpu_get_intrinsics_support(struct rte_cpu_intrinsics *intrinsics);
+
   /**
    * Enumeration of all CPU features supported
    */
lib/librte_eal/include/generic/rte_power_intrinsics.h +0 -0
index fb897d9060..03a326f076 100644

lib/librte_eal/include/generic/rte_power_intrinsics.h line 38

Quoted 7 lines
lib/librte_eal/include/generic/rte_power_intrinsics.h +4 -0
@@ -32,6 +32,10 @@
    * checked against the expected value, and if they match, the entering of
    * optimized power state may be aborted.
    *
+ * @warning It is responsibility of the user to check if this function is
+ *   supported at runtime using `rte_cpu_get_features()` API call. Failing to do
+ *   so may result in an illegal CPU instruction error.
+ *

- Reading this API description... what am I supposed to do in my application or driver who wants to use the new rte_power_monitor/rte_power_pause stuff?

I should call rte_cpu_get_features(TOTO) ?
This comment does not give a hint.

Quoted 7 lines

I suppose the intent was to refer to the rte_cpu_get_intrinsics_support() thing. This must be fixed.

- Again, I wonder why we are exposing all this stuff. This should be hidden in the rte_power API.

We're exposing all of this here because the intrinsics are *not* part of the power API but rather are generic headers within EAL. Therefore, any infrastructure checking for their support can *not* reside in the power library, but instead has to be in EAL.

The intended usage here is to call this function before calling rte_power_monitor(), such that:

	struct rte_cpu_intrinsics intrinsics;
Quoted 8 lines
Quoted 6 lines
	rte_cpu_get_intrinsics_support(&intrinsics);

	if (!intrinsics.power_monitor) {
		// rte_power_monitor not supported and cannot be used
		return;
	}

This check could be done inside the rte_power API.

Quoted 8 lines

I'm not quite clear on exactly what you're asking here.

Do you mean the example code above? If so, code like that is already present in the power library, at the callback enable stage.

If you mean to say, i should put this check into the rte_power_monitor intrinsic, then no, i don't think it's a good idea to have this expensive check every time you call rte_power_monitor.

No but it can be done at initialization time.
According to what you say above, it is alread done at callback enable stage. So the app does not need to do the check?

Quoted 6 lines

If you mean put this entire infrastructure into the power API - well, that kinda defeats the purpose of both having these intrinsics in generic headers and having a generic CPU feature check infrastructure that was requested of us during the review. We of course can move the intrinsic to the power library and outside of EAL, but then anything that requires UMWAIT will have to depend on the librte_power.

Yes the intrinsics can be in EAL if usable. But it seems DLB author cannot use what is in EAL.

Quoted 15 lines

Please clarify exactly what changes you would like to see here, and what is your objection.

Quoted 12 lines
Quoted 11 lines
	// proceed with rte_power_monitor usage

Failing to do that will result in either -ENOTSUP on non-x86, or illegal instruction crash on x86 that doesn't have that instruction (because we encode raw opcode).

I've *not* added this to the previous patches because i wanted to get this part reviewed specifically, and not mix it with other IA-specific stuff. It seems that i've succeeded in that goal, as this patch has 4 likes^W acks :)

Quoted 8 lines
Quoted 2 lines

You did not explain the need for rte_cpu_get_features() call.

Did not explain *where*? Are you suggesting i put things about rte_power_monitor into documentation for rte_cpu_get_intrinsics_support? The documentation for rte_power_monitor already states that one should use rte_cpu_get_intrinsics_support API to check if the rte_power_monitor is supported on current machine. What else is missing?

In your example above, you do not call rte_cpu_get_features() which is documented as required in the EAL doc.

#376 lore raw

Re: [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure

Quoted 126 lines · Thomas Monjalon

30/10/2020 14:37, Burakov, Anatoly:

Quoted 125 lines · Thomas Monjalon
Quoted 110 lines

30/10/2020 11:09, Burakov, Anatoly:

Quoted 109 lines · David Marchand
Quoted 90 lines · Liang Ma
Quoted 11 lines

Currently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.

This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.

Quoted 6 lines

Coming late to the party, it seems crowded...

Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +2 -0
index 872f0ebe3e..28a5aecde8 100644
@@ -13,6 +13,32 @@
    #include "rte_common.h"
    #include <errno.h>
 
+#include <rte_compat.h>
+

lib/librte_eal/include/generic/rte_cpuflags.h line 27

Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +10 -0
+/**
+ * Structure used to describe platform-specific intrinsics that may or may not
+ * be supported at runtime.
+ */
+struct rte_cpu_intrinsics {
+       uint32_t power_monitor : 1;
+       /**< indicates support for rte_power_monitor function */
+       uint32_t power_pause : 1;
+       /**< indicates support for rte_power_pause function */
+};

- The rte_power library is supposed to be built on top of cpuflags. Not the other way around.
Those capabilities should have been kept inside the rte_power_ API and not pollute the cpuflags API.

- All of this should have come as a single patch as the previously introduced API is unusable before.

Quoted 22 lines
lib/librte_eal/include/generic/rte_cpuflags.h +14 -0
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Check CPU support for various intrinsics at runtime.
+ *
+ * @param intrinsics
+ *     Pointer to a structure to be filled.
+ */
+__rte_experimental
+void
+rte_cpu_get_intrinsics_support(struct rte_cpu_intrinsics *intrinsics);
+
    /**
     * Enumeration of all CPU features supported
     */
lib/librte_eal/include/generic/rte_power_intrinsics.h +0 -0
index fb897d9060..03a326f076 100644

lib/librte_eal/include/generic/rte_power_intrinsics.h line 38

Quoted 7 lines
lib/librte_eal/include/generic/rte_power_intrinsics.h +4 -0
@@ -32,6 +32,10 @@
     * checked against the expected value, and if they match, the entering of
     * optimized power state may be aborted.
     *
+ * @warning It is responsibility of the user to check if this function is
+ *   supported at runtime using `rte_cpu_get_features()` API call. Failing to do
+ *   so may result in an illegal CPU instruction error.
+ *

- Reading this API description... what am I supposed to do in my application or driver who wants to use the new rte_power_monitor/rte_power_pause stuff?

I should call rte_cpu_get_features(TOTO) ?
This comment does not give a hint.

Quoted 7 lines

I suppose the intent was to refer to the rte_cpu_get_intrinsics_support() thing. This must be fixed.

- Again, I wonder why we are exposing all this stuff. This should be hidden in the rte_power API.

We're exposing all of this here because the intrinsics are *not* part of the power API but rather are generic headers within EAL. Therefore, any infrastructure checking for their support can *not* reside in the power library, but instead has to be in EAL.

The intended usage here is to call this function before calling rte_power_monitor(), such that:

	struct rte_cpu_intrinsics intrinsics;
Quoted 8 lines
Quoted 6 lines
	rte_cpu_get_intrinsics_support(&intrinsics);

	if (!intrinsics.power_monitor) {
		// rte_power_monitor not supported and cannot be used
		return;
	}

This check could be done inside the rte_power API.

I'm not quite clear on exactly what you're asking here.

Do you mean the example code above? If so, code like that is already present in the power library, at the callback enable stage.

Quoted 7 lines
Quoted 3 lines

If you mean to say, i should put this check into the rte_power_monitor intrinsic, then no, i don't think it's a good idea to have this expensive check every time you call rte_power_monitor.

No but it can be done at initialization time.
According to what you say above, it is alread done at callback enable stage. So the app does not need to do the check?

Admittedly it's a bit confusing, but please bear with me.

There are two separate issues at hand: the intrinsic itself, and the calling code. We provide both.

That means, the *calling code* should do the check. In our case, *our* calling code is the callback. However, nothing stops someone else from implementing their own scheme using our intrinsic - in that case, the user will be responsible to check if the intrinsic is supported before using it in their own code, because they won't be using our callback but will be using our intrinsic.

So, we have a check *in our calling code*. But if someone were to use the *intrinsic* directly (like DLB), they would have to add their own checks around the intrinsic usage.

Our power intrinsic is a static inline function. Are you proposing to add some sort of function pointer wrapper and make it an indirect call instead of a static inline function? (or indeed a proper function)

Quoted 10 lines
Quoted 7 lines

If you mean put this entire infrastructure into the power API - well, that kinda defeats the purpose of both having these intrinsics in generic headers and having a generic CPU feature check infrastructure that was requested of us during the review. We of course can move the intrinsic to the power library and outside of EAL, but then anything that requires UMWAIT will have to depend on the librte_power.

Yes the intrinsics can be in EAL if usable. But it seems DLB author cannot use what is in EAL.

I'll let the DLB authors clarify that themselves, but as far as i'm aware, it seems that this is not the case - while their current code wouldn't be able to use these intrinsics by search-and-replace, they will be able to use them with a couple of changes to their code that basically amounted to reimplementation of our intrinsics.

Quoted 19 lines
Quoted 19 lines

Please clarify exactly what changes you would like to see here, and what is your objection.

Quoted 14 lines
Quoted 11 lines
	// proceed with rte_power_monitor usage

Failing to do that will result in either -ENOTSUP on non-x86, or illegal instruction crash on x86 that doesn't have that instruction (because we encode raw opcode).

I've *not* added this to the previous patches because i wanted to get this part reviewed specifically, and not mix it with other IA-specific stuff. It seems that i've succeeded in that goal, as this patch has 4 likes^W acks :)

You did not explain the need for rte_cpu_get_features() call.

Quoted 9 lines
Quoted 5 lines

Did not explain *where*? Are you suggesting i put things about rte_power_monitor into documentation for rte_cpu_get_intrinsics_support? The documentation for rte_power_monitor already states that one should use rte_cpu_get_intrinsics_support API to check if the rte_power_monitor is supported on current machine. What else is missing?

In your example above, you do not call rte_cpu_get_features() which is documented as required in the EAL doc.

I'm not sure i follow. This is unrelated to rte_cpu_get_features call. The rte_cpu_get_features is a CPUID check, and it was decided not to use it because the WAITPKG CPUID flag is only defined for x86 and not for other archs. This new call (rte_cpu_get_intrinsics_support) is non-arch specific, but will have an arch-specific implementation (which happens to use rte_cpu_get_features to detect support for WAITPKG). I have given the example code of how to detect support for rte_power_monitor using this new code, in the code example you just referred to.

Signature · 2 lines
Thanks,
Anatoly
#377 lore raw

Re: [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure

30/10/2020 16:27, Burakov, Anatoly:

Quoted 201 lines · Thomas Monjalon
Quoted 126 lines

30/10/2020 14:37, Burakov, Anatoly:

Quoted 125 lines · Thomas Monjalon
Quoted 110 lines

30/10/2020 11:09, Burakov, Anatoly:

Quoted 109 lines · David Marchand
Quoted 90 lines · Liang Ma
Quoted 11 lines

Currently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.

This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.

Quoted 6 lines

Coming late to the party, it seems crowded...

Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +2 -0
index 872f0ebe3e..28a5aecde8 100644
@@ -13,6 +13,32 @@
    #include "rte_common.h"
    #include <errno.h>
 
+#include <rte_compat.h>
+

lib/librte_eal/include/generic/rte_cpuflags.h line 27

Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +10 -0
+/**
+ * Structure used to describe platform-specific intrinsics that may or may not
+ * be supported at runtime.
+ */
+struct rte_cpu_intrinsics {
+       uint32_t power_monitor : 1;
+       /**< indicates support for rte_power_monitor function */
+       uint32_t power_pause : 1;
+       /**< indicates support for rte_power_pause function */
+};

- The rte_power library is supposed to be built on top of cpuflags. Not the other way around.
Those capabilities should have been kept inside the rte_power_ API and not pollute the cpuflags API.

- All of this should have come as a single patch as the previously introduced API is unusable before.

Quoted 22 lines
lib/librte_eal/include/generic/rte_cpuflags.h +14 -0
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Check CPU support for various intrinsics at runtime.
+ *
+ * @param intrinsics
+ *     Pointer to a structure to be filled.
+ */
+__rte_experimental
+void
+rte_cpu_get_intrinsics_support(struct rte_cpu_intrinsics *intrinsics);
+
    /**
     * Enumeration of all CPU features supported
     */
lib/librte_eal/include/generic/rte_power_intrinsics.h +0 -0
index fb897d9060..03a326f076 100644

lib/librte_eal/include/generic/rte_power_intrinsics.h line 38

Quoted 7 lines
lib/librte_eal/include/generic/rte_power_intrinsics.h +4 -0
@@ -32,6 +32,10 @@
     * checked against the expected value, and if they match, the entering of
     * optimized power state may be aborted.
     *
+ * @warning It is responsibility of the user to check if this function is
+ *   supported at runtime using `rte_cpu_get_features()` API call. Failing to do
+ *   so may result in an illegal CPU instruction error.
+ *

- Reading this API description... what am I supposed to do in my application or driver who wants to use the new rte_power_monitor/rte_power_pause stuff?

I should call rte_cpu_get_features(TOTO) ?
This comment does not give a hint.

Quoted 7 lines

I suppose the intent was to refer to the rte_cpu_get_intrinsics_support() thing. This must be fixed.

- Again, I wonder why we are exposing all this stuff. This should be hidden in the rte_power API.

We're exposing all of this here because the intrinsics are *not* part of the power API but rather are generic headers within EAL. Therefore, any infrastructure checking for their support can *not* reside in the power library, but instead has to be in EAL.

The intended usage here is to call this function before calling rte_power_monitor(), such that:

	struct rte_cpu_intrinsics intrinsics;
Quoted 8 lines
Quoted 6 lines
	rte_cpu_get_intrinsics_support(&intrinsics);

	if (!intrinsics.power_monitor) {
		// rte_power_monitor not supported and cannot be used
		return;
	}

This check could be done inside the rte_power API.

I'm not quite clear on exactly what you're asking here.

Do you mean the example code above? If so, code like that is already present in the power library, at the callback enable stage.

Quoted 7 lines
Quoted 3 lines

If you mean to say, i should put this check into the rte_power_monitor intrinsic, then no, i don't think it's a good idea to have this expensive check every time you call rte_power_monitor.

No but it can be done at initialization time.
According to what you say above, it is alread done at callback enable stage. So the app does not need to do the check?

Admittedly it's a bit confusing, but please bear with me.

There are two separate issues at hand: the intrinsic itself, and the calling code. We provide both.

That means, the *calling code* should do the check. In our case, *our* calling code is the callback. However, nothing stops someone else from implementing their own scheme using our intrinsic - in that case, the user will be responsible to check if the intrinsic is supported before using it in their own code, because they won't be using our callback but will be using our intrinsic.

So, we have a check *in our calling code*. But if someone were to use the *intrinsic* directly (like DLB), they would have to add their own checks around the intrinsic usage.

Our power intrinsic is a static inline function. Are you proposing to add some sort of function pointer wrapper and make it an indirect call instead of a static inline function? (or indeed a proper function)

Quoted 10 lines
Quoted 7 lines

If you mean put this entire infrastructure into the power API - well, that kinda defeats the purpose of both having these intrinsics in generic headers and having a generic CPU feature check infrastructure that was requested of us during the review. We of course can move the intrinsic to the power library and outside of EAL, but then anything that requires UMWAIT will have to depend on the librte_power.

Yes the intrinsics can be in EAL if usable. But it seems DLB author cannot use what is in EAL.

I'll let the DLB authors clarify that themselves, but as far as i'm aware, it seems that this is not the case - while their current code wouldn't be able to use these intrinsics by search-and-replace, they will be able to use them with a couple of changes to their code that basically amounted to reimplementation of our intrinsics.

Quoted 19 lines
Quoted 19 lines

Please clarify exactly what changes you would like to see here, and what is your objection.

Quoted 14 lines
Quoted 11 lines
	// proceed with rte_power_monitor usage

Failing to do that will result in either -ENOTSUP on non-x86, or illegal instruction crash on x86 that doesn't have that instruction (because we encode raw opcode).

I've *not* added this to the previous patches because i wanted to get this part reviewed specifically, and not mix it with other IA-specific stuff. It seems that i've succeeded in that goal, as this patch has 4 likes^W acks :)

You did not explain the need for rte_cpu_get_features() call.

Quoted 9 lines
Quoted 5 lines

Did not explain *where*? Are you suggesting i put things about rte_power_monitor into documentation for rte_cpu_get_intrinsics_support? The documentation for rte_power_monitor already states that one should use rte_cpu_get_intrinsics_support API to check if the rte_power_monitor is supported on current machine. What else is missing?

In your example above, you do not call rte_cpu_get_features() which is documented as required in the EAL doc.

Quoted 8 lines

I'm not sure i follow. This is unrelated to rte_cpu_get_features call. The rte_cpu_get_features is a CPUID check, and it was decided not to use it because the WAITPKG CPUID flag is only defined for x86 and not for other archs. This new call (rte_cpu_get_intrinsics_support) is non-arch specific, but will have an arch-specific implementation (which happens to use rte_cpu_get_features to detect support for WAITPKG). I have given the example code of how to detect support for rte_power_monitor using this new code, in the code example you just referred to.

Please read the API again:
http://git.dpdk.org/dpdk/tree/lib/librte_eal/include/generic/rte_power_intrinsics.h
"
* @warning It is responsibility of the user to check if this function is
* supported at runtime using rte_cpu_get_features() API call.
* Failing to do so may result in an illegal CPU instruction error.
"
Why is it referring to rte_cpu_get_features?

#378 lore raw

Re: [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure

Quoted 211 lines · Thomas Monjalon

30/10/2020 16:27, Burakov, Anatoly:

Quoted 201 lines · Thomas Monjalon
Quoted 126 lines

30/10/2020 14:37, Burakov, Anatoly:

Quoted 125 lines · Thomas Monjalon
Quoted 110 lines

30/10/2020 11:09, Burakov, Anatoly:

Quoted 109 lines · David Marchand
Quoted 90 lines · Liang Ma
Quoted 11 lines

Currently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.

This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.

Quoted 6 lines

Coming late to the party, it seems crowded...

Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +2 -0
index 872f0ebe3e..28a5aecde8 100644
@@ -13,6 +13,32 @@
     #include "rte_common.h"
     #include <errno.h>
 
+#include <rte_compat.h>
+

lib/librte_eal/include/generic/rte_cpuflags.h line 27

Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +10 -0
+/**
+ * Structure used to describe platform-specific intrinsics that may or may not
+ * be supported at runtime.
+ */
+struct rte_cpu_intrinsics {
+       uint32_t power_monitor : 1;
+       /**< indicates support for rte_power_monitor function */
+       uint32_t power_pause : 1;
+       /**< indicates support for rte_power_pause function */
+};

- The rte_power library is supposed to be built on top of cpuflags. Not the other way around.
Those capabilities should have been kept inside the rte_power_ API and not pollute the cpuflags API.

- All of this should have come as a single patch as the previously introduced API is unusable before.

Quoted 22 lines
lib/librte_eal/include/generic/rte_cpuflags.h +14 -0
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Check CPU support for various intrinsics at runtime.
+ *
+ * @param intrinsics
+ *     Pointer to a structure to be filled.
+ */
+__rte_experimental
+void
+rte_cpu_get_intrinsics_support(struct rte_cpu_intrinsics *intrinsics);
+
     /**
      * Enumeration of all CPU features supported
      */
lib/librte_eal/include/generic/rte_power_intrinsics.h +0 -0
index fb897d9060..03a326f076 100644

lib/librte_eal/include/generic/rte_power_intrinsics.h line 38

Quoted 7 lines
lib/librte_eal/include/generic/rte_power_intrinsics.h +4 -0
@@ -32,6 +32,10 @@
      * checked against the expected value, and if they match, the entering of
      * optimized power state may be aborted.
      *
+ * @warning It is responsibility of the user to check if this function is
+ *   supported at runtime using `rte_cpu_get_features()` API call. Failing to do
+ *   so may result in an illegal CPU instruction error.
+ *

- Reading this API description... what am I supposed to do in my application or driver who wants to use the new rte_power_monitor/rte_power_pause stuff?

I should call rte_cpu_get_features(TOTO) ?
This comment does not give a hint.

Quoted 7 lines

I suppose the intent was to refer to the rte_cpu_get_intrinsics_support() thing. This must be fixed.

- Again, I wonder why we are exposing all this stuff. This should be hidden in the rte_power API.

We're exposing all of this here because the intrinsics are *not* part of the power API but rather are generic headers within EAL. Therefore, any infrastructure checking for their support can *not* reside in the power library, but instead has to be in EAL.

The intended usage here is to call this function before calling rte_power_monitor(), such that:

	struct rte_cpu_intrinsics intrinsics;
Quoted 8 lines
Quoted 6 lines
	rte_cpu_get_intrinsics_support(&intrinsics);

	if (!intrinsics.power_monitor) {
		// rte_power_monitor not supported and cannot be used
		return;
	}

This check could be done inside the rte_power API.

I'm not quite clear on exactly what you're asking here.

Do you mean the example code above? If so, code like that is already present in the power library, at the callback enable stage.

Quoted 7 lines
Quoted 3 lines

If you mean to say, i should put this check into the rte_power_monitor intrinsic, then no, i don't think it's a good idea to have this expensive check every time you call rte_power_monitor.

No but it can be done at initialization time.
According to what you say above, it is alread done at callback enable stage. So the app does not need to do the check?

Admittedly it's a bit confusing, but please bear with me.

There are two separate issues at hand: the intrinsic itself, and the calling code. We provide both.

That means, the *calling code* should do the check. In our case, *our* calling code is the callback. However, nothing stops someone else from implementing their own scheme using our intrinsic - in that case, the user will be responsible to check if the intrinsic is supported before using it in their own code, because they won't be using our callback but will be using our intrinsic.

So, we have a check *in our calling code*. But if someone were to use the *intrinsic* directly (like DLB), they would have to add their own checks around the intrinsic usage.

Our power intrinsic is a static inline function. Are you proposing to add some sort of function pointer wrapper and make it an indirect call instead of a static inline function? (or indeed a proper function)

Quoted 10 lines
Quoted 7 lines

If you mean put this entire infrastructure into the power API - well, that kinda defeats the purpose of both having these intrinsics in generic headers and having a generic CPU feature check infrastructure that was requested of us during the review. We of course can move the intrinsic to the power library and outside of EAL, but then anything that requires UMWAIT will have to depend on the librte_power.

Yes the intrinsics can be in EAL if usable. But it seems DLB author cannot use what is in EAL.

I'll let the DLB authors clarify that themselves, but as far as i'm aware, it seems that this is not the case - while their current code wouldn't be able to use these intrinsics by search-and-replace, they will be able to use them with a couple of changes to their code that basically amounted to reimplementation of our intrinsics.

Quoted 19 lines
Quoted 19 lines

Please clarify exactly what changes you would like to see here, and what is your objection.

Quoted 14 lines
Quoted 11 lines
	// proceed with rte_power_monitor usage

Failing to do that will result in either -ENOTSUP on non-x86, or illegal instruction crash on x86 that doesn't have that instruction (because we encode raw opcode).

I've *not* added this to the previous patches because i wanted to get this part reviewed specifically, and not mix it with other IA-specific stuff. It seems that i've succeeded in that goal, as this patch has 4 likes^W acks :)

You did not explain the need for rte_cpu_get_features() call.

Quoted 9 lines
Quoted 5 lines

Did not explain *where*? Are you suggesting i put things about rte_power_monitor into documentation for rte_cpu_get_intrinsics_support? The documentation for rte_power_monitor already states that one should use rte_cpu_get_intrinsics_support API to check if the rte_power_monitor is supported on current machine. What else is missing?

In your example above, you do not call rte_cpu_get_features() which is documented as required in the EAL doc.

Quoted 8 lines

I'm not sure i follow. This is unrelated to rte_cpu_get_features call. The rte_cpu_get_features is a CPUID check, and it was decided not to use it because the WAITPKG CPUID flag is only defined for x86 and not for other archs. This new call (rte_cpu_get_intrinsics_support) is non-arch specific, but will have an arch-specific implementation (which happens to use rte_cpu_get_features to detect support for WAITPKG). I have given the example code of how to detect support for rte_power_monitor using this new code, in the code example you just referred to.

Quoted 9 lines

Please read the API again:
http://git.dpdk.org/dpdk/tree/lib/librte_eal/include/generic/rte_power_intrinsics.h
"
* @warning It is responsibility of the user to check if this function is
* supported at runtime using rte_cpu_get_features() API call.
* Failing to do so may result in an illegal CPU instruction error.
"
Why is it referring to rte_cpu_get_features?

Aw, crap. You're right, it's an artifact of earlier implementation. We'll fix this. Thanks for letting us know!

Signature · 2 lines
Thanks,
Anatoly
#379 lore raw

Re: [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure

30/10/2020 17:36, Burakov, Anatoly:

Quoted 18 lines · Thomas Monjalon
Quoted 17 lines

30/10/2020 16:27, Burakov, Anatoly:

Quoted 16 lines · Thomas Monjalon
Quoted 15 lines

30/10/2020 14:37, Burakov, Anatoly:

Quoted 14 lines · Thomas Monjalon
Quoted 13 lines

30/10/2020 11:09, Burakov, Anatoly:

Quoted 11 lines

The intended usage here is to call this function before calling rte_power_monitor(), such that:

	struct rte_cpu_intrinsics intrinsics;

	rte_cpu_get_intrinsics_support(&intrinsics);

	if (!intrinsics.power_monitor) {
		// rte_power_monitor not supported and cannot be used
		return;
	}

[...]

Quoted 22 lines
Quoted 13 lines
Quoted 4 lines
Quoted 3 lines

In your example above, you do not call rte_cpu_get_features() which is documented as required in the EAL doc.

Quoted 8 lines

I'm not sure i follow. This is unrelated to rte_cpu_get_features call. The rte_cpu_get_features is a CPUID check, and it was decided not to use it because the WAITPKG CPUID flag is only defined for x86 and not for other archs. This new call (rte_cpu_get_intrinsics_support) is non-arch specific, but will have an arch-specific implementation (which happens to use rte_cpu_get_features to detect support for WAITPKG). I have given the example code of how to detect support for rte_power_monitor using this new code, in the code example you just referred to.

Quoted 8 lines

Please read the API again:
http://git.dpdk.org/dpdk/tree/lib/librte_eal/include/generic/rte_power_intrinsics.h
"
* @warning It is responsibility of the user to check if this function is
* supported at runtime using rte_cpu_get_features() API call.
* Failing to do so may result in an illegal CPU instruction error.
"
Why is it referring to rte_cpu_get_features?

Quoted 2 lines

Aw, crap. You're right, it's an artifact of earlier implementation. We'll fix this. Thanks for letting us know!

This is what happens when everybody pushes me to merge a patch that I believe not ready but with a lot of "acked but not reviewed".

The context around this patch series is not good to allow good quality. That's why I think we should not merge any more patch on top of it except DLB PMDs and fixes in this release.

#380 lore raw

PATCH v10 4/9 ethdev: add simple power management API

In reply to Liang Ma

Add a simple API to allow getting address of next RX descriptor from the PMD, as well as release notes information.

---

Notes:
v10:
- Address minor issue on comments and release notes

v8:
- Rename version map file name

v7:
- Fixed race condition (Konstantin)
- Slight rework of the structure of monitor code
- Added missing inline for wakeup

v6:
- Added wakeup mechanism for UMWAIT
- Removed memory allocation (everything is now allocated statically)
- Fixed various typos and comments
- Check for invalid queue ID
- Moved release notes to this patch

v5:
- Make error checking more robust
- Prevent initializing scaling if ACPI or PSTATE env wasn't set
- Prevent initializing UMWAIT path if PMD doesn't support get_wake_addr
- Add some debug logging
- Replace x86-specific code path to generic path using the intrinsic check

---
 doc/guides/rel_notes/release_20_11.rst |  4 ++++
 lib/librte_ethdev/rte_ethdev.c         | 23 ++++++++++++++++++
 lib/librte_ethdev/rte_ethdev.h         | 33 ++++++++++++++++++++++++++
 lib/librte_ethdev/rte_ethdev_driver.h  | 28 ++++++++++++++++++++++
 lib/librte_ethdev/version.map          |  1 +
 5 files changed, 89 insertions(+)
doc/guides/rel_notes/release_20_11.rst +4 -0
index dca8d41eb6..2bdc8f9948 100644
@@ -139,6 +139,10 @@ New Features
   Hairpin Tx part flow rules can be inserted explicitly.
   New API is added to get the hairpin peer ports list.
 
+* **ethdev: added 1 new API for PMD power management.**
+
+  * ``rte_eth_get_wake_addr()`` is added to get the wake up address from device.
+
 * **Updated Broadcom bnxt driver.**
 
   Updated the Broadcom bnxt driver with new features and improvements, including:
lib/librte_ethdev/rte_ethdev.c +23 -0
index b12bb3854d..4f3115fe8e 100644
@@ -5138,6 +5138,29 @@ rte_eth_tx_burst_mode_get(uint16_t port_id, uint16_t queue_id,
 		       dev->dev_ops->tx_burst_mode_get(dev, queue_id, mode));
 }
 
+int
+rte_eth_get_wake_addr(uint16_t port_id, uint16_t queue_id,
+		volatile void **wake_addr, uint64_t *expected, uint64_t *mask,
+		uint8_t *data_sz)
+{
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	dev = &rte_eth_devices[port_id];
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->get_wake_addr, -ENOTSUP);
+
+	if (queue_id >= dev->data->nb_rx_queues) {
+		RTE_ETHDEV_LOG(ERR, "Invalid RX queue_id=%u\n", queue_id);
+		return -EINVAL;
+	}
+
+	return eth_err(port_id,
+		dev->dev_ops->get_wake_addr(dev->data->rx_queues[queue_id],
+			wake_addr, expected, mask, data_sz));
+}
+
 int
 rte_eth_dev_set_mc_addr_list(uint16_t port_id,
 			     struct rte_ether_addr *mc_addr_set,
lib/librte_ethdev/rte_ethdev.h +33 -0
index e341a08817..e522d5d433 100644
@@ -4364,6 +4364,39 @@ __rte_experimental
 int rte_eth_tx_burst_mode_get(uint16_t port_id, uint16_t queue_id,
 	struct rte_eth_burst_mode *mode);
 
+/**
+ * Retrieve the wake up address for device queue or other data structure.
+ * Depend on the HW design, any write back mechanism can be used as a signal
+ * to wake up the processor.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param queue_id
+ *   The Rx queue on the Ethernet device for which information will be
+ *   retrieved.
+ * @param wake_addr
+ *   The pointer to the address which will be monitored and filled by driver.
+ * @param expected
+ *   The pointer to value to be expected when target address is set.
+ *   filled by driver.
+ * @param mask
+ *   The pointer to comparison bitmask for the expected value that is filled by
+ *   driver.
+ * @param data_sz
+ *   The pointer to data size for the expected value and comparison bitmask that
+ *   is filled by driver.
+ *
+ * @return
+ *   - 0: Success.
+ *   -ENOTSUP: Operation not supported.
+ *   -EINVAL: Invalid parameters.
+ *   -ENODEV: Invalid port ID.
+ */
+__rte_experimental
+int rte_eth_get_wake_addr(uint16_t port_id, uint16_t queue_id,
+		volatile void **wake_addr, uint64_t *expected, uint64_t *mask,
+		uint8_t *data_sz);
+
 /**
  * Retrieve device registers and register attributes (number of registers and
  * register size)
lib/librte_ethdev/rte_ethdev_driver.h +28 -0
index c63b9f7eb7..d7548dfe74 100644
@@ -752,6 +752,32 @@ typedef int (*eth_hairpin_queue_peer_unbind_t)
 	(struct rte_eth_dev *dev, uint16_t cur_queue, uint32_t direction);
 /**< @internal Unbind peer queue from the current queue. */
 
+/**
+ * @internal
+ * Get address of memory location whose contents will change whenever there is
+ * new data to be received on an RX queue.
+ *
+ * @param rxq
+ *   Ethdev queue pointer.
+ * @param tail_desc_addr
+ *   The pointer point to where the address will be stored.
+ * @param expected
+ *   The pointer point to value to be expected when descriptor is set.
+ * @param mask
+ *   The pointer point to comparison bitmask for the expected value.
+ * @param data_sz
+ *   Data size for the expected value (can be 1, 2, 4, or 8 bytes)
+ * @return
+ *   Negative errno value on error, 0 on success.
+ *
+ * @retval 0
+ *   Success
+ * @retval -EINVAL
+ *   Invalid parameters
+ */
+typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *mask, uint8_t *data_sz);
+
 /**
  * @internal A structure containing the functions exported by an Ethernet driver.
  */
@@ -910,6 +936,8 @@ struct eth_dev_ops {
 	/**< Set up the connection between the pair of hairpin queues. */
 	eth_hairpin_queue_peer_unbind_t hairpin_queue_peer_unbind;
 	/**< Disconnect the hairpin queues of a pair from each other. */
+	eth_get_wake_addr_t get_wake_addr;
+	/**< Get next RX queue ring entry address. */
 };
 
 /**
lib/librte_ethdev/version.map +1 -0
index 8ddda2547f..f9ce4e3c8d 100644
@@ -235,6 +235,7 @@ EXPERIMENTAL {
 	rte_eth_fec_get_capability;
 	rte_eth_fec_get;
 	rte_eth_fec_set;
+	rte_eth_get_wake_addr;
 	rte_flow_shared_action_create;
 	rte_flow_shared_action_destroy;
 	rte_flow_shared_action_query;
Signature · 1 line
2.17.1
#381 lore raw

PATCH v10 5/9 power: add PMD power management API and callback

In reply to Liang Ma

Add a simple on/off switch that will enable saving power when no packets are arriving. It is based on counting the number of empty polls and, when the number reaches a certain threshold, entering an architecture-defined optimized power state that will either wait until a TSC timestamp expires, or when packets arrive.

This API mandates a core-to-single-queue mapping (that is, multiple queued per device are supported, but they have to be polled on different cores).

This design is using PMD RX callbacks.

1. UMWAIT/UMONITOR:

When a certain threshold of empty polls is reached, the core will go into a power optimized sleep while waiting on an address of next RX descriptor to be written to.

2. Pause instruction

Instead of move the core into deeper C state, this method uses the pause instruction to avoid busy polling.

3. Frequency scaling
Reuse existing DPDK power library to scale up/down core frequency depending on traffic volume.

---

Notes:
v10:
- Updated power library document

v8:
- Rename version map file name

v7:
- Fixed race condition (Konstantin)
- Slight rework of the structure of monitor code
- Added missing inline for wakeup

v6:
- Added wakeup mechanism for UMWAIT
- Removed memory allocation (everything is now allocated statically)
- Fixed various typos and comments
- Check for invalid queue ID
- Moved release notes to this patch

v5:
- Make error checking more robust
- Prevent initializing scaling if ACPI or PSTATE env wasn't set
- Prevent initializing UMWAIT path if PMD doesn't support get_wake_addr
- Add some debug logging
- Replace x86-specific code path to generic path using the intrinsic check

---
 doc/guides/prog_guide/power_man.rst    |  48 ++++
 doc/guides/rel_notes/release_20_11.rst |  11 +
 lib/librte_power/meson.build           |   5 +-
 lib/librte_power/rte_power_pmd_mgmt.c  | 320 +++++++++++++++++++++++++
 lib/librte_power/rte_power_pmd_mgmt.h  |  92 +++++++
 lib/librte_power/version.map           |   4 +
 6 files changed, 478 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_power/rte_power_pmd_mgmt.c
 create mode 100644 lib/librte_power/rte_power_pmd_mgmt.h
doc/guides/prog_guide/power_man.rst +48 -0
index 0a3755a901..1b1064c749 100644
@@ -192,6 +192,51 @@ User Cases
 ----------
 The mechanism can applied to any device which is based on polling. e.g. NIC, FPGA.
 
+PMD Power Management API
+------------------------
+
+Abstract
+~~~~~~~~
+
+Existing power management mechanisms require developers to change application
+design or change code to make use of it. The PMD power management API provides a
+convenient alternative by utilizing Ethernet PMD RX callbacks, and triggering
+power saving whenever empty poll count reaches a certain number.
+
+There are multiple power saving schemes available for developer to choose.
+Although developer can configure each queue with different scheme, however,
+It's strongly recommend to configure the queue within same port with same scheme.
+
+  * UMWAIT/UMONITOR
+
+   This power saving scheme will put the CPU into optimized power state and use
+   the UMWAIT/UMONITOR instructions to monitor the Ethernet PMD RX descriptor
+   address, and wake the CPU up whenever there's new traffic.
+
+  * Pause
+
+   This power saving scheme will use the `rte_pause` function to avoid busy
+   polling.
+
+  * Frequency scaling
+
+   This power saving scheme will use existing power library functionality to
+   scale the core frequency up/down depending on traffic volume.
+
+
+.. note::
+
+   Currently, this power management API is limited to mandatory mapping of 1
+   queue to 1 core (multiple queues are supported, but they must be polled from
+   different cores).
+
+API Overview for PMD Power Management
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* **Queue Enable**: Enable specific power scheme for certain queue/port/core
+
+* **Queue Disable**: Disable power scheme for certain queue/port/core
+
 References
 ----------
 
@@ -200,3 +245,6 @@ References
 
 *   The :doc:`../sample_app_ug/vm_power_management`
     chapter in the :doc:`../sample_app_ug/index` section.
+
+*   The :doc:`../sample_app_ug/rxtx_callbacks`
+    chapter in the :doc:`../sample_app_ug/index` section.
doc/guides/rel_notes/release_20_11.rst +11 -0
index 2bdc8f9948..5fd8c16025 100644
@@ -349,6 +349,17 @@ New Features
   * Replaced ``--scalar`` command-line option with ``--alg=<value>``, to allow
     the user to select the desired classify method.
 
+* **Add PMD power management mechanism**
+
+  3 new Ethernet PMD power management mechanism is added through existing
+  RX callback infrastructure.
+
+  * Add power saving scheme based on UMWAIT instruction (x86 only)
+  * Add power saving scheme based on ``rte_pause()``
+  * Add power saving scheme based on frequency scaling through the power library
+  * Add new EXPERIMENTAL API ``rte_power_pmd_mgmt_queue_enable()``
+  * Add new EXPERIMENTAL API ``rte_power_pmd_mgmt_queue_disable()``
+
 
 Removed Items
 -------------
lib/librte_power/meson.build +3 -2
index 78c031c943..cc3c7a8646 100644
@@ -9,6 +9,7 @@ sources = files('rte_power.c', 'power_acpi_cpufreq.c',
 		'power_kvm_vm.c', 'guest_channel.c',
 		'rte_power_empty_poll.c',
 		'power_pstate_cpufreq.c',
+		'rte_power_pmd_mgmt.c',
 		'power_common.c')
-headers = files('rte_power.h','rte_power_empty_poll.h')
-deps += ['timer']
+headers = files('rte_power.h','rte_power_empty_poll.h','rte_power_pmd_mgmt.h')
+deps += ['timer' ,'ethdev']
lib/librte_power/rte_power_pmd_mgmt.c new file +320 -0

320 lines not on this page · raw message

lib/librte_power/rte_power_pmd_mgmt.h new file +92 -0

92 lines not on this page · raw message

lib/librte_power/version.map +4 -0
index 69ca9af616..3f2f6cd6f6 100644
@@ -34,4 +34,8 @@ EXPERIMENTAL {
 	rte_power_guest_channel_receive_msg;
 	rte_power_poll_stat_fetch;
 	rte_power_poll_stat_update;
+	# added in 20.11
+	rte_power_pmd_mgmt_queue_enable;
+	rte_power_pmd_mgmt_queue_disable;
+
 };
Signature · 1 line
2.17.1
#382 lore raw

PATCH v10 6/9 net/ixgbe: implement power management API

In reply to Liang Ma

Implement support for the power management API by implementing a get_wake_addr function that will return an address of an RX ring's status bit.

---
 drivers/net/ixgbe/ixgbe_ethdev.c |  1 +
 drivers/net/ixgbe/ixgbe_rxtx.c   | 25 +++++++++++++++++++++++++
 drivers/net/ixgbe/ixgbe_rxtx.h   |  2 ++
 3 files changed, 28 insertions(+)
drivers/net/ixgbe/ixgbe_ethdev.c +1 -0
index 00101c2eec..fcc4026372 100644
@@ -588,6 +588,7 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = {
 	.udp_tunnel_port_del  = ixgbe_dev_udp_tunnel_port_del,
 	.tm_ops_get           = ixgbe_tm_ops_get,
 	.tx_done_cleanup      = ixgbe_dev_tx_done_cleanup,
+	.get_wake_addr        = ixgbe_get_wake_addr,
 };
 
 /*
drivers/net/ixgbe/ixgbe_rxtx.c +25 -0
index 5f19972031..305822836d 100644
@@ -1367,6 +1367,31 @@ const uint32_t
 		RTE_PTYPE_INNER_L3_IPV4_EXT | RTE_PTYPE_INNER_L4_UDP,
 };
 
+int ixgbe_get_wake_addr(void *rx_queue, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *mask, uint8_t *data_sz)
+{
+	volatile union ixgbe_adv_rx_desc *rxdp;
+	struct ixgbe_rx_queue *rxq = rx_queue;
+	uint16_t desc;
+
+	desc = rxq->rx_tail;
+	rxdp = &rxq->rx_ring[desc];
+	/* watch for changes in status bit */
+	*tail_desc_addr = &rxdp->wb.upper.status_error;
+
+	/*
+	 * we expect the DD bit to be set to 1 if this descriptor was already
+	 * written to.
+	 */
+	*expected = rte_cpu_to_le_32(IXGBE_RXDADV_STAT_DD);
+	*mask = rte_cpu_to_le_32(IXGBE_RXDADV_STAT_DD);
+
+	/* the registers are 32-bit */
+	*data_sz = 4;
+
+	return 0;
+}
+
 /* @note: fix ixgbe_dev_supported_ptypes_get() if any change here. */
 static inline uint32_t
 ixgbe_rxd_pkt_info_to_pkt_type(uint32_t pkt_info, uint16_t ptype_mask)
drivers/net/ixgbe/ixgbe_rxtx.h +2 -0
index 6d2f7c9da3..1ef0b05e66 100644
@@ -299,5 +299,7 @@ uint64_t ixgbe_get_tx_port_offloads(struct rte_eth_dev *dev);
 uint64_t ixgbe_get_rx_queue_offloads(struct rte_eth_dev *dev);
 uint64_t ixgbe_get_rx_port_offloads(struct rte_eth_dev *dev);
 uint64_t ixgbe_get_tx_queue_offloads(struct rte_eth_dev *dev);
+int ixgbe_get_wake_addr(void *rx_queue, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *mask, uint8_t *data_sz);
 
 #endif /* _IXGBE_RXTX_H_ */
Signature · 1 line
2.17.1
#383 lore raw

PATCH v10 7/9 net/i40e: implement power management API

In reply to Liang Ma

Implement support for the power management API by implementing a get_wake_addr function that will return an address of an RX ring's status bit.

---
 drivers/net/i40e/i40e_ethdev.c |  1 +
 drivers/net/i40e/i40e_rxtx.c   | 26 ++++++++++++++++++++++++++
 drivers/net/i40e/i40e_rxtx.h   |  2 ++
 3 files changed, 29 insertions(+)
drivers/net/i40e/i40e_ethdev.c +1 -0
index 4778aaf299..358a38232b 100644
@@ -513,6 +513,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
 	.mtu_set                      = i40e_dev_mtu_set,
 	.tm_ops_get                   = i40e_tm_ops_get,
 	.tx_done_cleanup              = i40e_tx_done_cleanup,
+	.get_wake_addr	              = i40e_get_wake_addr,
 };
 
 /* store statistics names and its offset in stats structure */
drivers/net/i40e/i40e_rxtx.c +26 -0
index 5df9a9df56..78862fe3a2 100644
@@ -72,6 +72,32 @@
 #define I40E_TX_OFFLOAD_NOTSUP_MASK \
 		(PKT_TX_OFFLOAD_MASK ^ I40E_TX_OFFLOAD_MASK)
 
+int
+i40e_get_wake_addr(void *rx_queue, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *mask, uint8_t *data_sz)
+{
+	struct i40e_rx_queue *rxq = rx_queue;
+	volatile union i40e_rx_desc *rxdp;
+	uint16_t desc;
+
+	desc = rxq->rx_tail;
+	rxdp = &rxq->rx_ring[desc];
+	/* watch for changes in status bit */
+	*tail_desc_addr = &rxdp->wb.qword1.status_error_len;
+
+	/*
+	 * we expect the DD bit to be set to 1 if this descriptor was already
+	 * written to.
+	 */
+	*expected = rte_cpu_to_le_64(1 << I40E_RX_DESC_STATUS_DD_SHIFT);
+	*mask = rte_cpu_to_le_64(1 << I40E_RX_DESC_STATUS_DD_SHIFT);
+
+	/* registers are 64-bit */
+	*data_sz = 8;
+
+	return 0;
+}
+
 static inline void
 i40e_rxd_to_vlan_tci(struct rte_mbuf *mb, volatile union i40e_rx_desc *rxdp)
 {
drivers/net/i40e/i40e_rxtx.h +2 -0
index 57d7b4160b..5826cf1099 100644
@@ -248,6 +248,8 @@ uint16_t i40e_recv_scattered_pkts_vec_avx2(void *rx_queue,
 	struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
 uint16_t i40e_xmit_pkts_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
 	uint16_t nb_pkts);
+int i40e_get_wake_addr(void *rx_queue, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *value, uint8_t *data_sz);
 
 /* For each value it means, datasheet of hardware can tell more details
  *
Signature · 1 line
2.17.1
#384 lore raw

PATCH v10 8/9 net/ice: implement power management API

In reply to Liang Ma

Implement support for the power management API by implementing a get_wake_addr function that will return an address of an RX ring's status bit.

---
 drivers/net/ice/ice_ethdev.c |  1 +
 drivers/net/ice/ice_rxtx.c   | 26 ++++++++++++++++++++++++++
 drivers/net/ice/ice_rxtx.h   |  2 ++
 3 files changed, 29 insertions(+)
drivers/net/ice/ice_ethdev.c +1 -0
index c65125ff32..54f185ad4d 100644
@@ -216,6 +216,7 @@ static const struct eth_dev_ops ice_eth_dev_ops = {
 	.udp_tunnel_port_add          = ice_dev_udp_tunnel_port_add,
 	.udp_tunnel_port_del          = ice_dev_udp_tunnel_port_del,
 	.tx_done_cleanup              = ice_tx_done_cleanup,
+	.get_wake_addr	              = ice_get_wake_addr,
 };
 
 /* store statistics names and its offset in stats structure */
drivers/net/ice/ice_rxtx.c +26 -0
index ee576c362a..fafd6ada62 100644
@@ -26,6 +26,32 @@ uint64_t rte_net_ice_dynflag_proto_xtr_ipv6_flow_mask;
 uint64_t rte_net_ice_dynflag_proto_xtr_tcp_mask;
 uint64_t rte_net_ice_dynflag_proto_xtr_ip_offset_mask;
 
+int ice_get_wake_addr(void *rx_queue, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *mask, uint8_t *data_sz)
+{
+	volatile union ice_rx_flex_desc *rxdp;
+	struct ice_rx_queue *rxq = rx_queue;
+	uint16_t desc;
+
+	desc = rxq->rx_tail;
+	rxdp = &rxq->rx_ring[desc];
+	/* watch for changes in status bit */
+	*tail_desc_addr = &rxdp->wb.status_error0;
+
+	/*
+	 * we expect the DD bit to be set to 1 if this descriptor was already
+	 * written to.
+	 */
+	*expected = rte_cpu_to_le_16(1 << ICE_RX_FLEX_DESC_STATUS0_DD_S);
+	*mask = rte_cpu_to_le_16(1 << ICE_RX_FLEX_DESC_STATUS0_DD_S);
+
+	/* register is 16-bit */
+	*data_sz = 2;
+
+	return 0;
+}
+
+
 static inline uint8_t
 ice_proto_xtr_type_to_rxdid(uint8_t xtr_type)
 {
drivers/net/ice/ice_rxtx.h +2 -0
index 1c23c7541e..7eeb8d467e 100644
@@ -250,6 +250,8 @@ uint16_t ice_xmit_pkts_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
 				uint16_t nb_pkts);
 int ice_fdir_programming(struct ice_pf *pf, struct ice_fltr_desc *fdir_desc);
 int ice_tx_done_cleanup(void *txq, uint32_t free_cnt);
+int ice_get_wake_addr(void *rx_queue, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *mask, uint8_t *data_sz);
 
 #define FDIR_PARSING_ENABLE_PER_QUEUE(ad, on) do { \
 	int i; \
Signature · 1 line
2.17.1
#385 lore raw

PATCH v10 9/9 examples/l3fwd-power: enable PMD power mgmt

In reply to Liang Ma

Add PMD power management feature support to l3fwd-power sample app.

---

Notes:
v8:
- Add return status check for queue enable

v6:
- Fixed typos in documentation

---
 .../sample_app_ug/l3_forward_power_man.rst    | 13 ++++++
 examples/l3fwd-power/main.c                   | 46 ++++++++++++++++++-
 2 files changed, 58 insertions(+), 1 deletion(-)
doc/guides/sample_app_ug/l3_forward_power_man.rst +13 -0
index d7e1dc5813..b10ebe662e 100644
@@ -109,6 +109,8 @@ where,
 
 *   --telemetry:  Telemetry mode.
 
+*   --pmd-mgmt: PMD power management mode.
+
 See :doc:`l3_forward` for details.
 The L3fwd-power example reuses the L3fwd command line options.
 
@@ -455,3 +457,14 @@ reference cycles and accordingly busy rate is set  to either 0% or
 
 The new stats ``empty_poll`` , ``full_poll`` and ``busy_percent`` can be viewed by running the script
 ``/usertools/dpdk-telemetry-client.py`` and selecting the menu option ``Send for global Metrics``.
+
+PMD power management Mode
+-------------------------
+
+The PMD power management  mode support for ``l3fwd-power`` is a standalone mode, in this mode
+``l3fwd-power`` does simple l3fwding along with enable the power saving scheme on specific
+port/queue/lcore. Main purpose for this mode is to demonstrate how to use the PMD power management API.
+
+.. code-block:: console
+
+        ./build/examples/dpdk-l3fwd-power -l 1-3 --  --pmd-mgmt -p 0x0f --config="(0,0,2),(0,1,3)"
examples/l3fwd-power/main.c +45 -1

100 lines not on this page · raw message

Signature · 1 line
2.17.1
#386 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

In reply to Liang Ma

Please be more patient.
I asked some questions on v9 (ethdev API is not generic enough), and you send a v10 in the same minute you reply, without making sure I agree with your answers.

#387 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible. This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?

Regards
Liang

Quoted 7 lines · Thomas Monjalon

Please be more patient.
I asked some questions on v9 (ethdev API is not generic enough), and you send a v10 in the same minute you reply, without making sure I agree with your answers.

#388 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

Quoted 3 lines · Ma

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.

I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.

Quoted 13 lines

This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?

Regards
Liang

Quoted 7 lines · Thomas Monjalon

Please be more patient.
I asked some questions on v9 (ethdev API is not generic enough), and you send a v10 in the same minute you reply, without making sure I agree with your answers.

#389 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

28/10/2020 14:49, Jerin Jacob:

Quoted 5 lines · Ma
Quoted 3 lines

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.

Quoted 8 lines

I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.

I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)

Quoted 1 line
Quoted 1 line

This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?

Being experimental is not an excuse to throw something which is not satisfying.

#390 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

Quoted 17 lines

28/10/2020 14:49, Jerin Jacob:

Quoted 4 lines · Ma
Quoted 3 lines

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the

Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.

Quoted 9 lines

I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.

I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy: cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.

Quoted 4 lines

I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)

It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?

Being experimental is not an excuse to throw something which is not satisfying.

#391 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

Quoted 26 lines · Konstantin
Quoted 18 lines

28/10/2020 14:49, Jerin Jacob:

Quoted 4 lines · Ma
Quoted 3 lines

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the

Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.

Quoted 9 lines

I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.

I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy: cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.

Quoted 8 lines
Quoted 4 lines

I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)

It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.

Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, + uint64_t *expected, uint64_t *mask, uint8_t *data_sz);

Can we make it as
typedef void (*core_sleep_t)(void *rxq)

if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.

Note: core_sleep_t can take some more arguments such as enumerated policy if something more needs to be pushed to the driver.

Thoughts?

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?

Being experimental is not an excuse to throw something which is not satisfying.

#392 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

Quoted 37 lines · Jerin Jacob
Quoted 26 lines · Konstantin
Quoted 18 lines

28/10/2020 14:49, Jerin Jacob:

Quoted 4 lines · Ma
Quoted 3 lines

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the

Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.

Quoted 9 lines

I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.

I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy: cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.

Quoted 8 lines
Quoted 4 lines

I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)

It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.

Quoted 7 lines

Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, + uint64_t *expected, uint64_t *mask, uint8_t *data_sz);

Can we make it as
typedef void (*core_sleep_t)(void *rxq)

How about void (*eth_core_sleep_helper_t)(void *rxq, enum scheme, void *paramter) by this way, PMD can cast the parameter accorind to the scheme.
e.g. if scheme MEM_MONITOR then cast to umwait way.
however, this will introduce another problem.
we need add PMD query callback to figure out if PMD support this scheme.

Quoted 19 lines

if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.

Note: core_sleep_t can take some more arguments such as enumerated policy if something more needs to be pushed to the driver.

Thoughts?

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?

Being experimental is not an excuse to throw something which is not satisfying.

#393 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

Quoted 44 lines · Ma
Quoted 42 lines · Jerin Jacob
Quoted 26 lines · Konstantin
Quoted 18 lines

28/10/2020 14:49, Jerin Jacob:

Quoted 4 lines · Ma
Quoted 3 lines

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the

Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.

Quoted 9 lines

I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.

I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy: cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.

Quoted 8 lines
Quoted 4 lines

I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)

It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.

Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, + uint64_t *expected, uint64_t *mask, uint8_t *data_sz);

Quoted 7 lines
Quoted 2 lines

Can we make it as
typedef void (*core_sleep_t)(void *rxq)

How about void (*eth_core_sleep_helper_t)(void *rxq, enum scheme, void *paramter) by this way, PMD can cast the parameter accorind to the scheme.
e.g. if scheme MEM_MONITOR then cast to umwait way.
however, this will introduce another problem.
we need add PMD query callback to figure out if PMD support this scheme.

I thought scheme/policy something that "application cares" like below not arch specifics
1) wakeup up on first packet,
2) wake me up on first packet or timeout 100 us etc

Yes. We can have query on type of the policies supported.

Quoted 19 lines
Quoted 19 lines

if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.

Note: core_sleep_t can take some more arguments such as enumerated policy if something more needs to be pushed to the driver.

Thoughts?

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?

Being experimental is not an excuse to throw something which is not satisfying.

#394 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

Quoted 53 lines · Jerin Jacob
Quoted 44 lines · Ma
Quoted 42 lines · Jerin Jacob
Quoted 26 lines · Konstantin
Quoted 18 lines

28/10/2020 14:49, Jerin Jacob:

Quoted 4 lines · Ma
Quoted 3 lines

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the

Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.

Quoted 9 lines

I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.

I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy: cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.

Quoted 8 lines
Quoted 4 lines

I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)

It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.

Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, + uint64_t *expected, uint64_t *mask, uint8_t *data_sz);

Quoted 7 lines
Quoted 2 lines

Can we make it as
typedef void (*core_sleep_t)(void *rxq)

How about void (*eth_core_sleep_helper_t)(void *rxq, enum scheme, void *paramter) by this way, PMD can cast the parameter accorind to the scheme.
e.g. if scheme MEM_MONITOR then cast to umwait way.
however, this will introduce another problem.
we need add PMD query callback to figure out if PMD support this scheme.

Quoted 4 lines

I thought scheme/policy something that "application cares" like below not arch specifics
1) wakeup up on first packet,
2) wake me up on first packet or timeout 100 us etc

I need clarify about current design a bit. the purposed API just get target address. the API itself(include the PMD callback) will not demand the processor to idle/sleep. we use the post rx callback to do that. for ethdev layer, this API only is used to fetch target address from PMD, which make minmal impact to existing code.

Quoted 22 lines

Yes. We can have query on type of the policies supported.

Quoted 19 lines
Quoted 19 lines

if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.

Note: core_sleep_t can take some more arguments such as enumerated policy if something more needs to be pushed to the driver.

Thoughts?

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?

Being experimental is not an excuse to throw something which is not satisfying.

#395 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

Quoted 55 lines · Ma
Quoted 53 lines · Jerin Jacob
Quoted 51 lines · Ma
Quoted 37 lines · Jerin Jacob
Quoted 26 lines · Konstantin
Quoted 18 lines

28/10/2020 14:49, Jerin Jacob:

Quoted 4 lines · Ma
Quoted 3 lines

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the

Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.

Quoted 9 lines

I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.

I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy: cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.

Quoted 8 lines
Quoted 4 lines

I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)

It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.

Quoted 7 lines

Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, + uint64_t *expected, uint64_t *mask, uint8_t *data_sz);

Can we make it as
typedef void (*core_sleep_t)(void *rxq)

How about void (*eth_core_sleep_helper_t)(void *rxq, enum scheme, void *paramter) by this way, PMD can cast the parameter accorind to the scheme.
e.g. if scheme MEM_MONITOR then cast to umwait way.
however, this will introduce another problem.
we need add PMD query callback to figure out if PMD support this scheme.

Quoted 8 lines
Quoted 4 lines

I thought scheme/policy something that "application cares" like below not arch specifics
1) wakeup up on first packet,
2) wake me up on first packet or timeout 100 us etc

I need clarify about current design a bit. the purposed API just get target address. the API itself(include the PMD callback) will not demand the processor to idle/sleep. we use the post rx callback to do that. for ethdev layer, this API only is used to fetch target address from PMD, which make minmal impact to existing code.

I understand that. But if we move that logic to common code in ethdev as a set of internal
PMD helper functions(Helper functions for class of devices and/or arch) then it should be
possible. Right?

I do understand that, It will call for some rework in the code. I will leave up to ethdev maintainers
on specifics.

Quoted 23 lines
Quoted 23 lines

Yes. We can have query on type of the policies supported.

Quoted 19 lines
Quoted 19 lines

if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.

Note: core_sleep_t can take some more arguments such as enumerated policy if something more needs to be pushed to the driver.

Thoughts?

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?

Being experimental is not an excuse to throw something which is not satisfying.

#396 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

Quoted 65 lines · Jerin Jacob
Quoted 55 lines · Ma
Quoted 53 lines · Jerin Jacob
Quoted 51 lines · Ma
Quoted 37 lines · Jerin Jacob
Quoted 26 lines · Konstantin
Quoted 18 lines

28/10/2020 14:49, Jerin Jacob:

Quoted 4 lines · Ma
Quoted 3 lines

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the

Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.

Quoted 9 lines

I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.

I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy: cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.

Quoted 8 lines
Quoted 4 lines

I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)

It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.

Quoted 7 lines

Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, + uint64_t *expected, uint64_t *mask, uint8_t *data_sz);

Can we make it as
typedef void (*core_sleep_t)(void *rxq)

How about void (*eth_core_sleep_helper_t)(void *rxq, enum scheme, void *paramter) by this way, PMD can cast the parameter accorind to the scheme.
e.g. if scheme MEM_MONITOR then cast to umwait way.
however, this will introduce another problem.
we need add PMD query callback to figure out if PMD support this scheme.

Quoted 8 lines
Quoted 4 lines

I thought scheme/policy something that "application cares" like below not arch specifics
1) wakeup up on first packet,
2) wake me up on first packet or timeout 100 us etc

I need clarify about current design a bit. the purposed API just get target address. the API itself(include the PMD callback) will not demand the processor to idle/sleep. we use the post rx callback to do that. for ethdev layer, this API only is used to fetch target address from PMD, which make minmal impact to existing code.

Quoted 5 lines

I understand that. But if we move that logic to common code in ethdev as a set of internal
PMD helper functions(Helper functions for class of devices and/or arch) then it should be
possible. Right?

Although that's possible, but power manangment logic will be duplicated in all related PMD. Advantage of current design is decouple power mgmt logic with PMD, just need minmal help from PMD. there are 3 scheme already which are implemented as post rx callback. Put all power mgmt logic into PMD is way too heavy. Also current design is very easy to extend. Any new scheme has no impact to ethdev layer unless it need some kind of help from PMD. we have another 2 scheme, PAUSE and Freq Scale dont not need any PMD help.

Quoted 28 lines

I do understand that, It will call for some rework in the code. I will leave up to ethdev maintainers
on specifics.

Quoted 23 lines
Quoted 23 lines

Yes. We can have query on type of the policies supported.

Quoted 19 lines
Quoted 19 lines

if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.

Note: core_sleep_t can take some more arguments such as enumerated policy if something more needs to be pushed to the driver.

Thoughts?

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?

Being experimental is not an excuse to throw something which is not satisfying.

#397 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

In reply to Jerin Jacob
Quoted 39 lines
Quoted 25 lines
Quoted 17 lines

28/10/2020 14:49, Jerin Jacob:

Quoted 4 lines · Ma
Quoted 3 lines

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the

Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.

Quoted 9 lines

I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.

I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy: cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.

Quoted 8 lines
Quoted 4 lines

I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)

It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.

Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, + uint64_t *expected, uint64_t *mask, uint8_t *data_sz);

Quoted 6 lines

Can we make it as
typedef void (*core_sleep_t)(void *rxq)

if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.

Ok I see, thanks for explanation.
From my perspective main disadvantage of such approach -
it can't be extended easily.
If/when will have an ability for core to sleep/wake-up on multiple events (multiple addresses) will have to either rework that API again.

Quoted 15 lines

Note: core_sleep_t can take some more arguments such as enumerated policy if something more needs to be pushed to the driver.

Thoughts?

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?

Being experimental is not an excuse to throw something which is not satisfying.

#398 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

Quoted 47 lines · Konstantin
Quoted 40 lines
Quoted 26 lines
Quoted 18 lines

28/10/2020 14:49, Jerin Jacob:

Quoted 4 lines · Ma
Quoted 3 lines

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the

Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.

Quoted 9 lines

I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.

I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy: cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.

Quoted 8 lines
Quoted 4 lines

I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)

It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.

Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, + uint64_t *expected, uint64_t *mask, uint8_t *data_sz);

Quoted 6 lines

Can we make it as
typedef void (*core_sleep_t)(void *rxq)

if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.

Quoted 5 lines

Ok I see, thanks for explanation.
From my perspective main disadvantage of such approach -
it can't be extended easily.
If/when will have an ability for core to sleep/wake-up on multiple events (multiple addresses) will have to either rework that API again.

I think, we can enumerate the policies and pass the associated structures as input to the driver.

Quoted 16 lines
Quoted 16 lines

Note: core_sleep_t can take some more arguments such as enumerated policy if something more needs to be pushed to the driver.

Thoughts?

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?

Being experimental is not an excuse to throw something which is not satisfying.

#399 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

Quoted 64 lines

-----Original Message-----
From: Jerin Jacob <[email protected]>
Sent: Wednesday, October 28, 2020 3:40 PM
To: Ananyev, Konstantin <[email protected]>
Cc: Thomas Monjalon <[email protected]>; Ma, Liang J <[email protected]>; dpdk-dev <[email protected]>; Ruifeng Wang (Arm Technology China) <[email protected]>; Wang, Haiyue <[email protected]>; Richardson, Bruce <[email protected]>; Hunt, David <[email protected]>; Neil Horman <[email protected]>; McDaniel, Timothy <[email protected]>; Eads, Gage <[email protected]>; Marcin Wojtas <[email protected]>; Guy Tzalik <[email protected]>; Ajit Khaparde <[email protected]>; Harman Kalra <[email protected]>; John Daley <[email protected]>; Wei Hu (Xavier <[email protected]>; Ziyang Xuan <[email protected]>; [email protected]; Yong Wang <[email protected]>; [email protected]
Subject: Re: [PATCH v10 0/9] Add PMD power mgmt

Quoted 47 lines · Konstantin
Quoted 46 lines
Quoted 26 lines
Quoted 18 lines

28/10/2020 14:49, Jerin Jacob:

Quoted 4 lines · Ma
Quoted 3 lines

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the

Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.

Quoted 9 lines

I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.

I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy: cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.

Quoted 8 lines
Quoted 4 lines

I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)

It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.

Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, + uint64_t *expected, uint64_t *mask, uint8_t *data_sz);

Can we make it as
typedef void (*core_sleep_t)(void *rxq)

if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.

Quoted 8 lines
Quoted 5 lines

Ok I see, thanks for explanation.
From my perspective main disadvantage of such approach -
it can't be extended easily.
If/when will have an ability for core to sleep/wake-up on multiple events (multiple addresses) will have to either rework that API again.

I think, we can enumerate the policies and pass the associated structures as input to the driver.

What I am trying to say: with that API we will not be able to wait for events from multiple devices (HW queues).
I.E. something like that:

get_wake_addr(port=X, ..., &addr[0], ...);
get_wake_addr(port=Y,..., &addr[1],...);
wait_on_multi(addr, 2);

wouldn't be possible.

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines

Note: core_sleep_t can take some more arguments such as enumerated policy if something more needs to be pushed to the driver.

Thoughts?

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?

Being experimental is not an excuse to throw something which is not satisfying.

#400 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

Quoted 78 lines · Konstantin
Quoted 65 lines

-----Original Message-----
From: Jerin Jacob <[email protected]>
Sent: Wednesday, October 28, 2020 3:40 PM
To: Ananyev, Konstantin <[email protected]>
Cc: Thomas Monjalon <[email protected]>; Ma, Liang J <[email protected]>; dpdk-dev <[email protected]>; Ruifeng Wang (Arm Technology China) <[email protected]>; Wang, Haiyue <[email protected]>; Richardson, Bruce <[email protected]>; Hunt, David <[email protected]>; Neil Horman <[email protected]>; McDaniel, Timothy <[email protected]>; Eads, Gage <[email protected]>; Marcin Wojtas <[email protected]>; Guy Tzalik <[email protected]>; Ajit Khaparde <[email protected]>; Harman Kalra <[email protected]>; John Daley <[email protected]>; Wei Hu (Xavier <[email protected]>; Ziyang Xuan <[email protected]>; [email protected]; Yong Wang <[email protected]>; [email protected]
Subject: Re: [PATCH v10 0/9] Add PMD power mgmt

Quoted 47 lines · Konstantin
Quoted 46 lines
Quoted 26 lines
Quoted 18 lines

28/10/2020 14:49, Jerin Jacob:

Quoted 4 lines · Ma
Quoted 3 lines

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the

Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.

Quoted 9 lines

I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.

I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy: cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.

Quoted 8 lines
Quoted 4 lines

I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)

It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.

Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, + uint64_t *expected, uint64_t *mask, uint8_t *data_sz);

Can we make it as
typedef void (*core_sleep_t)(void *rxq)

if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.

Quoted 8 lines
Quoted 5 lines

Ok I see, thanks for explanation.
From my perspective main disadvantage of such approach -
it can't be extended easily.
If/when will have an ability for core to sleep/wake-up on multiple events (multiple addresses) will have to either rework that API again.

I think, we can enumerate the policies and pass the associated structures as input to the driver.

What I am trying to say: with that API we will not be able to wait for events from multiple devices (HW queues).
I.E. something like that:

Quoted 5 lines

get_wake_addr(port=X, ..., &addr[0], ...);
get_wake_addr(port=Y,..., &addr[1],...);
wait_on_multi(addr, 2);

wouldn't be possible.

I see. But the current implementation dictates the only queue bound to a core. Right?

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines

Note: core_sleep_t can take some more arguments such as enumerated policy if something more needs to be pushed to the driver.

Thoughts?

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?

Being experimental is not an excuse to throw something which is not satisfying.

#401 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

Quoted 85 lines
Quoted 13 lines · Konstantin
Quoted 13 lines

-----Original Message-----
From: Jerin Jacob <[email protected]>
Sent: Wednesday, October 28, 2020 3:40 PM
To: Ananyev, Konstantin <[email protected]>
Cc: Thomas Monjalon <[email protected]>; Ma, Liang J <[email protected]>; dpdk-dev <[email protected]>; Ruifeng Wang (Arm Technology China) <[email protected]>; Wang, Haiyue <[email protected]>; Richardson, Bruce <[email protected]>; Hunt, David <[email protected]>; Neil Horman <[email protected]>; McDaniel, Timothy <[email protected]>; Eads, Gage <[email protected]>; Marcin Wojtas <[email protected]>; Guy Tzalik <[email protected]>; Ajit Khaparde <[email protected]>; Harman Kalra <[email protected]>; John Daley <[email protected]>; Wei Hu (Xavier <[email protected]>; Ziyang Xuan <[email protected]>; [email protected];

Yong

Quoted 8 lines
Quoted 8 lines

Wang <[email protected]>; [email protected]
Subject: Re: [PATCH v10 0/9] Add PMD power mgmt

Quoted 3 lines · Konstantin
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

28/10/2020 14:49, Jerin Jacob:

Quoted 4 lines · Ma
Quoted 3 lines

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From

the

Quoted 52 lines
Quoted 39 lines
Quoted 39 lines
Quoted 38 lines
Quoted 18 lines
Quoted 10 lines

Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.

Quoted 9 lines

I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.

I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy: cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.

Quoted 8 lines
Quoted 4 lines

I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)

It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.

Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, + uint64_t *expected, uint64_t *mask, uint8_t *data_sz);

Can we make it as
typedef void (*core_sleep_t)(void *rxq)

if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.

Quoted 8 lines
Quoted 5 lines

Ok I see, thanks for explanation.
From my perspective main disadvantage of such approach -
it can't be extended easily.
If/when will have an ability for core to sleep/wake-up on multiple events (multiple addresses) will have to either rework that API again.

I think, we can enumerate the policies and pass the associated structures as input to the driver.

What I am trying to say: with that API we will not be able to wait for events from multiple devices (HW queues).
I.E. something like that:

Quoted 8 lines
Quoted 5 lines

get_wake_addr(port=X, ..., &addr[0], ...);
get_wake_addr(port=Y,..., &addr[1],...);
wait_on_multi(addr, 2);

wouldn't be possible.

I see. But the current implementation dictates the only queue bound to a core. Right?

Yes, current implementation of rte_power_monitor() supports only single address. Though proposed API for both ethdev (get_wake_addr) and power(rte_power_pmd_mgmt_queue_enable) don't dictate
one to one mapping as the only possible usage model.

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

Note: core_sleep_t can take some more arguments such as enumerated policy if something more needs to be pushed to the driver.

Thoughts?

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?

Being experimental is not an excuse to throw something which is not satisfying.

#402 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

In reply to Jerin Jacob
Quoted 59 lines · Jerin Jacob
Quoted 57 lines · Konstantin
Quoted 44 lines
Quoted 44 lines
Quoted 43 lines
Quoted 23 lines
Quoted 15 lines

28/10/2020 14:49, Jerin Jacob:

Quoted 4 lines · Ma
Quoted 3 lines

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the

Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.

Quoted 9 lines

I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.

I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy: cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.

Quoted 8 lines
Quoted 4 lines

I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)

It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.

Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, + uint64_t *expected, uint64_t *mask, uint8_t *data_sz);

Can we make it as
typedef void (*core_sleep_t)(void *rxq)

if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.

Quoted 8 lines
Quoted 5 lines

Ok I see, thanks for explanation.
From my perspective main disadvantage of such approach -
it can't be extended easily.
If/when will have an ability for core to sleep/wake-up on multiple events (multiple addresses) will have to either rework that API again.

I think, we can enumerate the policies and pass the associated structures as input to the driver.

What I am trying to say: with that API we will not be able to wait for events from multiple devices (HW queues).
I.E. something like that:

Quoted 8 lines
Quoted 5 lines

get_wake_addr(port=X, ..., &addr[0], ...);
get_wake_addr(port=Y,..., &addr[1],...);
wait_on_multi(addr, 2);

wouldn't be possible.

I see. But the current implementation dictates the only queue bound to a core. Right?

Current implementation only support 1:1 queue/core mapping is because of the limitation of umwait/umonitor which can not work with multiple address range. However, for other scheme like PASUE/Freq Scale have no such limitation. The proposed API itself doesn't limit the 1:1 queue/core mapping.

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

Note: core_sleep_t can take some more arguments such as enumerated policy if something more needs to be pushed to the driver.

Thoughts?

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?

Being experimental is not an excuse to throw something which is not satisfying.

#403 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

Quoted 111 lines

-----Original Message-----
From: Ma, Liang J <[email protected]>
Sent: Wednesday, October 28, 2020 11:48 AM
To: Jerin Jacob <[email protected]>
Cc: Ananyev, Konstantin <[email protected]>; Thomas Monjalon <[email protected]>; dpdk-dev <[email protected]>; Ruifeng Wang (Arm Technology China) <[email protected]>; Wang, Haiyue <[email protected]>; Richardson, Bruce <[email protected]>; Hunt, David <[email protected]>; Neil Horman <[email protected]>; McDaniel, Timothy <[email protected]>; Eads, Gage <[email protected]>; Marcin Wojtas <[email protected]>; Guy Tzalik <[email protected]>; Ajit Khaparde <[email protected]>; Harman Kalra <[email protected]>; John Daley <[email protected]>; Wei Hu (Xavier <[email protected]>; Ziyang Xuan <[email protected]>; [email protected]; Yong Wang <[email protected]>; [email protected]
Subject: Re: [PATCH v10 0/9] Add PMD power mgmt

Quoted 4 lines · Jerin Jacob
Quoted 2 lines · Konstantin
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

28/10/2020 14:49, Jerin Jacob:

Quoted 1 line

On Wed, Oct 28, 2020 at 7:05 PM Liang, Ma

Quoted 3 lines · <[email protected]> wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't

think I can solve the issue of a generic API on my own. From the

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Community Call last week Jerin also said that a generic was

investigated but that a single solution wasn't feasible.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc

by

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects?

or similar

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

thoughts on abstraction.

I think there is probably some sort of misunderstanding. This API is not about providing acync notification when next packet

arrives.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

This is about to putting core to sleep till some event (or timeout)

happens.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

From my perspective the closest analogy: cond_timedwait(). So we need PMD to tell us what will be the address of the condition

variable

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

we should sleep on.

Quoted 4 lines

I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen. Second problem: it is probably not generic enough (if we understand

it well)

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.

Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, + uint64_t *expected, uint64_t *mask, uint8_t *data_sz);

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Can we make it as
typedef void (*core_sleep_t)(void *rxq)

if we do such abstraction and "move the polling on memory by

HW/CPU"

Quoted 16 lines
Quoted 16 lines
Quoted 11 lines
Quoted 8 lines
Quoted 2 lines

to the driver using a helper function then I can think of abstracting in some way in all PMDs.

Ok I see, thanks for explanation.
From my perspective main disadvantage of such approach -
it can't be extended easily.
If/when will have an ability for core to sleep/wake-up on multiple events (multiple addresses) will have to either rework that API again.

I think, we can enumerate the policies and pass the associated structures as input to the driver.

What I am trying to say: with that API we will not be able to wait for events from multiple devices (HW queues).
I.E. something like that:

Quoted 8 lines
Quoted 5 lines

get_wake_addr(port=X, ..., &addr[0], ...);
get_wake_addr(port=Y,..., &addr[1],...);
wait_on_multi(addr, 2);

wouldn't be possible.

I see. But the current implementation dictates the only queue bound to a core. Right?

Current implementation only support 1:1 queue/core mapping is because of the limitation of umwait/umonitor which can not work with multiple address range. However, for other scheme like PASUE/Freq Scale have no such limitation.
The proposed API itself doesn't limit the 1:1 queue/core mapping.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Note: core_sleep_t can take some more arguments such as

enumerated

Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

policy if something more needs to be pushed to the driver.

Thoughts?

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

This API is experimental and other vendor support can be added

as needed. If there are any other open issue let me know?

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Being experimental is not an excuse to throw something which is not satisfying.

It would be nice if the low level definition of the UMWAIT and UMONOTOR instructions were split out into their own inline function or macro so that any PMD could use the intrinsic without being tied to ethdev or any of the other logic associated with this patch set. This would be similar to rte_wmb, and so on

#404 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

On 28 Oct 09:54, McDaniel, Timothy wrote:
<snip>

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

It would be nice if the low level definition of the UMWAIT and UMONOTOR instructions were split out into their own inline function or macro so that any PMD could use the intrinsic without being tied to ethdev or any of the other logic associated with this patch set. This would be similar to rte_wmb, and so on

Current patches split the intrinsics out. and already part of eal lib. any other PMD can re-use it easily.

Quoted 2 lines
#405 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

In reply to Liang, Ma
Quoted 65 lines · Ma
Quoted 63 lines · Jerin Jacob
Quoted 61 lines · Konstantin
Quoted 44 lines
Quoted 44 lines
Quoted 43 lines
Quoted 23 lines
Quoted 15 lines

28/10/2020 14:49, Jerin Jacob:

Quoted 4 lines · Ma
Quoted 3 lines

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the

Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.

Quoted 9 lines

I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.

I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy: cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.

Quoted 8 lines
Quoted 4 lines

I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)

It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.

Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, + uint64_t *expected, uint64_t *mask, uint8_t *data_sz);

Can we make it as
typedef void (*core_sleep_t)(void *rxq)

if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.

Quoted 8 lines
Quoted 5 lines

Ok I see, thanks for explanation.
From my perspective main disadvantage of such approach -
it can't be extended easily.
If/when will have an ability for core to sleep/wake-up on multiple events (multiple addresses) will have to either rework that API again.

I think, we can enumerate the policies and pass the associated structures as input to the driver.

What I am trying to say: with that API we will not be able to wait for events from multiple devices (HW queues).
I.E. something like that:

get_wake_addr(port=X, ..., &addr[0], ...);
get_wake_addr(port=Y,..., &addr[1],...);
wait_on_multi(addr, 2);

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

wouldn't be possible.

I see. But the current implementation dictates the only queue bound to a core. Right?

Current implementation only support 1:1 queue/core mapping is because of the limitation of umwait/umonitor which can not work with multiple address range. However, for other scheme like PASUE/Freq Scale have no such limitation. The proposed API itself doesn't limit the 1:1 queue/core mapping.

The PMD would not know if it is 1:1 queue/core or any other shared scheme. So the intelligence and decision making is best left to the application. I think PMD and the underlying hardware does not need to know what kind of power management scheme is implemented.
IMHO the original API which provides the address, value and mask should suffice. Any other callback or handshake between PMD and application may be an overkill.

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

Note: core_sleep_t can take some more arguments such as enumerated policy if something more needs to be pushed to the driver.

Thoughts?

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?

Being experimental is not an excuse to throw something which is not satisfying.

#406 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

Quoted 91 lines

-----Original Message-----
From: Ajit Khaparde <[email protected]>
Sent: Wednesday, October 28, 2020 5:02 PM
To: Ma, Liang J <[email protected]>
Cc: Jerin Jacob <[email protected]>; Ananyev, Konstantin <[email protected]>; Thomas Monjalon <[email protected]>; dpdk-dev <[email protected]>; Ruifeng Wang (Arm Technology China) <[email protected]>; Wang, Haiyue <[email protected]>; Richardson, Bruce <[email protected]>; Hunt, David <[email protected]>; Neil Horman <[email protected]>; McDaniel, Timothy <[email protected]>; Eads, Gage <[email protected]>; Marcin Wojtas <[email protected]>; Guy Tzalik <[email protected]>; Harman Kalra <[email protected]>; John Daley <[email protected]>; Wei Hu (Xavier <[email protected]>; Ziyang Xuan <[email protected]>; [email protected]; Yong Wang <[email protected]>; [email protected]
Subject: Re: [PATCH v10 0/9] Add PMD power mgmt

Quoted 9 lines · Ma
Quoted 7 lines · Jerin Jacob
Quoted 5 lines · Konstantin
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

28/10/2020 14:49, Jerin Jacob:

Quoted 4 lines · Ma
Quoted 3 lines

Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own.

From the

Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 39 lines
Quoted 39 lines
Quoted 38 lines
Quoted 18 lines
Quoted 10 lines

Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.

Quoted 9 lines

I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.

I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy: cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.

Quoted 8 lines
Quoted 4 lines

I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)

It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.

Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, + uint64_t *expected, uint64_t *mask, uint8_t *data_sz);

Can we make it as
typedef void (*core_sleep_t)(void *rxq)

if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.

Quoted 8 lines
Quoted 5 lines

Ok I see, thanks for explanation.
From my perspective main disadvantage of such approach -
it can't be extended easily.
If/when will have an ability for core to sleep/wake-up on multiple events (multiple addresses) will have to either rework that API again.

I think, we can enumerate the policies and pass the associated structures as input to the driver.

What I am trying to say: with that API we will not be able to wait for events from multiple devices (HW queues).
I.E. something like that:

get_wake_addr(port=X, ..., &addr[0], ...);
get_wake_addr(port=Y,..., &addr[1],...);
wait_on_multi(addr, 2);

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

wouldn't be possible.

I see. But the current implementation dictates the only queue bound to a core. Right?

Current implementation only support 1:1 queue/core mapping is because of the limitation of umwait/umonitor which can not work with multiple address range. However, for other scheme like PASUE/Freq Scale have no such limitation. The proposed API itself doesn't limit the 1:1 queue/core mapping.

Quoted 4 lines

The PMD would not know if it is 1:1 queue/core or any other shared scheme. So the intelligence and decision making is best left to the application. I think PMD and the underlying hardware does not need to know what kind of power management scheme is implemented.

Yep, good point. 100% agree.

Quoted 24 lines

IMHO the original API which provides the address, value and mask should suffice. Any other callback or handshake between PMD and application may be an overkill.

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

Note: core_sleep_t can take some more arguments such as enumerated policy if something more needs to be pushed to the driver.

Thoughts?

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?

Being experimental is not an excuse to throw something which is not satisfying.

#407 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

In reply to Liang Ma
Quoted 8 lines · Liang Ma

This patchset proposes a simple API for Ethernet drivers to cause the CPU to enter a power-optimized state while waiting for packets to arrive, along with a set of generic intrinsics that facilitate that. This is achieved through cooperation with the NIC driver that will allow us to know address of wake up event, and wait for writes on it.

Is the wake event the same as ring status or interrupt status register?

So in a way the PMD is passing the address of the next ring descriptor? So that instead of the PMD polling it, the application peeks at it and when ready asks the PMD to actually process the packet(s)?

Quoted 20 lines

On IA, this is achieved through using UMONITOR/UMWAIT instructions. They are used in their raw opcode form because there is no widespread compiler support for them yet. Still, the API is made generic enough to hopefully support other architectures, if they happen to implement similar instructions.

To achieve power savings, there is a very simple mechanism
used: we're counting empty polls, and if a certain threshold is reached, we get the address of next RX ring descriptor from the NIC driver, arm the monitoring hardware, and enter a power-optimized state. We will then wake up when either a timeout happens, or a write happens (or generally whenever CPU feels like waking up - this is platform- specific), and proceed as normal. The empty poll counter is reset whenever we actually get packets, so we only go to sleep when we know nothing is going on. The mechanism is generic which can be used for any write back descriptor.

Quoted 5 lines

Why are we putting it into ethdev as opposed to leaving this up to the application? Our customers specifically requested a way to do it wit minimal changes to the application code. The current approach allows to just flip a switch and automatically have power savings.

The application still has to know address of wake up event. Right? And then it will need the logic to count empty polls and the threshold? This will be done by application or something else?

Quoted 65 lines

- Only 1:1 core to queue mapping is supported, meaning that each lcore must at most handle RX on a single queue
- Support 3 type policies. UMWAIT/PAUSE/Frequency_Scale
- Power management is enabled per-queue
- The API doesn't extend to other device types

Liang Ma (9):
eal: add new x86 cpuid support for WAITPKG
eal: add power management intrinsics
eal: add intrinsics support check infrastructure
ethdev: add simple power management API
power: add PMD power management API and callback net/ixgbe: implement power management API net/i40e: implement power management API net/ice: implement power management API examples/l3fwd-power: enable PMD power mgmt

 doc/guides/prog_guide/power_man.rst           |  48 +++
 doc/guides/rel_notes/release_20_11.rst        |  15 +
 .../sample_app_ug/l3_forward_power_man.rst    |  13 +
 drivers/net/i40e/i40e_ethdev.c                |   1 +
 drivers/net/i40e/i40e_rxtx.c                  |  26 ++
 drivers/net/i40e/i40e_rxtx.h                  |   2 +
 drivers/net/ice/ice_ethdev.c                  |   1 +
 drivers/net/ice/ice_rxtx.c                    |  26 ++
 drivers/net/ice/ice_rxtx.h                    |   2 +
 drivers/net/ixgbe/ixgbe_ethdev.c              |   1 +
 drivers/net/ixgbe/ixgbe_rxtx.c                |  25 ++
 drivers/net/ixgbe/ixgbe_rxtx.h                |   2 +
 examples/l3fwd-power/main.c                   |  46 ++-
 lib/librte_eal/arm/include/meson.build        |   1 +
 .../arm/include/rte_power_intrinsics.h        |  60 ++++
 lib/librte_eal/arm/rte_cpuflags.c             |   6 +
 lib/librte_eal/include/generic/rte_cpuflags.h |  26 ++
 .../include/generic/rte_power_intrinsics.h    | 123 +++++++
 lib/librte_eal/include/meson.build            |   1 +
 lib/librte_eal/ppc/include/meson.build        |   1 +
 .../ppc/include/rte_power_intrinsics.h        |  60 ++++
 lib/librte_eal/ppc/rte_cpuflags.c             |   7 +
 lib/librte_eal/version.map                    |   1 +
 lib/librte_eal/x86/include/meson.build        |   1 +
 lib/librte_eal/x86/include/rte_cpuflags.h     |   1 +
 .../x86/include/rte_power_intrinsics.h        | 135 ++++++++
 lib/librte_eal/x86/rte_cpuflags.c             |  14 +
 lib/librte_ethdev/rte_ethdev.c                |  23 ++
 lib/librte_ethdev/rte_ethdev.h                |  33 ++
 lib/librte_ethdev/rte_ethdev_driver.h         |  28 ++
 lib/librte_ethdev/version.map                 |   1 +
 lib/librte_power/meson.build                  |   5 +-
 lib/librte_power/rte_power_pmd_mgmt.c         | 320 ++++++++++++++++++
 lib/librte_power/rte_power_pmd_mgmt.h         |  92 +++++
 lib/librte_power/version.map                  |   4 +
 35 files changed, 1148 insertions(+), 3 deletions(-)
 create mode 100644 lib/librte_eal/arm/include/rte_power_intrinsics.h
 create mode 100644 lib/librte_eal/include/generic/rte_power_intrinsics.h
 create mode 100644 lib/librte_eal/ppc/include/rte_power_intrinsics.h
 create mode 100644 lib/librte_eal/x86/include/rte_power_intrinsics.h
 create mode 100644 lib/librte_power/rte_power_pmd_mgmt.c
 create mode 100644 lib/librte_power/rte_power_pmd_mgmt.h

--
2.17.1

#408 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

Quoted 10 lines · Ajit Khaparde
Quoted 8 lines · Liang Ma

This patchset proposes a simple API for Ethernet drivers to cause the CPU to enter a power-optimized state while waiting for packets to arrive, along with a set of generic intrinsics that facilitate that. This is achieved through cooperation with the NIC driver that will allow us to know address of wake up event, and wait for writes on it.

Is the wake event the same as ring status or interrupt status register?

the wake event is any change happen to the monitoring address range. can be a ring descriptor(e.g. Done Bits) but not limiti to that.

Quoted 4 lines

So in a way the PMD is passing the address of the next ring descriptor? So that instead of the PMD polling it, the application peeks at it and when ready asks the PMD to actually process the packet(s)?

NO, after PMD pssing the address, processor will monitoring and wait on the address range(aka, move to a sub-state). any change to the address range caused by other source(another processor, NIC etc) will wake up the processor then start polling again.

Quoted 21 lines
Quoted 21 lines

On IA, this is achieved through using UMONITOR/UMWAIT instructions. They are used in their raw opcode form because there is no widespread compiler support for them yet. Still, the API is made generic enough to hopefully support other architectures, if they happen to implement similar instructions.

To achieve power savings, there is a very simple mechanism
used: we're counting empty polls, and if a certain threshold is reached, we get the address of next RX ring descriptor from the NIC driver, arm the monitoring hardware, and enter a power-optimized state. We will then wake up when either a timeout happens, or a write happens (or generally whenever CPU feels like waking up - this is platform- specific), and proceed as normal. The empty poll counter is reset whenever we actually get packets, so we only go to sleep when we know nothing is going on. The mechanism is generic which can be used for any write back descriptor.

Quoted 8 lines
Quoted 5 lines

Why are we putting it into ethdev as opposed to leaving this up to the application? Our customers specifically requested a way to do it wit minimal changes to the application code. The current approach allows to just flip a switch and automatically have power savings.

The application still has to know address of wake up event. Right? And then it will need the logic to count empty polls and the threshold? This will be done by application or something else?

the empty poll counting is done by the power library callback function which is included in the patch 5. APP has no need change any thing beside the initilization code(call the enable/disable API). please Ref the patch 9 which demostrate how to use it with l3fwd-power.

Quoted 66 lines
Quoted 66 lines

- Only 1:1 core to queue mapping is supported, meaning that each lcore must at most handle RX on a single queue
- Support 3 type policies. UMWAIT/PAUSE/Frequency_Scale
- Power management is enabled per-queue
- The API doesn't extend to other device types

Liang Ma (9):
eal: add new x86 cpuid support for WAITPKG
eal: add power management intrinsics
eal: add intrinsics support check infrastructure
ethdev: add simple power management API
power: add PMD power management API and callback net/ixgbe: implement power management API net/i40e: implement power management API net/ice: implement power management API examples/l3fwd-power: enable PMD power mgmt

 doc/guides/prog_guide/power_man.rst           |  48 +++
 doc/guides/rel_notes/release_20_11.rst        |  15 +
 .../sample_app_ug/l3_forward_power_man.rst    |  13 +
 drivers/net/i40e/i40e_ethdev.c                |   1 +
 drivers/net/i40e/i40e_rxtx.c                  |  26 ++
 drivers/net/i40e/i40e_rxtx.h                  |   2 +
 drivers/net/ice/ice_ethdev.c                  |   1 +
 drivers/net/ice/ice_rxtx.c                    |  26 ++
 drivers/net/ice/ice_rxtx.h                    |   2 +
 drivers/net/ixgbe/ixgbe_ethdev.c              |   1 +
 drivers/net/ixgbe/ixgbe_rxtx.c                |  25 ++
 drivers/net/ixgbe/ixgbe_rxtx.h                |   2 +
 examples/l3fwd-power/main.c                   |  46 ++-
 lib/librte_eal/arm/include/meson.build        |   1 +
 .../arm/include/rte_power_intrinsics.h        |  60 ++++
 lib/librte_eal/arm/rte_cpuflags.c             |   6 +
 lib/librte_eal/include/generic/rte_cpuflags.h |  26 ++
 .../include/generic/rte_power_intrinsics.h    | 123 +++++++
 lib/librte_eal/include/meson.build            |   1 +
 lib/librte_eal/ppc/include/meson.build        |   1 +
 .../ppc/include/rte_power_intrinsics.h        |  60 ++++
 lib/librte_eal/ppc/rte_cpuflags.c             |   7 +
 lib/librte_eal/version.map                    |   1 +
 lib/librte_eal/x86/include/meson.build        |   1 +
 lib/librte_eal/x86/include/rte_cpuflags.h     |   1 +
 .../x86/include/rte_power_intrinsics.h        | 135 ++++++++
 lib/librte_eal/x86/rte_cpuflags.c             |  14 +
 lib/librte_ethdev/rte_ethdev.c                |  23 ++
 lib/librte_ethdev/rte_ethdev.h                |  33 ++
 lib/librte_ethdev/rte_ethdev_driver.h         |  28 ++
 lib/librte_ethdev/version.map                 |   1 +
 lib/librte_power/meson.build                  |   5 +-
 lib/librte_power/rte_power_pmd_mgmt.c         | 320 ++++++++++++++++++
 lib/librte_power/rte_power_pmd_mgmt.h         |  92 +++++
 lib/librte_power/version.map                  |   4 +
 35 files changed, 1148 insertions(+), 3 deletions(-)
 create mode 100644 lib/librte_eal/arm/include/rte_power_intrinsics.h
 create mode 100644 lib/librte_eal/include/generic/rte_power_intrinsics.h
 create mode 100644 lib/librte_eal/ppc/include/rte_power_intrinsics.h
 create mode 100644 lib/librte_eal/x86/include/rte_power_intrinsics.h
 create mode 100644 lib/librte_power/rte_power_pmd_mgmt.c
 create mode 100644 lib/librte_power/rte_power_pmd_mgmt.h

--
2.17.1

#409 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

In reply to Liang Ma

27/10/2020 15:59, Liang Ma:

Quoted 4 lines

Liang Ma (9):
eal: add new x86 cpuid support for WAITPKG
eal: add power management intrinsics
eal: add intrinsics support check infrastructure

EAL patches applied, thanks.

Quoted 1 line

ethdev: add simple power management API

Waiting for doxygen reword which may help to get acks for ethdev API.

Quoted 5 lines

power: add PMD power management API and callback net/ixgbe: implement power management API net/i40e: implement power management API net/ice: implement power management API examples/l3fwd-power: enable PMD power mgmt

#410 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

Quoted 7 lines · Thomas Monjalon

27/10/2020 15:59, Liang Ma:

Quoted 4 lines

Liang Ma (9):
eal: add new x86 cpuid support for WAITPKG
eal: add power management intrinsics
eal: add intrinsics support check infrastructure

EAL patches applied, thanks.

thanks!

Quoted 4 lines
Quoted 2 lines

ethdev: add simple power management API

Waiting for doxygen reword which may help to get acks for ethdev API.

we are working on this. I will send out v11 today.

Quoted 9 lines
Quoted 6 lines

power: add PMD power management API and callback net/ixgbe: implement power management API net/i40e: implement power management API net/ice: implement power management API examples/l3fwd-power: enable PMD power mgmt

#411 lore raw

Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt

30/10/2020 10:36, Liang, Ma:

Quoted 7 lines · Thomas Monjalon
Quoted 6 lines

27/10/2020 15:59, Liang Ma:

Quoted 4 lines

Liang Ma (9):
eal: add new x86 cpuid support for WAITPKG
eal: add power management intrinsics
eal: add intrinsics support check infrastructure

Quoted 7 lines
Quoted 1 line

EAL patches applied, thanks.

thanks!

Quoted 4 lines
Quoted 2 lines

ethdev: add simple power management API

Waiting for doxygen reword which may help to get acks for ethdev API.

we are working on this. I will send out v11 today.

David sent some interesting questions about the EAL API. Maybe some explanations are missing, and we could improve the API. Please reply, I would like to solve the EAL questions quickly.

#412 lore raw

PATCH v8 2/10 eal: add power management intrinsics

In reply to Anatoly Burakov

Add two new power management intrinsics, and provide an implementation in eal/x86 based on UMONITOR/UMWAIT instructions. The instructions are implemented as raw byte opcodes because there is not yet widespread compiler support for these instructions.

The power management instructions provide an architecture-specific function to either wait until a specified TSC timestamp is reached, or optionally wait until either a TSC timestamp is reached or a memory location is written to. The monitor function also provides an optional comparison, to avoid sleeping when the expected write has already happened, and no more writes are expected.

For more details, please refer to Intel(R) 64 and IA-32 Architectures Software Developer's Manual, Volume 2.

---
 lib/librte_eal/arm/include/meson.build        |   1 +
 .../arm/include/rte_power_intrinsics.h        |  60 ++++++++
 .../include/generic/rte_power_intrinsics.h    | 111 ++++++++++++++
 lib/librte_eal/include/meson.build            |   1 +
 lib/librte_eal/ppc/include/meson.build        |   1 +
 .../ppc/include/rte_power_intrinsics.h        |  60 ++++++++
 lib/librte_eal/x86/include/meson.build        |   1 +
 .../x86/include/rte_power_intrinsics.h        | 135 ++++++++++++++++++
 8 files changed, 370 insertions(+)
 create mode 100644 lib/librte_eal/arm/include/rte_power_intrinsics.h
 create mode 100644 lib/librte_eal/include/generic/rte_power_intrinsics.h
 create mode 100644 lib/librte_eal/ppc/include/rte_power_intrinsics.h
 create mode 100644 lib/librte_eal/x86/include/rte_power_intrinsics.h
lib/librte_eal/arm/include/meson.build +1 -0
index 73b750a18f..c6a9f70d73 100644
@@ -20,6 +20,7 @@ arch_headers = files(
 	'rte_pause_32.h',
 	'rte_pause_64.h',
 	'rte_pause.h',
+	'rte_power_intrinsics.h',
 	'rte_prefetch_32.h',
 	'rte_prefetch_64.h',
 	'rte_prefetch.h',
lib/librte_eal/arm/include/rte_power_intrinsics.h new file +60 -0

60 lines not on this page · raw message

lib/librte_eal/include/generic/rte_power_intrinsics.h new file +111 -0

111 lines not on this page · raw message

lib/librte_eal/include/meson.build +1 -0
index cd09027958..3a12e87e19 100644
@@ -60,6 +60,7 @@ generic_headers = files(
 	'generic/rte_memcpy.h',
 	'generic/rte_pause.h',
 	'generic/rte_prefetch.h',
+	'generic/rte_power_intrinsics.h',
 	'generic/rte_rwlock.h',
 	'generic/rte_spinlock.h',
 	'generic/rte_ticketlock.h',
lib/librte_eal/ppc/include/meson.build +1 -0
index ab4bd28092..0873b2aecb 100644
@@ -10,6 +10,7 @@ arch_headers = files(
 	'rte_io.h',
 	'rte_memcpy.h',
 	'rte_pause.h',
+	'rte_power_intrinsics.h',
 	'rte_prefetch.h',
 	'rte_rwlock.h',
 	'rte_spinlock.h',
lib/librte_eal/ppc/include/rte_power_intrinsics.h new file +60 -0

60 lines not on this page · raw message

lib/librte_eal/x86/include/meson.build +1 -0
index f0e998c2fe..494a8142a2 100644
@@ -13,6 +13,7 @@ arch_headers = files(
 	'rte_io.h',
 	'rte_memcpy.h',
 	'rte_prefetch.h',
+	'rte_power_intrinsics.h',
 	'rte_pause.h',
 	'rte_rtm.h',
 	'rte_rwlock.h',
lib/librte_eal/x86/include/rte_power_intrinsics.h new file +135 -0

135 lines not on this page · raw message

Signature · 1 line
2.17.1
#413 lore raw

PATCH v8 3/10 eal: add intrinsics support check infrastructure

In reply to Anatoly Burakov

Currently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.

This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.

---

Notes:
v6:
- Fix the comments
v8:
- Rename eal version.map

---
 lib/librte_eal/arm/rte_cpuflags.c             |  6 +++++
 lib/librte_eal/include/generic/rte_cpuflags.h | 26 +++++++++++++++++++
 .../include/generic/rte_power_intrinsics.h    | 12 +++++++++
 lib/librte_eal/ppc/rte_cpuflags.c             |  7 +++++
 lib/librte_eal/version.map                    |  1 +
 lib/librte_eal/x86/rte_cpuflags.c             | 12 +++++++++
 6 files changed, 64 insertions(+)
lib/librte_eal/arm/rte_cpuflags.c +6 -0
index 7b257b7873..e3a53bcece 100644
@@ -151,3 +151,9 @@ rte_cpu_get_flag_name(enum rte_cpu_flag_t feature)
 		return NULL;
 	return rte_cpu_feature_table[feature].name;
 }
+
+void
+rte_cpu_get_intrinsics_support(struct rte_cpu_intrinsics *intrinsics)
+{
+	memset(intrinsics, 0, sizeof(*intrinsics));
+}
lib/librte_eal/include/generic/rte_cpuflags.h +26 -0
index 872f0ebe3e..28a5aecde8 100644
@@ -13,6 +13,32 @@
 #include "rte_common.h"
 #include <errno.h>
 
+#include <rte_compat.h>
+
+/**
+ * Structure used to describe platform-specific intrinsics that may or may not
+ * be supported at runtime.
+ */
+struct rte_cpu_intrinsics {
+	uint32_t power_monitor : 1;
+	/**< indicates support for rte_power_monitor function */
+	uint32_t power_pause : 1;
+	/**< indicates support for rte_power_pause function */
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Check CPU support for various intrinsics at runtime.
+ *
+ * @param intrinsics
+ *     Pointer to a structure to be filled.
+ */
+__rte_experimental
+void
+rte_cpu_get_intrinsics_support(struct rte_cpu_intrinsics *intrinsics);
+
 /**
  * Enumeration of all CPU features supported
  */
lib/librte_eal/include/generic/rte_power_intrinsics.h +12 -0
index fb897d9060..03a326f076 100644
@@ -32,6 +32,10 @@
  * checked against the expected value, and if they match, the entering of
  * optimized power state may be aborted.
  *
+ * @warning It is responsibility of the user to check if this function is
+ *   supported at runtime using `rte_cpu_get_features()` API call. Failing to do
+ *   so may result in an illegal CPU instruction error.
+ *
  * @param p
  *   Address to monitor for changes.
  * @param expected_value
@@ -69,6 +73,10 @@ static inline void rte_power_monitor(const volatile void *p,
  * This call will also lock a spinlock on entering sleep, and release it on
  * waking up the CPU.
  *
+ * @warning It is responsibility of the user to check if this function is
+ *   supported at runtime using `rte_cpu_get_features()` API call. Failing to do
+ *   so may result in an illegal CPU instruction error.
+ *
  * @param p
  *   Address to monitor for changes.
  * @param expected_value
@@ -101,6 +109,10 @@ static inline void rte_power_monitor_sync(const volatile void *p,
  * Enter an architecture-defined optimized power state until a certain TSC
  * timestamp is reached.
  *
+ * @warning It is responsibility of the user to check if this function is
+ *   supported at runtime using `rte_cpu_get_features()` API call. Failing to do
+ *   so may result in an illegal CPU instruction error.
+ *
  * @param tsc_timestamp
  *   Maximum TSC timestamp to wait for. Note that the wait behavior is
  *   architecture-dependent.
lib/librte_eal/ppc/rte_cpuflags.c +7 -0
index 3bb7563ce9..61db5c216d 100644
@@ -8,6 +8,7 @@
 #include <elf.h>
 #include <fcntl.h>
 #include <assert.h>
+#include <string.h>
 #include <unistd.h>
 
 /* Symbolic values for the entries in the auxiliary table */
@@ -108,3 +109,9 @@ rte_cpu_get_flag_name(enum rte_cpu_flag_t feature)
 		return NULL;
 	return rte_cpu_feature_table[feature].name;
 }
+
+void
+rte_cpu_get_intrinsics_support(struct rte_cpu_intrinsics *intrinsics)
+{
+	memset(intrinsics, 0, sizeof(*intrinsics));
+}
lib/librte_eal/version.map +1 -0
index c23ff57ce6..269cdccfd3 100644
@@ -402,6 +402,7 @@ EXPERIMENTAL {
 	rte_service_lcore_may_be_active;
 	rte_vect_get_max_simd_bitwidth;
 	rte_vect_set_max_simd_bitwidth;
+	rte_cpu_get_intrinsics_support;
 };
 
 INTERNAL {
lib/librte_eal/x86/rte_cpuflags.c +12 -0
index 0325c4b93b..a96312ff7f 100644
@@ -7,6 +7,7 @@
 #include <stdio.h>
 #include <errno.h>
 #include <stdint.h>
+#include <string.h>
 
 #include "rte_cpuid.h"
 
@@ -179,3 +180,14 @@ rte_cpu_get_flag_name(enum rte_cpu_flag_t feature)
 		return NULL;
 	return rte_cpu_feature_table[feature].name;
 }
+
+void
+rte_cpu_get_intrinsics_support(struct rte_cpu_intrinsics *intrinsics)
+{
+	memset(intrinsics, 0, sizeof(*intrinsics));
+
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_WAITPKG)) {
+		intrinsics->power_monitor = 1;
+		intrinsics->power_pause = 1;
+	}
+}
Signature · 1 line
2.17.1
#414 lore raw

PATCH v8 4/10 ethdev: add simple power management API

In reply to Anatoly Burakov

Add a simple API to allow getting address of next RX descriptor from the PMD, as well as release notes information.

---

Notes:
v8:
- Rename version map file name.

v7:
- Fixed queue ID validation
- Fixed documentation

v6:
- Rebase on top of latest main
- Ensure the API checks queue ID (Konstantin)
- Removed accidental inclusion of unrelated release notes
v5:
- Bring function format in line with other functions in the file
- Ensure the API is supported by the driver before calling it (Konstantin)

---
 doc/guides/rel_notes/release_20_11.rst |  5 +++++
 lib/librte_ethdev/rte_ethdev.c         | 23 +++++++++++++++++++++
 lib/librte_ethdev/rte_ethdev.h         | 28 ++++++++++++++++++++++++++
 lib/librte_ethdev/rte_ethdev_driver.h  | 28 ++++++++++++++++++++++++++
 lib/librte_ethdev/version.map          |  1 +
 5 files changed, 85 insertions(+)
doc/guides/rel_notes/release_20_11.rst +5 -0
index d8ac359e51..2827a000db 100644
@@ -139,6 +139,11 @@ New Features
   Hairpin Tx part flow rules can be inserted explicitly.
   New API is added to get the hairpin peer ports list.
 
+* **ethdev: add 1 new EXPERIMENTAL API for PMD power management.**
+
+  * ``rte_eth_get_wake_addr()``
+  * add new eth_dev_ops ``get_wake_addr``
+
 * **Updated Broadcom bnxt driver.**
 
   Updated the Broadcom bnxt driver with new features and improvements, including:
lib/librte_ethdev/rte_ethdev.c +23 -0
index b12bb3854d..4f3115fe8e 100644
@@ -5138,6 +5138,29 @@ rte_eth_tx_burst_mode_get(uint16_t port_id, uint16_t queue_id,
 		       dev->dev_ops->tx_burst_mode_get(dev, queue_id, mode));
 }
 
+int
+rte_eth_get_wake_addr(uint16_t port_id, uint16_t queue_id,
+		volatile void **wake_addr, uint64_t *expected, uint64_t *mask,
+		uint8_t *data_sz)
+{
+	struct rte_eth_dev *dev;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	dev = &rte_eth_devices[port_id];
+
+	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->get_wake_addr, -ENOTSUP);
+
+	if (queue_id >= dev->data->nb_rx_queues) {
+		RTE_ETHDEV_LOG(ERR, "Invalid RX queue_id=%u\n", queue_id);
+		return -EINVAL;
+	}
+
+	return eth_err(port_id,
+		dev->dev_ops->get_wake_addr(dev->data->rx_queues[queue_id],
+			wake_addr, expected, mask, data_sz));
+}
+
 int
 rte_eth_dev_set_mc_addr_list(uint16_t port_id,
 			     struct rte_ether_addr *mc_addr_set,
lib/librte_ethdev/rte_ethdev.h +28 -0
index e341a08817..11559e7bc8 100644
@@ -4364,6 +4364,34 @@ __rte_experimental
 int rte_eth_tx_burst_mode_get(uint16_t port_id, uint16_t queue_id,
 	struct rte_eth_burst_mode *mode);
 
+/**
+ * Retrieve the wake up address for the receive queue.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param queue_id
+ *   The Rx queue on the Ethernet device for which information will be
+ *   retrieved.
+ * @param wake_addr
+ *   The pointer to the address which will be monitored.
+ * @param expected
+ *   The pointer to value to be expected when descriptor is set.
+ * @param mask
+ *   The pointer to comparison bitmask for the expected value.
+ * @param data_sz
+ *   The pointer to data size for the expected value and comparison bitmask.
+ *
+ * @return
+ *   - 0: Success.
+ *   -ENOTSUP: Operation not supported.
+ *   -EINVAL: Invalid parameters.
+ *   -ENODEV: Invalid port ID.
+ */
+__rte_experimental
+int rte_eth_get_wake_addr(uint16_t port_id, uint16_t queue_id,
+		volatile void **wake_addr, uint64_t *expected, uint64_t *mask,
+		uint8_t *data_sz);
+
 /**
  * Retrieve device registers and register attributes (number of registers and
  * register size)
lib/librte_ethdev/rte_ethdev_driver.h +28 -0
index c63b9f7eb7..d7548dfe74 100644
@@ -752,6 +752,32 @@ typedef int (*eth_hairpin_queue_peer_unbind_t)
 	(struct rte_eth_dev *dev, uint16_t cur_queue, uint32_t direction);
 /**< @internal Unbind peer queue from the current queue. */
 
+/**
+ * @internal
+ * Get address of memory location whose contents will change whenever there is
+ * new data to be received on an RX queue.
+ *
+ * @param rxq
+ *   Ethdev queue pointer.
+ * @param tail_desc_addr
+ *   The pointer point to where the address will be stored.
+ * @param expected
+ *   The pointer point to value to be expected when descriptor is set.
+ * @param mask
+ *   The pointer point to comparison bitmask for the expected value.
+ * @param data_sz
+ *   Data size for the expected value (can be 1, 2, 4, or 8 bytes)
+ * @return
+ *   Negative errno value on error, 0 on success.
+ *
+ * @retval 0
+ *   Success
+ * @retval -EINVAL
+ *   Invalid parameters
+ */
+typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *mask, uint8_t *data_sz);
+
 /**
  * @internal A structure containing the functions exported by an Ethernet driver.
  */
@@ -910,6 +936,8 @@ struct eth_dev_ops {
 	/**< Set up the connection between the pair of hairpin queues. */
 	eth_hairpin_queue_peer_unbind_t hairpin_queue_peer_unbind;
 	/**< Disconnect the hairpin queues of a pair from each other. */
+	eth_get_wake_addr_t get_wake_addr;
+	/**< Get next RX queue ring entry address. */
 };
 
 /**
lib/librte_ethdev/version.map +1 -0
index 8ddda2547f..392c273712 100644
@@ -244,6 +244,7 @@ EXPERIMENTAL {
 	rte_flow_get_restore_info;
 	rte_flow_tunnel_action_decap_release;
 	rte_flow_tunnel_item_release;
+	rte_eth_get_wake_addr;
 };
 
 INTERNAL {
Signature · 1 line
2.17.1
#415 lore raw

PATCH v8 5/10 power: add PMD power management API and callback

In reply to Anatoly Burakov

Add a simple on/off switch that will enable saving power when no packets are arriving. It is based on counting the number of empty polls and, when the number reaches a certain threshold, entering an architecture-defined optimized power state that will either wait until a TSC timestamp expires, or when packets arrive.

This API mandates a core-to-single-queue mapping (that is, multiple queued per device are supported, but they have to be polled on different cores).

This design is using PMD RX callbacks.

1. UMWAIT/UMONITOR:

When a certain threshold of empty polls is reached, the core will go into a power optimized sleep while waiting on an address of next RX descriptor to be written to.

2. Pause instruction

Instead of move the core into deeper C state, this method uses the pause instruction to avoid busy polling.

3. Frequency scaling
Reuse existing DPDK power library to scale up/down core frequency depending on traffic volume.

---

Notes:
v8:
- Rename version map file name

v7:
- Fixed race condition (Konstantin)
- Slight rework of the structure of monitor code
- Added missing inline for wakeup

v6:
- Added wakeup mechanism for UMWAIT
- Removed memory allocation (everything is now allocated statically)
- Fixed various typos and comments
- Check for invalid queue ID
- Moved release notes to this patch

v5:
- Make error checking more robust
- Prevent initializing scaling if ACPI or PSTATE env wasn't set
- Prevent initializing UMWAIT path if PMD doesn't support get_wake_addr
- Add some debug logging
- Replace x86-specific code path to generic path using the intrinsic check

---
 doc/guides/rel_notes/release_20_11.rst |  11 +
 lib/librte_power/meson.build           |   5 +-
 lib/librte_power/rte_power_pmd_mgmt.c  | 320 +++++++++++++++++++++++++
 lib/librte_power/rte_power_pmd_mgmt.h  |  92 +++++++
 lib/librte_power/version.map           |   4 +
 5 files changed, 430 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_power/rte_power_pmd_mgmt.c
 create mode 100644 lib/librte_power/rte_power_pmd_mgmt.h
doc/guides/rel_notes/release_20_11.rst +11 -0
index 2827a000db..5f32a5da1d 100644
@@ -350,6 +350,17 @@ New Features
   * Replaced ``--scalar`` command-line option with ``--alg=<value>``, to allow
     the user to select the desired classify method.
 
+* **Add PMD power management mechanism**
+
+  3 new Ethernet PMD power management mechanism is added through existing
+  RX callback infrastructure.
+
+  * Add power saving scheme based on UMWAIT instruction (x86 only)
+  * Add power saving scheme based on ``rte_pause()``
+  * Add power saving scheme based on frequency scaling through the power library
+  * Add new EXPERIMENTAL API ``rte_power_pmd_mgmt_queue_enable()``
+  * Add new EXPERIMENTAL API ``rte_power_pmd_mgmt_queue_disable()``
+
 
 Removed Items
 -------------
lib/librte_power/meson.build +3 -2
index 78c031c943..cc3c7a8646 100644
@@ -9,6 +9,7 @@ sources = files('rte_power.c', 'power_acpi_cpufreq.c',
 		'power_kvm_vm.c', 'guest_channel.c',
 		'rte_power_empty_poll.c',
 		'power_pstate_cpufreq.c',
+		'rte_power_pmd_mgmt.c',
 		'power_common.c')
-headers = files('rte_power.h','rte_power_empty_poll.h')
-deps += ['timer']
+headers = files('rte_power.h','rte_power_empty_poll.h','rte_power_pmd_mgmt.h')
+deps += ['timer' ,'ethdev']
lib/librte_power/rte_power_pmd_mgmt.c new file +320 -0

320 lines not on this page · raw message

lib/librte_power/rte_power_pmd_mgmt.h new file +92 -0

92 lines not on this page · raw message

lib/librte_power/version.map +4 -0
index 69ca9af616..3f2f6cd6f6 100644
@@ -34,4 +34,8 @@ EXPERIMENTAL {
 	rte_power_guest_channel_receive_msg;
 	rte_power_poll_stat_fetch;
 	rte_power_poll_stat_update;
+	# added in 20.11
+	rte_power_pmd_mgmt_queue_enable;
+	rte_power_pmd_mgmt_queue_disable;
+
 };
Signature · 1 line
2.17.1
#416 lore raw

PATCH v8 6/10 net/ixgbe: implement power management API

In reply to Anatoly Burakov

Implement support for the power management API by implementing a get_wake_addr function that will return an address of an RX ring's status bit.

---
 drivers/net/ixgbe/ixgbe_ethdev.c |  1 +
 drivers/net/ixgbe/ixgbe_rxtx.c   | 25 +++++++++++++++++++++++++
 drivers/net/ixgbe/ixgbe_rxtx.h   |  2 ++
 3 files changed, 28 insertions(+)
drivers/net/ixgbe/ixgbe_ethdev.c +1 -0
index 00101c2eec..fcc4026372 100644
@@ -588,6 +588,7 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = {
 	.udp_tunnel_port_del  = ixgbe_dev_udp_tunnel_port_del,
 	.tm_ops_get           = ixgbe_tm_ops_get,
 	.tx_done_cleanup      = ixgbe_dev_tx_done_cleanup,
+	.get_wake_addr        = ixgbe_get_wake_addr,
 };
 
 /*
drivers/net/ixgbe/ixgbe_rxtx.c +25 -0
index d1d3baff90..096dff37ba 100644
@@ -1367,6 +1367,31 @@ const uint32_t
 		RTE_PTYPE_INNER_L3_IPV4_EXT | RTE_PTYPE_INNER_L4_UDP,
 };
 
+int ixgbe_get_wake_addr(void *rx_queue, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *mask, uint8_t *data_sz)
+{
+	volatile union ixgbe_adv_rx_desc *rxdp;
+	struct ixgbe_rx_queue *rxq = rx_queue;
+	uint16_t desc;
+
+	desc = rxq->rx_tail;
+	rxdp = &rxq->rx_ring[desc];
+	/* watch for changes in status bit */
+	*tail_desc_addr = &rxdp->wb.upper.status_error;
+
+	/*
+	 * we expect the DD bit to be set to 1 if this descriptor was already
+	 * written to.
+	 */
+	*expected = rte_cpu_to_le_32(IXGBE_RXDADV_STAT_DD);
+	*mask = rte_cpu_to_le_32(IXGBE_RXDADV_STAT_DD);
+
+	/* the registers are 32-bit */
+	*data_sz = 4;
+
+	return 0;
+}
+
 /* @note: fix ixgbe_dev_supported_ptypes_get() if any change here. */
 static inline uint32_t
 ixgbe_rxd_pkt_info_to_pkt_type(uint32_t pkt_info, uint16_t ptype_mask)
drivers/net/ixgbe/ixgbe_rxtx.h +2 -0
index 6d2f7c9da3..1ef0b05e66 100644
@@ -299,5 +299,7 @@ uint64_t ixgbe_get_tx_port_offloads(struct rte_eth_dev *dev);
 uint64_t ixgbe_get_rx_queue_offloads(struct rte_eth_dev *dev);
 uint64_t ixgbe_get_rx_port_offloads(struct rte_eth_dev *dev);
 uint64_t ixgbe_get_tx_queue_offloads(struct rte_eth_dev *dev);
+int ixgbe_get_wake_addr(void *rx_queue, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *mask, uint8_t *data_sz);
 
 #endif /* _IXGBE_RXTX_H_ */
Signature · 1 line
2.17.1
#417 lore raw

PATCH v8 7/10 net/i40e: implement power management API

In reply to Anatoly Burakov

Implement support for the power management API by implementing a get_wake_addr function that will return an address of an RX ring's status bit.

---
 drivers/net/i40e/i40e_ethdev.c |  1 +
 drivers/net/i40e/i40e_rxtx.c   | 26 ++++++++++++++++++++++++++
 drivers/net/i40e/i40e_rxtx.h   |  2 ++
 3 files changed, 29 insertions(+)
drivers/net/i40e/i40e_ethdev.c +1 -0
index 4778aaf299..358a38232b 100644
@@ -513,6 +513,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
 	.mtu_set                      = i40e_dev_mtu_set,
 	.tm_ops_get                   = i40e_tm_ops_get,
 	.tx_done_cleanup              = i40e_tx_done_cleanup,
+	.get_wake_addr	              = i40e_get_wake_addr,
 };
 
 /* store statistics names and its offset in stats structure */
drivers/net/i40e/i40e_rxtx.c +26 -0
index 5df9a9df56..78862fe3a2 100644
@@ -72,6 +72,32 @@
 #define I40E_TX_OFFLOAD_NOTSUP_MASK \
 		(PKT_TX_OFFLOAD_MASK ^ I40E_TX_OFFLOAD_MASK)
 
+int
+i40e_get_wake_addr(void *rx_queue, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *mask, uint8_t *data_sz)
+{
+	struct i40e_rx_queue *rxq = rx_queue;
+	volatile union i40e_rx_desc *rxdp;
+	uint16_t desc;
+
+	desc = rxq->rx_tail;
+	rxdp = &rxq->rx_ring[desc];
+	/* watch for changes in status bit */
+	*tail_desc_addr = &rxdp->wb.qword1.status_error_len;
+
+	/*
+	 * we expect the DD bit to be set to 1 if this descriptor was already
+	 * written to.
+	 */
+	*expected = rte_cpu_to_le_64(1 << I40E_RX_DESC_STATUS_DD_SHIFT);
+	*mask = rte_cpu_to_le_64(1 << I40E_RX_DESC_STATUS_DD_SHIFT);
+
+	/* registers are 64-bit */
+	*data_sz = 8;
+
+	return 0;
+}
+
 static inline void
 i40e_rxd_to_vlan_tci(struct rte_mbuf *mb, volatile union i40e_rx_desc *rxdp)
 {
drivers/net/i40e/i40e_rxtx.h +2 -0
index 57d7b4160b..5826cf1099 100644
@@ -248,6 +248,8 @@ uint16_t i40e_recv_scattered_pkts_vec_avx2(void *rx_queue,
 	struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
 uint16_t i40e_xmit_pkts_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
 	uint16_t nb_pkts);
+int i40e_get_wake_addr(void *rx_queue, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *value, uint8_t *data_sz);
 
 /* For each value it means, datasheet of hardware can tell more details
  *
Signature · 1 line
2.17.1
#418 lore raw

PATCH v8 8/10 net/ice: implement power management API

In reply to Anatoly Burakov

Implement support for the power management API by implementing a get_wake_addr function that will return an address of an RX ring's status bit.

---
 drivers/net/ice/ice_ethdev.c |  1 +
 drivers/net/ice/ice_rxtx.c   | 26 ++++++++++++++++++++++++++
 drivers/net/ice/ice_rxtx.h   |  2 ++
 3 files changed, 29 insertions(+)
drivers/net/ice/ice_ethdev.c +1 -0
index c65125ff32..54f185ad4d 100644
@@ -216,6 +216,7 @@ static const struct eth_dev_ops ice_eth_dev_ops = {
 	.udp_tunnel_port_add          = ice_dev_udp_tunnel_port_add,
 	.udp_tunnel_port_del          = ice_dev_udp_tunnel_port_del,
 	.tx_done_cleanup              = ice_tx_done_cleanup,
+	.get_wake_addr	              = ice_get_wake_addr,
 };
 
 /* store statistics names and its offset in stats structure */
drivers/net/ice/ice_rxtx.c +26 -0
index ee576c362a..fafd6ada62 100644
@@ -26,6 +26,32 @@ uint64_t rte_net_ice_dynflag_proto_xtr_ipv6_flow_mask;
 uint64_t rte_net_ice_dynflag_proto_xtr_tcp_mask;
 uint64_t rte_net_ice_dynflag_proto_xtr_ip_offset_mask;
 
+int ice_get_wake_addr(void *rx_queue, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *mask, uint8_t *data_sz)
+{
+	volatile union ice_rx_flex_desc *rxdp;
+	struct ice_rx_queue *rxq = rx_queue;
+	uint16_t desc;
+
+	desc = rxq->rx_tail;
+	rxdp = &rxq->rx_ring[desc];
+	/* watch for changes in status bit */
+	*tail_desc_addr = &rxdp->wb.status_error0;
+
+	/*
+	 * we expect the DD bit to be set to 1 if this descriptor was already
+	 * written to.
+	 */
+	*expected = rte_cpu_to_le_16(1 << ICE_RX_FLEX_DESC_STATUS0_DD_S);
+	*mask = rte_cpu_to_le_16(1 << ICE_RX_FLEX_DESC_STATUS0_DD_S);
+
+	/* register is 16-bit */
+	*data_sz = 2;
+
+	return 0;
+}
+
+
 static inline uint8_t
 ice_proto_xtr_type_to_rxdid(uint8_t xtr_type)
 {
drivers/net/ice/ice_rxtx.h +2 -0
index 1c23c7541e..7eeb8d467e 100644
@@ -250,6 +250,8 @@ uint16_t ice_xmit_pkts_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,
 				uint16_t nb_pkts);
 int ice_fdir_programming(struct ice_pf *pf, struct ice_fltr_desc *fdir_desc);
 int ice_tx_done_cleanup(void *txq, uint32_t free_cnt);
+int ice_get_wake_addr(void *rx_queue, volatile void **tail_desc_addr,
+		uint64_t *expected, uint64_t *mask, uint8_t *data_sz);
 
 #define FDIR_PARSING_ENABLE_PER_QUEUE(ad, on) do { \
 	int i; \
Signature · 1 line
2.17.1
#419 lore raw

PATCH v8 9/10 examples/l3fwd-power: enable PMD power mgmt

In reply to Anatoly Burakov

Add PMD power management feature support to l3fwd-power sample app.

---

Notes:
v8:
- Add return status check for queue enable

v6:
- Fixed typos in documentation

---
 .../sample_app_ug/l3_forward_power_man.rst    | 13 ++++++
 examples/l3fwd-power/main.c                   | 46 ++++++++++++++++++-
 2 files changed, 58 insertions(+), 1 deletion(-)
doc/guides/sample_app_ug/l3_forward_power_man.rst +13 -0
index d7e1dc5813..b10ebe662e 100644
@@ -109,6 +109,8 @@ where,
 
 *   --telemetry:  Telemetry mode.
 
+*   --pmd-mgmt: PMD power management mode.
+
 See :doc:`l3_forward` for details.
 The L3fwd-power example reuses the L3fwd command line options.
 
@@ -455,3 +457,14 @@ reference cycles and accordingly busy rate is set  to either 0% or
 
 The new stats ``empty_poll`` , ``full_poll`` and ``busy_percent`` can be viewed by running the script
 ``/usertools/dpdk-telemetry-client.py`` and selecting the menu option ``Send for global Metrics``.
+
+PMD power management Mode
+-------------------------
+
+The PMD power management  mode support for ``l3fwd-power`` is a standalone mode, in this mode
+``l3fwd-power`` does simple l3fwding along with enable the power saving scheme on specific
+port/queue/lcore. Main purpose for this mode is to demonstrate how to use the PMD power management API.
+
+.. code-block:: console
+
+        ./build/examples/dpdk-l3fwd-power -l 1-3 --  --pmd-mgmt -p 0x0f --config="(0,0,2),(0,1,3)"
examples/l3fwd-power/main.c +45 -1

100 lines not on this page · raw message

Signature · 1 line
2.17.1
#420 lore raw

PATCH v8 10/10 doc: update programmer's guide for power library

In reply to Anatoly Burakov

Update programmer's guide to document PMD power management usage.

---
 doc/guides/prog_guide/power_man.rst | 42 +++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)
doc/guides/prog_guide/power_man.rst +42 -0
index 0a3755a901..38c64d31e4 100644
@@ -192,6 +192,45 @@ User Cases
 ----------
 The mechanism can applied to any device which is based on polling. e.g. NIC, FPGA.
 
+PMD Power Management API
+------------------------
+
+Abstract
+~~~~~~~~
+Existing power management mechanisms require developers to change application
+design or change code to make use of it. The PMD power management API provides a
+convenient alternative by utilizing Ethernet PMD RX callbacks, and triggering
+power saving whenever empty poll count reaches a certain number.
+
+  * UMWAIT/UMONITOR
+
+   This power saving scheme will put the CPU into optimized power state and use
+   the UMWAIT/UMONITOR instructions to monitor the Ethernet PMD RX descriptor
+   address, and wake the CPU up whenever there's new traffic.
+
+  * Pause
+
+   This power saving scheme will use the `rte_pause` function to avoid busy
+   polling.
+
+  * Frequency scaling
+
+   This power saving scheme will use existing power library functionality to
+   scale the core frequency up/down depending on traffic volume.
+
+
+.. note::
+
+   Currently, this power management API is limited to mandatory mapping of 1
+   queue to 1 core (multiple queues are supported, but they must be polled from
+   different cores).
+
+API Overview for PMD Power Management
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* **Queue Enable**: Enable specific power scheme for certain queue/port/core
+
+* **Queue Disable**: Disable power scheme for certain queue/port/core
+
 References
 ----------
 
@@ -200,3 +239,6 @@ References
 
 *   The :doc:`../sample_app_ug/vm_power_management`
     chapter in the :doc:`../sample_app_ug/index` section.
+
+*   The :doc:`../sample_app_ug/rxtx_callbacks`
+    chapter in the :doc:`../sample_app_ug/index` section.
Signature · 1 line
2.17.1