Power-optimized RX for Ethernet devices
Subsystems (3)
PATCH v9 4/10 ethdev: add simple power management API
In reply to Liang MaAdd a simple API to allow getting address of next RX descriptor from the PMD, as well as release notes information.
---
Notes:
v8:
- Rename version map file name.
v7:
- Fixed queue ID validation
- Fixed documentation
v6:
- Rebase on top of latest main
- Ensure the API checks queue ID (Konstantin)
- Removed accidental inclusion of unrelated release notes
v5:
- Bring function format in line with other functions in the file
- Ensure the API is supported by the driver before calling it (Konstantin)
--- doc/guides/rel_notes/release_20_11.rst | 5 +++++ lib/librte_ethdev/rte_ethdev.c | 23 +++++++++++++++++++++ lib/librte_ethdev/rte_ethdev.h | 28 ++++++++++++++++++++++++++ lib/librte_ethdev/rte_ethdev_driver.h | 28 ++++++++++++++++++++++++++ lib/librte_ethdev/version.map | 1 + 5 files changed, 85 insertions(+)
doc/guides/rel_notes/release_20_11.rst +5 -0
@@ -139,6 +139,11 @@ New Features Hairpin Tx part flow rules can be inserted explicitly. New API is added to get the hairpin peer ports list. +* **ethdev: add 1 new EXPERIMENTAL API for PMD power management.** + + * ``rte_eth_get_wake_addr()`` + * add new eth_dev_ops ``get_wake_addr`` + * **Updated Broadcom bnxt driver.** Updated the Broadcom bnxt driver with new features and improvements, including:
lib/librte_ethdev/rte_ethdev.c +23 -0
@@ -5138,6 +5138,29 @@ rte_eth_tx_burst_mode_get(uint16_t port_id, uint16_t queue_id, dev->dev_ops->tx_burst_mode_get(dev, queue_id, mode)); } +int +rte_eth_get_wake_addr (uint16_t port_id,uint16_t queue_id, +volatile void **wake_addr,uint64_t *expected,uint64_t *mask, +uint8_t *data_sz) +{ +struct rte_eth_dev *dev; + +RTE_ETH_VALID_PORTID_OR_ERR_RET (port_id, -ENODEV ); + + dev = &rte_eth_devices[port_id]; + +RTE_FUNC_PTR_OR_ERR_RET (*dev->dev_ops->get_wake_addr, -ENOTSUP ); + +if (queue_id >= dev->data->nb_rx_queues) { +RTE_ETHDEV_LOG (ERR ,"Invalid RX queue_id=%u\n" , queue_id); +return -EINVAL ; + } + +return eth_err (port_id, + dev->dev_ops->get_wake_addr (dev->data->rx_queues[queue_id], + wake_addr, expected, mask, data_sz)); +} + int rte_eth_dev_set_mc_addr_list(uint16_t port_id, struct rte_ether_addr *mc_addr_set,
lib/librte_ethdev/rte_ethdev.h +28 -0
@@ -4364,6 +4364,34 @@ __rte_experimentalint rte_eth_tx_burst_mode_get (uint16_t port_id,uint16_t queue_id,struct rte_eth_burst_mode *mode); +/** +* Retrieve the wake up address for the receive queue. +* +* @param port_id +* The port identifier of the Ethernet device. +* @param queue_id +* The Rx queue on the Ethernet device for which information will be +* retrieved. +* @param wake_addr +* The pointer to the address which will be monitored. +* @param expected +* The pointer to value to be expected when descriptor is set. +* @param mask +* The pointer to comparison bitmask for the expected value. +* @param data_sz +* The pointer to data size for the expected value and comparison bitmask. +* +* @return +* - 0: Success. +* -ENOTSUP: Operation not supported. +* -EINVAL: Invalid parameters. +* -ENODEV: Invalid port ID. +*/ +__rte_experimental +int rte_eth_get_wake_addr(uint16_t port_id, uint16_t queue_id, +volatile void **wake_addr,uint64_t *expected,uint64_t *mask, +uint8_t *data_sz); +/** * Retrieve device registers and register attributes (number of registers and * register size)
lib/librte_ethdev/rte_ethdev_driver.h +28 -0
@@ -752,6 +752,32 @@ typedef int (*eth_hairpin_queue_peer_unbind_t) (struct rte_eth_dev *dev, uint16_t cur_queue, uint32_t direction);/**< @internal Unbind peer queue from the current queue. */ +/** +* @internal +* Get address of memory location whose contents will change whenever there is +* new data to be received on an RX queue. +* +* @param rxq +* Ethdev queue pointer. +* @param tail_desc_addr +* The pointer point to where the address will be stored. +* @param expected +* The pointer point to value to be expected when descriptor is set. +* @param mask +* The pointer point to comparison bitmask for the expected value. +* @param data_sz +* Data size for the expected value (can be 1, 2, 4, or 8 bytes) +* @return +* Negative errno value on error, 0 on success. +* +* @retval 0 +* Success +* @retval -EINVAL +* Invalid parameters +*/ +typedef int (*eth_get_wake_addr_t )(void *rxq,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *mask,uint8_t *data_sz); +/** * @internal A structure containing the functions exported by an Ethernet driver. */
@@ -910,6 +936,8 @@ struct eth_dev_ops {/**< Set up the connection between the pair of hairpin queues. */ eth_hairpin_queue_peer_unbind_t hairpin_queue_peer_unbind;/**< Disconnect the hairpin queues of a pair from each other. */ +eth_get_wake_addr_t get_wake_addr; +/**< Get next RX queue ring entry address. */ };/**
lib/librte_ethdev/version.map +1 -0
@@ -244,6 +244,7 @@ EXPERIMENTAL { rte_flow_get_restore_info; rte_flow_tunnel_action_decap_release; rte_flow_tunnel_item_release; + rte_eth_get_wake_addr; }; INTERNAL {
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API
24/10/2020 01:06, Liang Ma:
Quoted 6 lines
Add a simple API to allow getting address of next RX descriptor from the PMD, as well as release notes information.
Signed-off-by Liang Ma <[email protected]>Signed-off-by Anatoly Burakov <[email protected]>
You keep forgetting Cc ethdev maintainers
(it is automatic when using --cc-cmd devtools/get-maintainer.sh). As a result we still don't have any feedback from Ferruh and Andrew. And I believe such API requires having feedback from maintainers of other NIC drivers. I tried to explain this concern already in email and community meetings, but I see no progress.
Below are my comments.
doc/guides/rel_notes/release_20_11.rst New Features · line 142
Quoted 7 lines
doc/guides/rel_notes/release_20_11.rst +1
-0@@ -139,6 +139,11 @@ New Features Hairpin Tx part flow rules can be inserted explicitly. New API is added to get the hairpin peer ports list. +* **ethdev: add 1 new EXPERIMENTAL API for PMD power management.**
The guidelines at the top of the file say using past tense. No need to mention it is experimental as any new API.
doc/guides/rel_notes/release_20_11.rst New Features · line 145
Quoted 3 lines
doc/guides/rel_notes/release_20_11.rst +3
-0+ + * ``rte_eth_get_wake_addr()`` + * add new eth_dev_ops ``get_wake_addr``
No need to mention the dev_ops in the release notes. Better to explain what it brings from an user perspective.
lib/librte_ethdev/rte_ethdev.h __rte_experimental · line 4368
Quoted 4 lines
lib/librte_ethdev/rte_ethdev.h +2
-0+/** +* Retrieve the wake up address for the receive queue.
I guess how this function should be used,
but a bit more explanations would not hurt here.
Quoted 8 lines
+ *
+ * @paramport_id
+ * The port identifier of the Ethernet device.
+ * @paramqueue_id
+ * The Rx queue on the Ethernet device for which information will be
+ * retrieved.
+ * @paramwake_addr
+ * The pointer to the address which will be monitored.
This function does not make the address monitored, right?
Quoted 2 lines
+ * @param expected
+ * The pointer to value to be expected when descriptor is set.
Not sure we should restrict it to a "descriptor".
Expecting a value or some bits looks too much restrictive. I understand it probably fits well for Intel NICs, but in the general case, we can imagine that any change in a byte array could be a wake up signal.
Quoted 4 lines
+ * @param mask
+ * The pointer to comparison bitmask for the expected value.
+ * @paramdata_sz
+ * The pointer to data size for the expected value and comparison bitmask.
It is not clear that above 4 parameters are filled by the driver.
Quoted 11 lines
+ *
+ * @return+ * - 0: Success. + * -ENOTSUP: Operation not supported. + * -EINVAL: Invalid parameters. + * -ENODEV: Invalid port ID.+ */
lib/librte_ethdev/rte_ethdev.h +4
-0+__rte_experimental +int rte_eth_get_wake_addr(uint16_t port_id, uint16_t queue_id, +volatile void **wake_addr,uint64_t *expected,uint64_t *mask, +uint8_t *data_sz);
[...]
lib/librte_ethdev/version.map EXPERIMENTAL { · line 248
Quoted 8 lines
lib/librte_ethdev/version.map +1
-0@@ -244,6 +244,7 @@ EXPERIMENTAL { rte_flow_get_restore_info; rte_flow_tunnel_action_decap_release; rte_flow_tunnel_item_release; + rte_eth_get_wake_addr; };
Please sort in alphabetical order.
Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API
<snip>
thanks for your information.
Sorry for that.
All related maintainer(include other NIC PMD) will be Cced in v10.
Quoted 15 lines
You keep forgetting Cc ethdev maintainers
(it is automatic when using --cc-cmddevtools/get-maintainer.sh). As a result we still don't have any feedback from Ferruh and Andrew. And I believe such API requires having feedback from maintainers of other NIC drivers. I tried to explain this concern already in email and community meetings, but I see no progress.Below are my comments.
Quoted 6 lines
doc/guides/rel_notes/release_20_11.rst +0
-0@@ -139,6 +139,11 @@ New Features Hairpin Tx part flow rules can be inserted explicitly. New API is added to get the hairpin peer ports list.
Quoted 4 lines
doc/guides/rel_notes/release_20_11.rst New Features · line 142
Quoted 1 line
doc/guides/rel_notes/release_20_11.rst +1
-0+* **ethdev: add 1 new EXPERIMENTAL API for PMD power management.**The guidelines at the top of the file say using past tense. No need to mention it is experimental as any new API.
agree
Quoted 7 lines
doc/guides/rel_notes/release_20_11.rst New Features · line 145
Quoted 4 lines
doc/guides/rel_notes/release_20_11.rst +3
-0+ + * ``rte_eth_get_wake_addr()`` + * add new eth_dev_ops ``get_wake_addr``No need to mention the
dev_opsin the release notes. Better to explain what it brings from an user perspective.
agree
Quoted 8 lines
lib/librte_ethdev/rte_ethdev.h __rte_experimental · line 4368
Quoted 5 lines
lib/librte_ethdev/rte_ethdev.h +2
-0+/** +* Retrieve the wake up address for the receive queue. I guess how this function should be used,
but a bit more explanations would not hurt here.
agree
Quoted 10 lines
Quoted 8 lines
+ *
+ * @paramport_id
+ * The port identifier of the Ethernet device.
+ * @paramqueue_id
+ * The Rx queue on the Ethernet device for which information will be
+ * retrieved.
+ * @paramwake_addr
+ * The pointer to the address which will be monitored.This function does not make the address monitored, right?
This function only get the target wakeup address. that does not monitor this address.
Quoted 5 lines
Quoted 3 lines
+ * @param expected
+ * The pointer to value to be expected when descriptor is set.Not sure we should restrict it to a "descriptor".
actully that is not limited to a descriptor, any writeback content should work.
Quoted 5 lines
Expecting a value or some bits looks too much restrictive. I understand it probably fits well for Intel NICs, but in the general case, we can imagine that any change in a byte array could be a wake up signal.
this parameter doesn not limited user how to use it.
In fact, current design can support any bits change within 64 bits content.
Quoted 20 lines
Quoted 5 lines
+ * @param mask
+ * The pointer to comparison bitmask for the expected value.
+ * @paramdata_sz
+ * The pointer to data size for the expected value and comparison bitmask.It is not clear that above 4 parameters are filled by the driver.
Quoted 11 lines
+ *
+ * @return+ * - 0: Success. + * -ENOTSUP: Operation not supported. + * -EINVAL: Invalid parameters. + * -ENODEV: Invalid port ID.+ */
lib/librte_ethdev/rte_ethdev.h +4
-0+__rte_experimental +int rte_eth_get_wake_addr(uint16_t port_id, uint16_t queue_id, +volatile void **wake_addr,uint64_t *expected,uint64_t *mask, +uint8_t *data_sz);[...]
Quoted 10 lines
lib/librte_ethdev/version.map EXPERIMENTAL { · line 248
Quoted 8 lines
lib/librte_ethdev/version.map +1
-0@@ -244,6 +244,7 @@ EXPERIMENTAL { rte_flow_get_restore_info; rte_flow_tunnel_action_decap_release; rte_flow_tunnel_item_release; + rte_eth_get_wake_addr; };Please sort in alphabetical order.
agree will do
Quoted 2 lines
Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API
27/10/2020 12:15, Liang, Ma:
Quoted 26 lines
Quoted 7 lines
lib/librte_ethdev/rte_ethdev.h __rte_experimental · line 4368
Quoted 4 lines
lib/librte_ethdev/rte_ethdev.h +2
-0+/** +* Retrieve the wake up address for the receive queue. I guess how this function should be used,
but a bit more explanations would not hurt here.agree
Quoted 10 lines
Quoted 8 lines
+ *
+ * @paramport_id
+ * The port identifier of the Ethernet device.
+ * @paramqueue_id
+ * The Rx queue on the Ethernet device for which information will be
+ * retrieved.
+ * @paramwake_addr
+ * The pointer to the address which will be monitored.This function does not make the address monitored, right?
This function only get the target wakeup address. that does not monitor this address.
Quoted 5 lines
Quoted 3 lines
+ * @param expected
+ * The pointer to value to be expected when descriptor is set.Not sure we should restrict it to a "descriptor".
actully that is not limited to a descriptor, any writeback content should work.
Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Expecting a value or some bits looks too much restrictive. I understand it probably fits well for Intel NICs, but in the general case, we can imagine that any change in a byte array could be a wake up signal.
this parameter doesn not limited user how to use it.
In fact, current design can support any bits change within 64 bits content.
How the driver can specify that any value change should be monitored? I understand that it is only a value/mask pair,
it does not give room for "any value".
Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API
Quoted 34 lines
27/10/2020 12:15, Liang, Ma:
Quoted 31 lines
Quoted 7 lines
lib/librte_ethdev/rte_ethdev.h __rte_experimental · line 4368
Quoted 4 lines
lib/librte_ethdev/rte_ethdev.h +2
-0+/** +* Retrieve the wake up address for the receive queue. I guess how this function should be used,
but a bit more explanations would not hurt here.agree
Quoted 10 lines
Quoted 8 lines
+ *
+ * @paramport_id
+ * The port identifier of the Ethernet device.
+ * @paramqueue_id
+ * The Rx queue on the Ethernet device for which information will be
+ * retrieved.
+ * @paramwake_addr
+ * The pointer to the address which will be monitored.This function does not make the address monitored, right?
This function only get the target wakeup address. that does not monitor this address.
Quoted 5 lines
Quoted 3 lines
+ * @param expected
+ * The pointer to value to be expected when descriptor is set.Not sure we should restrict it to a "descriptor".
actully that is not limited to a descriptor, any writeback content should work.
Quoted 5 lines
Expecting a value or some bits looks too much restrictive. I understand it probably fits well for Intel NICs, but in the general case, we can imagine that any change in a byte array could be a wake up signal.
Quoted 6 lines
Quoted 2 lines
this parameter doesn not limited user how to use it.
In fact, current design can support any bits change within 64 bits content.How the driver can specify that any value change should be monitored? I understand that it is only a value/mask pair,
it does not give room for "any value".
As I can read the code, value=0, mask=0 will provide you with 'any value'. Though it would mean that rte_power_monitor() will *always* go into sleep, so not sure what will be there any practical usage for such case. Konstantin
Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API
27/10/2020 18:43, Ananyev, Konstantin:
Quoted 35 lines
Quoted 35 lines
27/10/2020 12:15, Liang, Ma:
Quoted 33 lines
Quoted 7 lines
lib/librte_ethdev/rte_ethdev.h __rte_experimental · line 4368
Quoted 4 lines
lib/librte_ethdev/rte_ethdev.h +2
-0+/** +* Retrieve the wake up address for the receive queue. I guess how this function should be used,
but a bit more explanations would not hurt here.agree
Quoted 10 lines
Quoted 8 lines
+ *
+ * @paramport_id
+ * The port identifier of the Ethernet device.
+ * @paramqueue_id
+ * The Rx queue on the Ethernet device for which information will be
+ * retrieved.
+ * @paramwake_addr
+ * The pointer to the address which will be monitored.This function does not make the address monitored, right?
This function only get the target wakeup address. that does not monitor this address.
Quoted 5 lines
Quoted 3 lines
+ * @param expected
+ * The pointer to value to be expected when descriptor is set.Not sure we should restrict it to a "descriptor".
actully that is not limited to a descriptor, any writeback content should work.
Quoted 5 lines
Expecting a value or some bits looks too much restrictive. I understand it probably fits well for Intel NICs, but in the general case, we can imagine that any change in a byte array could be a wake up signal.
this parameter doesn not limited user how to use it.
In fact, current design can support any bits change within 64 bits content.
Quoted 7 lines
Quoted 3 lines
How the driver can specify that any value change should be monitored? I understand that it is only a value/mask pair,
it does not give room for "any value".As I can read the code, value=0, mask=0 will provide you with 'any value'. Though it would mean that
rte_power_monitor()will *always* go into sleep, so not sure what will be there any practical usage for such case.
I think what is missing is to allow waking up when the value of a byte array is changing, without specifiying any value.
Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API
Quoted 56 lines
-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Tuesday, October 27, 2020 6:31 PM
To: Ma, Liang J <[email protected]>; Ananyev, Konstantin <[email protected]>
Cc: [email protected]; Burakov, Anatoly <[email protected]>; [email protected]; Zhang, Qi Z <[email protected]>; [email protected]; Xing, Beilei <[email protected]>; Guo, Jia <[email protected]>; Yang, Qiming <[email protected]>; Wang, Haiyue <[email protected]>; Richardson, Bruce <[email protected]>; Hunt, David <[email protected]>; [email protected]; [email protected]; McDaniel, Timothy <[email protected]>; Eads, Gage <[email protected]>; [email protected]; Andrew Rybchenko <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API27/10/2020 18:43, Ananyev, Konstantin:
Quoted 39 lines
Quoted 38 lines
27/10/2020 12:15, Liang, Ma:
Quoted 26 lines
Quoted 7 lines
lib/librte_ethdev/rte_ethdev.h __rte_experimental · line 4368
Quoted 4 lines
lib/librte_ethdev/rte_ethdev.h +2
-0+/** +* Retrieve the wake up address for the receive queue. I guess how this function should be used,
but a bit more explanations would not hurt here.agree
Quoted 10 lines
Quoted 8 lines
+ *
+ * @paramport_id
+ * The port identifier of the Ethernet device.
+ * @paramqueue_id
+ * The Rx queue on the Ethernet device for which information will be
+ * retrieved.
+ * @paramwake_addr
+ * The pointer to the address which will be monitored.This function does not make the address monitored, right?
This function only get the target wakeup address. that does not monitor this address.
Quoted 5 lines
Quoted 3 lines
+ * @param expected
+ * The pointer to value to be expected when descriptor is set.Not sure we should restrict it to a "descriptor".
actully that is not limited to a descriptor, any writeback content should work.
Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Expecting a value or some bits looks too much restrictive. I understand it probably fits well for Intel NICs, but in the general case, we can imagine that any change in a byte array could be a wake up signal.
this parameter doesn not limited user how to use it.
In fact, current design can support any bits change within 64 bits content.How the driver can specify that any value change should be monitored? I understand that it is only a value/mask pair,
it does not give room for "any value".
Quoted 6 lines
Quoted 3 lines
As I can read the code, value=0, mask=0 will provide you with 'any value'. Though it would mean that
rte_power_monitor()will *always* go into sleep, so not sure what will be there any practical usage for such case.I think what is missing is to allow waking up when the value of a byte array is changing, without specifiying any value.
I think it will always wakeup on any write to wait_addr.
What you control with value/mask pair - when we should go to sleep. In other words:
ret = rte_eth_get_wake_addr(port, queue, &wait_addr, &value, &mask, ....);
mask==0: always go to sleep, wakeup at any store to wait_addr.
mask!=0: go to sleep only if (*wait_addr & mask) == value, wakeup at any store to wait_addr.
Liang, Anatoly - feel free to correct me here, if I missed something.
Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API
Quoted 65 lines · Konstantin
Quoted 57 lines
-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Tuesday, October 27, 2020 6:31 PM
To: Ma, Liang J <[email protected]>; Ananyev, Konstantin <[email protected]>
Cc: [email protected]; Burakov, Anatoly <[email protected]>; [email protected]; Zhang, Qi Z <[email protected]>; [email protected]; Xing, Beilei <[email protected]>; Guo, Jia <[email protected]>; Yang, Qiming <[email protected]>; Wang, Haiyue <[email protected]>; Richardson, Bruce <[email protected]>; Hunt, David <[email protected]>; [email protected]; [email protected]; McDaniel, Timothy <[email protected]>; Eads, Gage <[email protected]>; [email protected]; Andrew Rybchenko <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API27/10/2020 18:43, Ananyev, Konstantin:
Quoted 39 lines
Quoted 38 lines
27/10/2020 12:15, Liang, Ma:
Quoted 26 lines
Quoted 7 lines
lib/librte_ethdev/rte_ethdev.h
Quoted 4 lines
lib/librte_ethdev/rte_ethdev.h +2
-0+/** +* Retrieve the wake up address for the receive queue. I guess how this function should be used,
but a bit more explanations would not hurt here.agree
Quoted 10 lines
Quoted 8 lines
+ *
+ * @paramport_id
+ * The port identifier of the Ethernet device.
+ * @paramqueue_id
+ * The Rx queue on the Ethernet device for which information will be
+ * retrieved.
+ * @paramwake_addr
+ * The pointer to the address which will be monitored.This function does not make the address monitored, right?
This function only get the target wakeup address. that does not monitor this address.
Quoted 5 lines
Quoted 3 lines
+ * @param expected
+ * The pointer to value to be expected when descriptor is set.Not sure we should restrict it to a "descriptor".
actully that is not limited to a descriptor, any writeback content should work.
Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Expecting a value or some bits looks too much restrictive. I understand it probably fits well for Intel NICs, but in the general case, we can imagine that any change in a byte array could be a wake up signal.
this parameter doesn not limited user how to use it.
In fact, current design can support any bits change within 64 bits content.How the driver can specify that any value change should be monitored? I understand that it is only a value/mask pair,
it does not give room for "any value".Quoted 6 lines
Quoted 3 lines
As I can read the code, value=0, mask=0 will provide you with 'any value'. Though it would mean that
rte_power_monitor()will *always* go into sleep, so not sure what will be there any practical usage for such case.I think what is missing is to allow waking up when the value of a byte array is changing, without specifiying any value.
Quoted 7 lines
I think it will always wakeup on any write to
wait_addr.
What you control with value/mask pair - when we should go to sleep. In other words:
ret =rte_eth_get_wake_addr(port, queue, &wait_addr, &value, &mask, ....);mask==0: always go to sleep, wakeup at any store to
wait_addr.
mask!=0: go to sleep only if (*wait_addr& mask) == value, wakeup at any store towait_addr.
I did not get this impression on reading it first time. Till you put it this way.
The comment "if the masked value is already matching, abort" stumped me.
Quoted 3 lines
Liang, Anatoly - feel free to correct me here, if I missed something.
Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API
In reply to Ananyev, KonstantinQuoted 69 lines · Konstantin
Quoted 56 lines
-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Tuesday, October 27, 2020 6:31 PM
To: Ma, Liang J <[email protected]>; Ananyev, Konstantin <[email protected]>
Cc: [email protected]; Burakov, Anatoly <[email protected]>; [email protected]; Zhang, Qi Z <[email protected]>; [email protected]; Xing, Beilei <[email protected]>; Guo, Jia <[email protected]>; Yang, Qiming <[email protected]>; Wang, Haiyue <[email protected]>; Richardson, Bruce <[email protected]>; Hunt, David <[email protected]>; [email protected]; [email protected]; McDaniel, Timothy <[email protected]>; Eads, Gage <[email protected]>; [email protected]; Andrew Rybchenko <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [dpdk-dev] [PATCH v9 04/10] ethdev: add simple power management API27/10/2020 18:43, Ananyev, Konstantin:
Quoted 39 lines
Quoted 38 lines
27/10/2020 12:15, Liang, Ma:
Quoted 26 lines
Quoted 7 lines
lib/librte_ethdev/rte_ethdev.h
Quoted 4 lines
lib/librte_ethdev/rte_ethdev.h +2
-0+/** +* Retrieve the wake up address for the receive queue. I guess how this function should be used,
but a bit more explanations would not hurt here.agree
Quoted 10 lines
Quoted 8 lines
+ *
+ * @paramport_id
+ * The port identifier of the Ethernet device.
+ * @paramqueue_id
+ * The Rx queue on the Ethernet device for which information will be
+ * retrieved.
+ * @paramwake_addr
+ * The pointer to the address which will be monitored.This function does not make the address monitored, right?
This function only get the target wakeup address. that does not monitor this address.
Quoted 5 lines
Quoted 3 lines
+ * @param expected
+ * The pointer to value to be expected when descriptor is set.Not sure we should restrict it to a "descriptor".
actully that is not limited to a descriptor, any writeback content should work.
Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Expecting a value or some bits looks too much restrictive. I understand it probably fits well for Intel NICs, but in the general case, we can imagine that any change in a byte array could be a wake up signal.
this parameter doesn not limited user how to use it.
In fact, current design can support any bits change within 64 bits content.How the driver can specify that any value change should be monitored? I understand that it is only a value/mask pair,
it does not give room for "any value".Quoted 6 lines
Quoted 3 lines
As I can read the code, value=0, mask=0 will provide you with 'any value'. Though it would mean that
rte_power_monitor()will *always* go into sleep, so not sure what will be there any practical usage for such case.I think what is missing is to allow waking up when the value of a byte array is changing, without specifiying any value.
I think it will always wakeup on any write to
wait_addr.
What you control with value/mask pair - when we should go to sleep. In other words:
ret =rte_eth_get_wake_addr(port, queue, &wait_addr, &value, &mask, ....);
Quoted 4 lines
mask==0: always go to sleep, wakeup at any store to
wait_addr.
mask!=0: go to sleep only if (*wait_addr& mask) == value, wakeup at any store towait_addr.Liang, Anatoly - feel free to correct me here, if I missed something.
mask and expected value is used to prevent race condition with NIC. there is a interval between code get target address and issue umwait action. therefore, the mask is a selector of the bits, if the current value & mask already == the expected value just before issue the umwait, that mean that address is no longer useful for umwait because of DMA has happend.
then the code path is abort and back to normal logic.
PATCH v9 5/10 power: add PMD power management API and callback
In reply to Liang MaAdd a simple on/off switch that will enable saving power when no packets are arriving. It is based on counting the number of empty polls and, when the number reaches a certain threshold, entering an architecture-defined optimized power state that will either wait until a TSC timestamp expires, or when packets arrive.
This API mandates a core-to-single-queue mapping (that is, multiple queued per device are supported, but they have to be polled on different cores).
This design is using PMD RX callbacks.
1. UMWAIT/UMONITOR:
When a certain threshold of empty polls is reached, the core will go into a power optimized sleep while waiting on an address of next RX descriptor to be written to.
2. Pause instruction
Instead of move the core into deeper C state, this method uses the pause instruction to avoid busy polling.
3. Frequency scaling
Reuse existing DPDK power library to scale up/down core frequency depending on traffic volume.
---
Notes:
v8:
- Rename version map file name
v7:
- Fixed race condition (Konstantin)
- Slight rework of the structure of monitor code
- Added missing inline for wakeup
v6:
- Added wakeup mechanism for UMWAIT
- Removed memory allocation (everything is now allocated statically)
- Fixed various typos and comments
- Check for invalid queue ID
- Moved release notes to this patch
v5:
- Make error checking more robust
- Prevent initializing scaling if ACPI or PSTATE env wasn't set
- Prevent initializing UMWAIT path if PMD doesn't support get_wake_addr
- Add some debug logging
- Replace x86-specific code path to generic path using the intrinsic check
--- doc/guides/rel_notes/release_20_11.rst | 11 + lib/librte_power/meson.build | 5 +- lib/librte_power/rte_power_pmd_mgmt.c | 320 +++++++++++++++++++++++++ lib/librte_power/rte_power_pmd_mgmt.h | 92 +++++++ lib/librte_power/version.map | 4 + 5 files changed, 430 insertions(+), 2 deletions(-) create mode 100644 lib/librte_power/rte_power_pmd_mgmt.c create mode 100644 lib/librte_power/rte_power_pmd_mgmt.h
doc/guides/rel_notes/release_20_11.rst +11 -0
@@ -350,6 +350,17 @@ New Features * Replaced ``--scalar`` command-line option with ``--alg=<value>``, to allow the user to select the desired classify method. +* **Add PMD power management mechanism** + + 3 new Ethernet PMD power management mechanism is added through existing + RX callback infrastructure. + + * Add power saving scheme based on UMWAIT instruction (x86 only) + * Add power saving scheme based on ``rte_pause()`` + * Add power saving scheme based on frequency scaling through the power library + * Add new EXPERIMENTAL API ``rte_power_pmd_mgmt_queue_enable()`` + * Add new EXPERIMENTAL API ``rte_power_pmd_mgmt_queue_disable()`` + Removed Items -------------
lib/librte_power/meson.build +3 -2
@@ -9,6 +9,7 @@ sources = files('rte_power.c', 'power_acpi_cpufreq.c', 'power_kvm_vm.c', 'guest_channel.c', 'rte_power_empty_poll.c', 'power_pstate_cpufreq.c', + 'rte_power_pmd_mgmt.c', 'power_common.c') -headers = files('rte_power.h','rte_power_empty_poll.h') -deps += ['timer'] +headers = files('rte_power.h','rte_power_empty_poll.h','rte_power_pmd_mgmt.h') +deps += ['timer' ,'ethdev']
lib/librte_power/rte_power_pmd_mgmt.c new file +320 -0
320 lines not on this page · raw message
lib/librte_power/rte_power_pmd_mgmt.h new file +92 -0
92 lines not on this page · raw message
lib/librte_power/version.map +4 -0
@@ -34,4 +34,8 @@ EXPERIMENTAL { rte_power_guest_channel_receive_msg; rte_power_poll_stat_fetch; rte_power_poll_stat_update; + # added in 20.11 + rte_power_pmd_mgmt_queue_enable; + rte_power_pmd_mgmt_queue_disable; + };
Signature · 1 line
2.17.1
PATCH v9 6/10 net/ixgbe: implement power management API
In reply to Liang MaImplement support for the power management API by implementing a get_wake_addr function that will return an address of an RX ring's status bit.
--- drivers/net/ixgbe/ixgbe_ethdev.c | 1 + drivers/net/ixgbe/ixgbe_rxtx.c | 25 +++++++++++++++++++++++++ drivers/net/ixgbe/ixgbe_rxtx.h | 2 ++ 3 files changed, 28 insertions(+)
drivers/net/ixgbe/ixgbe_ethdev.c +1 -0
@@ -588,6 +588,7 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = { .udp_tunnel_port_del = ixgbe_dev_udp_tunnel_port_del, .tm_ops_get = ixgbe_tm_ops_get, .tx_done_cleanup = ixgbe_dev_tx_done_cleanup, + .get_wake_addr = ixgbe_get_wake_addr, };/*
drivers/net/ixgbe/ixgbe_rxtx.c +25 -0
@@ -1367,6 +1367,31 @@ const uint32_t RTE_PTYPE_INNER_L3_IPV4_EXT | RTE_PTYPE_INNER_L4_UDP, }; +int ixgbe_get_wake_addr (void *rx_queue,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *mask,uint8_t *data_sz) +{ +volatile union ixgbe_adv_rx_desc *rxdp; +struct ixgbe_rx_queue *rxq = rx_queue; +uint16_t desc; + + desc = rxq->rx_tail; + rxdp = &rxq->rx_ring[desc]; +/* watch for changes in status bit */ + *tail_desc_addr = &rxdp->wb.upper.status_error; + +/* +* we expect the DD bit to be set to 1 if this descriptor was already +* written to. +*/ + *expected =rte_cpu_to_le_32 (IXGBE_RXDADV_STAT_DD ); + *mask =rte_cpu_to_le_32 (IXGBE_RXDADV_STAT_DD ); + +/* the registers are 32-bit */ + *data_sz =4 ; + +return 0 ; +} +/* @note: fix ixgbe_dev_supported_ptypes_get() if any change here. */ static inline uint32_t ixgbe_rxd_pkt_info_to_pkt_type (uint32_t pkt_info,uint16_t ptype_mask)
drivers/net/ixgbe/ixgbe_rxtx.h +2 -0
@@ -299,5 +299,7 @@ uint64_t ixgbe_get_tx_port_offloads(struct rte_eth_dev *dev);uint64_t ixgbe_get_rx_queue_offloads (struct rte_eth_dev *dev);uint64_t ixgbe_get_rx_port_offloads (struct rte_eth_dev *dev);uint64_t ixgbe_get_tx_queue_offloads (struct rte_eth_dev *dev); +int ixgbe_get_wake_addr (void *rx_queue,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *mask,uint8_t *data_sz);#endif /* _IXGBE_RXTX_H_ */
Signature · 1 line
2.17.1
PATCH v9 7/10 net/i40e: implement power management API
In reply to Liang MaImplement support for the power management API by implementing a get_wake_addr function that will return an address of an RX ring's status bit.
--- drivers/net/i40e/i40e_ethdev.c | 1 + drivers/net/i40e/i40e_rxtx.c | 26 ++++++++++++++++++++++++++ drivers/net/i40e/i40e_rxtx.h | 2 ++ 3 files changed, 29 insertions(+)
drivers/net/i40e/i40e_ethdev.c +1 -0
@@ -513,6 +513,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = { .mtu_set = i40e_dev_mtu_set, .tm_ops_get = i40e_tm_ops_get, .tx_done_cleanup = i40e_tx_done_cleanup, + .get_wake_addr = i40e_get_wake_addr, };/* store statistics names and its offset in stats structure */
drivers/net/i40e/i40e_rxtx.c +26 -0
@@ -72,6 +72,32 @@#define I40E_TX_OFFLOAD_NOTSUP_MASK \ (PKT_TX_OFFLOAD_MASK ^ I40E_TX_OFFLOAD_MASK) +int +i40e_get_wake_addr (void *rx_queue,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *mask,uint8_t *data_sz) +{ +struct i40e_rx_queue *rxq = rx_queue; +volatile union i40e_rx_desc *rxdp; +uint16_t desc; + + desc = rxq->rx_tail; + rxdp = &rxq->rx_ring[desc]; +/* watch for changes in status bit */ + *tail_desc_addr = &rxdp->wb.qword1.status_error_len; + +/* +* we expect the DD bit to be set to 1 if this descriptor was already +* written to. +*/ + *expected =rte_cpu_to_le_64 (1 <<I40E_RX_DESC_STATUS_DD_SHIFT ); + *mask =rte_cpu_to_le_64 (1 <<I40E_RX_DESC_STATUS_DD_SHIFT ); + +/* registers are 64-bit */ + *data_sz =8 ; + +return 0 ; +} +static inline void i40e_rxd_to_vlan_tci (struct rte_mbuf *mb,volatile union i40e_rx_desc *rxdp) {
drivers/net/i40e/i40e_rxtx.h +2 -0
@@ -248,6 +248,8 @@ uint16_t i40e_recv_scattered_pkts_vec_avx2(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);uint16_t i40e_xmit_pkts_vec_avx2 (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts); +int i40e_get_wake_addr (void *rx_queue,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *value,uint8_t *data_sz);/* For each value it means, datasheet of hardware can tell more details *
Signature · 1 line
2.17.1
PATCH v9 8/10 net/ice: implement power management API
In reply to Liang MaImplement support for the power management API by implementing a get_wake_addr function that will return an address of an RX ring's status bit.
--- drivers/net/ice/ice_ethdev.c | 1 + drivers/net/ice/ice_rxtx.c | 26 ++++++++++++++++++++++++++ drivers/net/ice/ice_rxtx.h | 2 ++ 3 files changed, 29 insertions(+)
drivers/net/ice/ice_ethdev.c +1 -0
@@ -216,6 +216,7 @@ static const struct eth_dev_ops ice_eth_dev_ops = { .udp_tunnel_port_add = ice_dev_udp_tunnel_port_add, .udp_tunnel_port_del = ice_dev_udp_tunnel_port_del, .tx_done_cleanup = ice_tx_done_cleanup, + .get_wake_addr = ice_get_wake_addr, };/* store statistics names and its offset in stats structure */
drivers/net/ice/ice_rxtx.c +26 -0
@@ -26,6 +26,32 @@ uint64_t rte_net_ice_dynflag_proto_xtr_ipv6_flow_mask;uint64_t rte_net_ice_dynflag_proto_xtr_tcp_mask;uint64_t rte_net_ice_dynflag_proto_xtr_ip_offset_mask; +int ice_get_wake_addr (void *rx_queue,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *mask,uint8_t *data_sz) +{ +volatile union ice_rx_flex_desc *rxdp; +struct ice_rx_queue *rxq = rx_queue; +uint16_t desc; + + desc = rxq->rx_tail; + rxdp = &rxq->rx_ring[desc]; +/* watch for changes in status bit */ + *tail_desc_addr = &rxdp->wb.status_error0; + +/* +* we expect the DD bit to be set to 1 if this descriptor was already +* written to. +*/ + *expected =rte_cpu_to_le_16 (1 <<ICE_RX_FLEX_DESC_STATUS0_DD_S ); + *mask =rte_cpu_to_le_16 (1 <<ICE_RX_FLEX_DESC_STATUS0_DD_S ); + +/* register is 16-bit */ + *data_sz =2 ; + +return 0 ; +} + +static inline uint8_t ice_proto_xtr_type_to_rxdid (uint8_t xtr_type) {
drivers/net/ice/ice_rxtx.h +2 -0
@@ -250,6 +250,8 @@ uint16_t ice_xmit_pkts_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);int ice_fdir_programming (struct ice_pf *pf,struct ice_fltr_desc *fdir_desc);int ice_tx_done_cleanup (void *txq,uint32_t free_cnt); +int ice_get_wake_addr (void *rx_queue,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *mask,uint8_t *data_sz);#define FDIR_PARSING_ENABLE_PER_QUEUE (ad, on) do { \ int i; \
Signature · 1 line
2.17.1
PATCH v9 9/10 examples/l3fwd-power: enable PMD power mgmt
In reply to Liang MaAdd PMD power management feature support to l3fwd-power sample app.
---
Notes:
v8:
- Add return status check for queue enable
v6:
- Fixed typos in documentation
--- .../sample_app_ug/l3_forward_power_man.rst | 13 ++++++ examples/l3fwd-power/main.c | 46 ++++++++++++++++++- 2 files changed, 58 insertions(+), 1 deletion(-)
doc/guides/sample_app_ug/l3_forward_power_man.rst +13 -0
@@ -109,6 +109,8 @@ where, * --telemetry: Telemetry mode. +* --pmd-mgmt: PMD power management mode. + See :doc:`l3_forward` for details. The L3fwd-power example reuses the L3fwd command line options.
@@ -455,3 +457,14 @@ reference cycles and accordingly busy rate is set to either 0% or The new stats ``empty_poll`` , ``full_poll`` and ``busy_percent`` can be viewed by running the script ``/usertools/dpdk-telemetry-client.py`` and selecting the menu option ``Send for global Metrics``. + +PMD power management Mode +------------------------- + +The PMD power management mode support for ``l3fwd-power`` is a standalone mode, in this mode +``l3fwd-power`` does simple l3fwding along with enable the power saving scheme on specific +port/queue/lcore. Main purpose for this mode is to demonstrate how to use the PMD power management API. + +.. code-block:: console + + ./build/examples/dpdk-l3fwd-power -l 1-3 -- --pmd-mgmt -p 0x0f --config="(0,0,2),(0,1,3)"
examples/l3fwd-power/main.c +45 -1
100 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v9 10/10 doc: update programmer's guide for power library
In reply to Liang MaUpdate programmer's guide to document PMD power management usage.
--- doc/guides/prog_guide/power_man.rst | 42 +++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)
doc/guides/prog_guide/power_man.rst +42 -0
@@ -192,6 +192,45 @@ User Cases ---------- The mechanism can applied to any device which is based on polling. e.g. NIC, FPGA. +PMD Power Management API +------------------------ + +Abstract +~~~~~~~~ +Existing power management mechanisms require developers to change application +design or change code to make use of it. The PMD power management API provides a +convenient alternative by utilizing Ethernet PMD RX callbacks, and triggering +power saving whenever empty poll count reaches a certain number. + + * UMWAIT/UMONITOR + + This power saving scheme will put the CPU into optimized power state and use + the UMWAIT/UMONITOR instructions to monitor the Ethernet PMD RX descriptor + address, and wake the CPU up whenever there's new traffic. + + * Pause + + This power saving scheme will use the `rte_pause` function to avoid busy + polling. + + * Frequency scaling + + This power saving scheme will use existing power library functionality to + scale the core frequency up/down depending on traffic volume. + + +.. note:: + + Currently, this power management API is limited to mandatory mapping of 1 + queue to 1 core (multiple queues are supported, but they must be polled from + different cores). + +API Overview for PMD Power Management +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +* **Queue Enable**: Enable specific power scheme for certain queue/port/core + +* **Queue Disable**: Disable power scheme for certain queue/port/core + References ----------
@@ -200,3 +239,6 @@ References * The :doc:`../sample_app_ug/vm_power_management` chapter in the :doc:`../sample_app_ug/index` section. + +* The :doc:`../sample_app_ug/rxtx_callbacks` + chapter in the :doc:`../sample_app_ug/index` section.
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v9 10/10] doc: update programmer's guide for power library
24/10/2020 01:06, Liang Ma:
Quoted 8 lines
Update programmer's guide to document PMD power management usage.
Signed-off-by Liang Ma <[email protected]>Signed-off-by Anatoly Burakov <[email protected]>Acked-by David Hunt <[email protected]>--- doc/guides/prog_guide/power_man.rst | 42 +++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)
Why don't you update this doc in the patch 5
adding the code in the power library?
doc/guides/prog_guide/power_man.rst User Cases · line 197
Quoted 9 lines
doc/guides/prog_guide/power_man.rst +3
-0@@ -192,6 +192,45 @@ User Cases ---------- The mechanism can applied to any device which is based on polling. e.g. NIC, FPGA. +PMD Power Management API +------------------------ +
Blank line missing below:
doc/guides/prog_guide/power_man.rst User Cases · line 204
Quoted 7 lines
doc/guides/prog_guide/power_man.rst +7
-0+Abstract +~~~~~~~~ +Existing power management mechanisms require developers to change application +design or change code to make use of it. The PMD power management API provides a +convenient alternative by utilizing Ethernet PMD RX callbacks, and triggering +power saving whenever empty poll count reaches a certain number. +
Here you start a list of schemes, without introducing it.
Quoted 15 lines
doc/guides/prog_guide/power_man.rst +15
-0+ * UMWAIT/UMONITOR + + This power saving scheme will put the CPU into optimized power state and use + the UMWAIT/UMONITOR instructions to monitor the Ethernet PMD RX descriptor + address, and wake the CPU up whenever there's new traffic. + + * Pause + + This power saving scheme will use the `rte_pause` function to avoid busy + polling. + + * Frequency scaling + + This power saving scheme will use existing power library functionality to + scale the core frequency up/down depending on traffic volume.
doc/guides/prog_guide/power_man.rst User Cases · line 229
Quoted 10 lines
doc/guides/prog_guide/power_man.rst +10
-0+ + +.. note:: + + Currently, this power management API is limited to mandatory mapping of 1 + queue to 1 core (multiple queues are supported, but they must be polled from + different cores). + +API Overview for PMD Power Management +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Underlining is long enough...
... but a blank line is missing after the title.
Quoted 3 lines
doc/guides/prog_guide/power_man.rst +3
-0+* **Queue Enable**: Enable specific power scheme for certain queue/port/core + +* **Queue Disable**: Disable power scheme for certain queue/port/core
Re: [dpdk-dev] [PATCH v9 10/10] doc: update programmer's guide for power library
Hi Thomas,
all is fixed in v10.
Regards
Liang
Quoted 71 lines · Thomas Monjalon
24/10/2020 01:06, Liang Ma:
Quoted 8 lines
Update programmer's guide to document PMD power management usage.
Signed-off-by Liang Ma <[email protected]>Signed-off-by Anatoly Burakov <[email protected]>Acked-by David Hunt <[email protected]>--- doc/guides/prog_guide/power_man.rst | 42 +++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)Why don't you update this doc in the patch 5
adding the code in the power library?doc/guides/prog_guide/power_man.rst User Cases · line 197
Quoted 9 lines
doc/guides/prog_guide/power_man.rst +3
-0@@ -192,6 +192,45 @@ User Cases ---------- The mechanism can applied to any device which is based on polling. e.g. NIC, FPGA. +PMD Power Management API +------------------------ +Blank line missing below:
doc/guides/prog_guide/power_man.rst User Cases · line 204
Quoted 7 lines
doc/guides/prog_guide/power_man.rst +7
-0+Abstract +~~~~~~~~ +Existing power management mechanisms require developers to change application +design or change code to make use of it. The PMD power management API provides a +convenient alternative by utilizing Ethernet PMD RX callbacks, and triggering +power saving whenever empty poll count reaches a certain number. +Here you start a list of schemes, without introducing it.
Quoted 15 lines
doc/guides/prog_guide/power_man.rst +15
-0+ * UMWAIT/UMONITOR + + This power saving scheme will put the CPU into optimized power state and use + the UMWAIT/UMONITOR instructions to monitor the Ethernet PMD RX descriptor + address, and wake the CPU up whenever there's new traffic. + + * Pause + + This power saving scheme will use the `rte_pause` function to avoid busy + polling. + + * Frequency scaling + + This power saving scheme will use existing power library functionality to + scale the core frequency up/down depending on traffic volume.doc/guides/prog_guide/power_man.rst User Cases · line 229
Quoted 10 lines
doc/guides/prog_guide/power_man.rst +10
-0+ + +.. note:: + + Currently, this power management API is limited to mandatory mapping of 1 + queue to 1 core (multiple queues are supported, but they must be polled from + different cores). + +API Overview for PMD Power Management +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Underlining is long enough...
... but a blank line is missing after the title.Quoted 3 lines
doc/guides/prog_guide/power_man.rst +3
-0+* **Queue Enable**: Enable specific power scheme for certain queue/port/core + +* **Queue Disable**: Disable power scheme for certain queue/port/core
v10 cover Add PMD power mgmt
In reply to Liang MaThis patchset proposes a simple API for Ethernet drivers to cause the CPU to enter a power-optimized state while waiting for packets to arrive, along with a set of generic intrinsics that facilitate that. This is achieved through cooperation with the NIC driver that will allow us to know address of wake up event, and wait for writes on it.
On IA, this is achieved through using UMONITOR/UMWAIT instructions. They are used in their raw opcode form because there is no widespread compiler support for them yet. Still, the API is made generic enough to hopefully support other architectures, if they happen to implement similar instructions.
To achieve power savings, there is a very simple mechanism
used: we're counting empty polls, and if a certain threshold is reached, we get the address of next RX ring descriptor from the NIC driver, arm the monitoring hardware, and enter a power-optimized state. We will then wake up when either a timeout happens, or a write happens (or generally whenever CPU feels like waking up - this is platform- specific), and proceed as normal. The empty poll counter is reset whenever we actually get packets, so we only go to sleep when we know nothing is going on. The mechanism is generic which can be used for any write back descriptor.
Why are we putting it into ethdev as opposed to leaving this up to the application? Our customers specifically requested a way to do it wit minimal changes to the application code. The current approach allows to just flip a switch and automatically have power savings.
- Only 1:1 core to queue mapping is supported, meaning that each lcore must at most handle RX on a single queue
- Support 3 type policies. UMWAIT/PAUSE/Frequency_Scale
- Power management is enabled per-queue
- The API doesn't extend to other device types
Liang Ma (9):
eal: add new x86 cpuid support for WAITPKG
eal: add power management intrinsics
eal: add intrinsics support check infrastructure
ethdev: add simple power management API
power: add PMD power management API and callback net/ixgbe: implement power management API net/i40e: implement power management API net/ice: implement power management API examples/l3fwd-power: enable PMD power mgmt
doc/guides/prog_guide/power_man.rst | 48 +++ doc/guides/rel_notes/release_20_11.rst | 15 + .../sample_app_ug/l3_forward_power_man.rst | 13 + drivers/net/i40e/i40e_ethdev.c | 1 + drivers/net/i40e/i40e_rxtx.c | 26 ++ drivers/net/i40e/i40e_rxtx.h | 2 + drivers/net/ice/ice_ethdev.c | 1 + drivers/net/ice/ice_rxtx.c | 26 ++ drivers/net/ice/ice_rxtx.h | 2 + drivers/net/ixgbe/ixgbe_ethdev.c | 1 + drivers/net/ixgbe/ixgbe_rxtx.c | 25 ++ drivers/net/ixgbe/ixgbe_rxtx.h | 2 + examples/l3fwd-power/main.c | 46 ++- lib/librte_eal/arm/include/meson.build | 1 + .../arm/include/rte_power_intrinsics.h | 60 ++++ lib/librte_eal/arm/rte_cpuflags.c | 6 + lib/librte_eal/include/generic/rte_cpuflags.h | 26 ++ .../include/generic/rte_power_intrinsics.h | 123 +++++++ lib/librte_eal/include/meson.build | 1 + lib/librte_eal/ppc/include/meson.build | 1 + .../ppc/include/rte_power_intrinsics.h | 60 ++++ lib/librte_eal/ppc/rte_cpuflags.c | 7 + lib/librte_eal/version.map | 1 + lib/librte_eal/x86/include/meson.build | 1 + lib/librte_eal/x86/include/rte_cpuflags.h | 1 + .../x86/include/rte_power_intrinsics.h | 135 ++++++++ lib/librte_eal/x86/rte_cpuflags.c | 14 + lib/librte_ethdev/rte_ethdev.c | 23 ++ lib/librte_ethdev/rte_ethdev.h | 33 ++ lib/librte_ethdev/rte_ethdev_driver.h | 28 ++ lib/librte_ethdev/version.map | 1 + lib/librte_power/meson.build | 5 +- lib/librte_power/rte_power_pmd_mgmt.c | 320 ++++++++++++++++++ lib/librte_power/rte_power_pmd_mgmt.h | 92 +++++ lib/librte_power/version.map | 4 + 35 files changed, 1148 insertions(+), 3 deletions(-) create mode 100644 lib/librte_eal/arm/include/rte_power_intrinsics.h create mode 100644 lib/librte_eal/include/generic/rte_power_intrinsics.h create mode 100644 lib/librte_eal/ppc/include/rte_power_intrinsics.h create mode 100644 lib/librte_eal/x86/include/rte_power_intrinsics.h create mode 100644 lib/librte_power/rte_power_pmd_mgmt.c create mode 100644 lib/librte_power/rte_power_pmd_mgmt.h
Signature · 1 line
2.17.1
PATCH v10 1/9 eal: add new x86 cpuid support for WAITPKG
Add a new CPUID flag indicating processor support for UMONITOR/UMWAIT and TPAUSE instructions instruction.
--- lib/librte_eal/x86/include/rte_cpuflags.h | 1 + lib/librte_eal/x86/rte_cpuflags.c | 2 ++ 2 files changed, 3 insertions(+)
lib/librte_eal/x86/include/rte_cpuflags.h +1 -0
@@ -132,6 +132,7 @@ enum rte_cpu_flag_t {RTE_CPUFLAG_MOVDIR64B ,/**< Direct Store Instructions 64B */ RTE_CPUFLAG_AVX512VP2INTERSECT ,/**< AVX512 Two Register Intersection */ +RTE_CPUFLAG_WAITPKG ,/**< UMONITOR/UMWAIT/TPAUSE */ /* The last item */ RTE_CPUFLAG_NUMFLAGS, /**< This should always be the last! */ };
lib/librte_eal/x86/rte_cpuflags.c +2 -0
@@ -110,6 +110,8 @@ const struct feature_entry rte_cpu_feature_table[] = { FEAT_DEF(AVX512F, 0x00000007, 0, RTE_REG_EBX, 16) FEAT_DEF(RDSEED, 0x00000007, 0, RTE_REG_EBX, 18) + FEAT_DEF(WAITPKG, 0x00000007, 0, RTE_REG_ECX, 5) + FEAT_DEF(LAHF_SAHF, 0x80000001, 0, RTE_REG_ECX, 0)FEAT_DEF (LZCNT ,0x80000001 ,0 ,RTE_REG_ECX ,4 )
Signature · 1 line
2.17.1
PATCH v10 2/9 eal: add power management intrinsics
In reply to Liang MaAdd two new power management intrinsics, and provide an implementation in eal/x86 based on UMONITOR/UMWAIT instructions. The instructions are implemented as raw byte opcodes because there is not yet widespread compiler support for these instructions.
The power management instructions provide an architecture-specific function to either wait until a specified TSC timestamp is reached, or optionally wait until either a TSC timestamp is reached or a memory location is written to. The monitor function also provides an optional comparison, to avoid sleeping when the expected write has already happened, and no more writes are expected.
For more details, please refer to Intel(R) 64 and IA-32 Architectures Software Developer's Manual, Volume 2.
--- lib/librte_eal/arm/include/meson.build | 1 + .../arm/include/rte_power_intrinsics.h | 60 ++++++++ .../include/generic/rte_power_intrinsics.h | 111 ++++++++++++++ lib/librte_eal/include/meson.build | 1 + lib/librte_eal/ppc/include/meson.build | 1 + .../ppc/include/rte_power_intrinsics.h | 60 ++++++++ lib/librte_eal/x86/include/meson.build | 1 + .../x86/include/rte_power_intrinsics.h | 135 ++++++++++++++++++ 8 files changed, 370 insertions(+) create mode 100644 lib/librte_eal/arm/include/rte_power_intrinsics.h create mode 100644 lib/librte_eal/include/generic/rte_power_intrinsics.h create mode 100644 lib/librte_eal/ppc/include/rte_power_intrinsics.h create mode 100644 lib/librte_eal/x86/include/rte_power_intrinsics.h
lib/librte_eal/arm/include/meson.build +1 -0
@@ -20,6 +20,7 @@ arch_headers = files( 'rte_pause_32.h', 'rte_pause_64.h', 'rte_pause.h', + 'rte_power_intrinsics.h', 'rte_prefetch_32.h', 'rte_prefetch_64.h', 'rte_prefetch.h',
lib/librte_eal/arm/include/rte_power_intrinsics.h new file +60 -0
60 lines not on this page · raw message
lib/librte_eal/include/generic/rte_power_intrinsics.h new file +111 -0
111 lines not on this page · raw message
lib/librte_eal/include/meson.build +1 -0
@@ -60,6 +60,7 @@ generic_headers = files( 'generic/rte_memcpy.h', 'generic/rte_pause.h', 'generic/rte_prefetch.h', + 'generic/rte_power_intrinsics.h', 'generic/rte_rwlock.h', 'generic/rte_spinlock.h', 'generic/rte_ticketlock.h',
lib/librte_eal/ppc/include/meson.build +1 -0
@@ -10,6 +10,7 @@ arch_headers = files( 'rte_io.h', 'rte_memcpy.h', 'rte_pause.h', + 'rte_power_intrinsics.h', 'rte_prefetch.h', 'rte_rwlock.h', 'rte_spinlock.h',
lib/librte_eal/ppc/include/rte_power_intrinsics.h new file +60 -0
60 lines not on this page · raw message
lib/librte_eal/x86/include/meson.build +1 -0
@@ -13,6 +13,7 @@ arch_headers = files( 'rte_io.h', 'rte_memcpy.h', 'rte_prefetch.h', + 'rte_power_intrinsics.h', 'rte_pause.h', 'rte_rtm.h', 'rte_rwlock.h',
lib/librte_eal/x86/include/rte_power_intrinsics.h new file +135 -0
135 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v10 2/9] eal: add power management intrinsics
27/10/2020 15:59, Liang Ma:
Quoted 8 lines
lib/librte_eal/x86/include/rte_power_intrinsics.h new file +8
-0+static inline uint64_t +__get_umwait_val (const volatile void *p,const uint8_t sz) +{ +switch (sz) { +case sizeof (uint8_t ): +return *(const volatile uint8_t *)p; +case sizeof (uint16_t ): +return *(const volatile uint16_t *)p;
lib/librte_eal/x86/include/rte_power_intrinsics.h line 33
Quoted 10 lines
lib/librte_eal/x86/include/rte_power_intrinsics.h new file +10
-0+case sizeof (uint32_t ): +return *(const volatile uint32_t *)p; +case sizeof (uint64_t ): +return *(const volatile uint64_t *)p; +default : +/* this is an intrinsic, so we can't have any error handling */ +RTE_ASSERT (0 ); +return 0 ; + } +}
This function is going to pollute the namespace.
I will fix it to __rte_power_get_umwait_val().
PATCH v10 3/9 eal: add intrinsics support check infrastructure
In reply to Liang MaCurrently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.
This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.
--
Notes:
v8:
- Rename eal version.map
v6:
- Fix the comments
--- lib/librte_eal/arm/rte_cpuflags.c | 6 +++++ lib/librte_eal/include/generic/rte_cpuflags.h | 26 +++++++++++++++++++ .../include/generic/rte_power_intrinsics.h | 12 +++++++++ lib/librte_eal/ppc/rte_cpuflags.c | 7 +++++ lib/librte_eal/version.map | 1 + lib/librte_eal/x86/rte_cpuflags.c | 12 +++++++++ 6 files changed, 64 insertions(+)
lib/librte_eal/arm/rte_cpuflags.c +6 -0
@@ -151,3 +151,9 @@ rte_cpu_get_flag_name(enum rte_cpu_flag_t feature)return NULL ;return rte_cpu_feature_table[feature].name; } + +void +rte_cpu_get_intrinsics_support (struct rte_cpu_intrinsics *intrinsics) +{ +memset (intrinsics,0 ,sizeof (*intrinsics)); +}
lib/librte_eal/include/generic/rte_cpuflags.h +26 -0
@@ -13,6 +13,32 @@#include "rte_common.h" #include <errno.h> +#include <rte_compat.h> + +/** +* Structure used to describe platform-specific intrinsics that may or may not +* be supported at runtime. +*/ +struct rte_cpu_intrinsics { +uint32_t power_monitor :1 ; +/**< indicates support for rte_power_monitor function */ +uint32_t power_pause :1 ; +/**< indicates support for rte_power_pause function */ +}; + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Check CPU support for various intrinsics at runtime. +* +* @param intrinsics +* Pointer to a structure to be filled. +*/ +__rte_experimental +void +rte_cpu_get_intrinsics_support (struct rte_cpu_intrinsics *intrinsics); +/** * Enumeration of all CPU features supported */
lib/librte_eal/include/generic/rte_power_intrinsics.h +12 -0
@@ -32,6 +32,10 @@* checked against the expected value, and if they match, the entering of * optimized power state may be aborted. * +* @warning It is responsibility of the user to check if this function is +* supported at runtime using `rte_cpu_get_features()` API call. Failing to do +* so may result in an illegal CPU instruction error. +* * @param p * Address to monitor for changes. * @param expected_value
@@ -69,6 +73,10 @@ static inline void rte_power_monitor(const volatile void *p,* This call will also lock a spinlock on entering sleep, and release it on * waking up the CPU. * +* @warning It is responsibility of the user to check if this function is +* supported at runtime using `rte_cpu_get_features()` API call. Failing to do +* so may result in an illegal CPU instruction error. +* * @param p * Address to monitor for changes. * @param expected_value
@@ -101,6 +109,10 @@ static inline void rte_power_monitor_sync(const volatile void *p,* Enter an architecture-defined optimized power state until a certain TSC * timestamp is reached. * +* @warning It is responsibility of the user to check if this function is +* supported at runtime using `rte_cpu_get_features()` API call. Failing to do +* so may result in an illegal CPU instruction error. +* * @param tsc_timestamp * Maximum TSC timestamp to wait for. Note that the wait behavior is * architecture-dependent.
lib/librte_eal/ppc/rte_cpuflags.c +7 -0
@@ -8,6 +8,7 @@#include <elf.h> #include <fcntl.h> #include <assert.h> +#include <string.h> #include <unistd.h> /* Symbolic values for the entries in the auxiliary table */
@@ -108,3 +109,9 @@ rte_cpu_get_flag_name(enum rte_cpu_flag_t feature)return NULL ;return rte_cpu_feature_table[feature].name; } + +void +rte_cpu_get_intrinsics_support (struct rte_cpu_intrinsics *intrinsics) +{ +memset (intrinsics,0 ,sizeof (*intrinsics)); +}
lib/librte_eal/version.map +1 -0
@@ -402,6 +402,7 @@ EXPERIMENTAL { rte_service_lcore_may_be_active; rte_vect_get_max_simd_bitwidth; rte_vect_set_max_simd_bitwidth; + rte_cpu_get_intrinsics_support; }; INTERNAL {
lib/librte_eal/x86/rte_cpuflags.c +12 -0
@@ -7,6 +7,7 @@#include <stdio.h> #include <errno.h> #include <stdint.h> +#include <string.h> #include "rte_cpuid.h"
@@ -179,3 +180,14 @@ rte_cpu_get_flag_name(enum rte_cpu_flag_t feature)return NULL ;return rte_cpu_feature_table[feature].name; } + +void +rte_cpu_get_intrinsics_support (struct rte_cpu_intrinsics *intrinsics) +{ +memset (intrinsics,0 ,sizeof (*intrinsics)); + +if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_WAITPKG )) { + intrinsics->power_monitor =1 ; + intrinsics->power_pause =1 ; + } +}
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure
Quoted 11 lines · Liang Ma
Currently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.
This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.
Quoted 6 lines
Signed-off-by Anatoly Burakov <[email protected]>Signed-off-by Liang Ma <[email protected]>Acked-by Jerin Jacob <[email protected]>Acked-by Ruifeng Wang <[email protected]>Acked-by Ray Kinsella <[email protected]>
Coming late to the party, it seems crowded...
Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +2
-0@@ -13,6 +13,32 @@#include "rte_common.h" #include <errno.h> +#include <rte_compat.h> +
lib/librte_eal/include/generic/rte_cpuflags.h line 27
Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +10
-0+/** +* Structure used to describe platform-specific intrinsics that may or may not +* be supported at runtime. +*/ +struct rte_cpu_intrinsics { +uint32_t power_monitor :1 ; +/**< indicates support for rte_power_monitor function */ +uint32_t power_pause :1 ; +/**< indicates support for rte_power_pause function */ +};
- The rte_power library is supposed to be built on top of cpuflags. Not the other way around.
Those capabilities should have been kept inside the rte_power_ API and not pollute the cpuflags API.
- All of this should have come as a single patch as the previously introduced API is unusable before.
Quoted 22 lines
lib/librte_eal/include/generic/rte_cpuflags.h +14
-0+ +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Check CPU support for various intrinsics at runtime. +* +* @param intrinsics +* Pointer to a structure to be filled. +*/ +__rte_experimental +void +rte_cpu_get_intrinsics_support (struct rte_cpu_intrinsics *intrinsics); +/** * Enumeration of all CPU features supported */ lib/librte_eal/include/generic/rte_power_intrinsics.h +0
-0
lib/librte_eal/include/generic/rte_power_intrinsics.h line 38
Quoted 7 lines
lib/librte_eal/include/generic/rte_power_intrinsics.h +4
-0@@ -32,6 +32,10 @@* checked against the expected value, and if they match, the entering of * optimized power state may be aborted. * +* @warning It is responsibility of the user to check if this function is +* supported at runtime using `rte_cpu_get_features()` API call. Failing to do +* so may result in an illegal CPU instruction error. +*
- Reading this API description... what am I supposed to do in my application or driver who wants to use the new rte_power_monitor/rte_power_pause stuff?
I should call rte_cpu_get_features(TOTO) ?
This comment does not give a hint.
I suppose the intent was to refer to the rte_cpu_get_intrinsics_support() thing. This must be fixed.
- Again, I wonder why we are exposing all this stuff. This should be hidden in the rte_power API.
Signature · 1 line
David Marchand
Re: [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure
Quoted 90 lines · David Marchand
Quoted 11 lines · Liang Ma
Currently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.
This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.
Quoted 6 lines
Signed-off-by Anatoly Burakov <[email protected]>Signed-off-by Liang Ma <[email protected]>Acked-by Jerin Jacob <[email protected]>Acked-by Ruifeng Wang <[email protected]>Acked-by Ray Kinsella <[email protected]>Coming late to the party, it seems crowded...
Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +2
-0@@ -13,6 +13,32 @@#include "rte_common.h" #include <errno.h> +#include <rte_compat.h> +lib/librte_eal/include/generic/rte_cpuflags.h line 27
Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +10
-0+/** +* Structure used to describe platform-specific intrinsics that may or may not +* be supported at runtime. +*/ +struct rte_cpu_intrinsics { +uint32_t power_monitor :1 ; +/**< indicates support for rte_power_monitor function */ +uint32_t power_pause :1 ; +/**< indicates support for rte_power_pause function */ +};- The
rte_powerlibrary is supposed to be built on top of cpuflags. Not the other way around.
Those capabilities should have been kept inside therte_power_API and not pollute the cpuflags API.- All of this should have come as a single patch as the previously introduced API is unusable before.
Quoted 22 lines
lib/librte_eal/include/generic/rte_cpuflags.h +14
-0+ +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Check CPU support for various intrinsics at runtime. +* +* @param intrinsics +* Pointer to a structure to be filled. +*/ +__rte_experimental +void +rte_cpu_get_intrinsics_support (struct rte_cpu_intrinsics *intrinsics); +/** * Enumeration of all CPU features supported */ lib/librte_eal/include/generic/rte_power_intrinsics.h +0
-0lib/librte_eal/include/generic/rte_power_intrinsics.h line 38
Quoted 7 lines
lib/librte_eal/include/generic/rte_power_intrinsics.h +4
-0@@ -32,6 +32,10 @@* checked against the expected value, and if they match, the entering of * optimized power state may be aborted. * +* @warning It is responsibility of the user to check if this function is +* supported at runtime using `rte_cpu_get_features()` API call. Failing to do +* so may result in an illegal CPU instruction error. +* - Reading this API description... what am I supposed to do in my application or driver who wants to use the new rte_power_monitor/rte_power_pause stuff?
I should call
rte_cpu_get_features(TOTO)?
This comment does not give a hint.
Quoted 8 lines
I suppose the intent was to refer to the
rte_cpu_get_intrinsics_support()thing. This must be fixed.- Again, I wonder why we are exposing all this stuff. This should be hidden in the
rte_powerAPI.
We're exposing all of this here because the intrinsics are *not* part of the power API but rather are generic headers within EAL. Therefore, any infrastructure checking for their support can *not* reside in the power library, but instead has to be in EAL.
The intended usage here is to call this function before calling rte_power_monitor(), such that:
struct rte_cpu_intrinsics intrinsics;
rte_cpu_get_intrinsics_support(&intrinsics);
if (!intrinsics.power_monitor) {
// rte_power_monitor not supported and cannot be used
return;
}
// proceed with rte_power_monitor usageFailing to do that will result in either -ENOTSUP on non-x86, or illegal instruction crash on x86 that doesn't have that instruction (because we encode raw opcode).
I've *not* added this to the previous patches because i wanted to get this part reviewed specifically, and not mix it with other IA-specific stuff. It seems that i've succeeded in that goal, as this patch has 4 likes^W acks :)
Signature · 2 lines
Thanks, Anatoly
Re: [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure
30/10/2020 11:09, Burakov, Anatoly:
Quoted 107 lines · David Marchand
Quoted 90 lines · Liang Ma
Quoted 11 lines
Currently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.
This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.
Quoted 6 lines
Signed-off-by Anatoly Burakov <[email protected]>Signed-off-by Liang Ma <[email protected]>Acked-by Jerin Jacob <[email protected]>Acked-by Ruifeng Wang <[email protected]>Acked-by Ray Kinsella <[email protected]>Coming late to the party, it seems crowded...
Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +2
-0@@ -13,6 +13,32 @@#include "rte_common.h" #include <errno.h> +#include <rte_compat.h> +lib/librte_eal/include/generic/rte_cpuflags.h line 27
Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +10
-0+/** +* Structure used to describe platform-specific intrinsics that may or may not +* be supported at runtime. +*/ +struct rte_cpu_intrinsics { +uint32_t power_monitor :1 ; +/**< indicates support for rte_power_monitor function */ +uint32_t power_pause :1 ; +/**< indicates support for rte_power_pause function */ +};- The
rte_powerlibrary is supposed to be built on top of cpuflags. Not the other way around.
Those capabilities should have been kept inside therte_power_API and not pollute the cpuflags API.- All of this should have come as a single patch as the previously introduced API is unusable before.
Quoted 22 lines
lib/librte_eal/include/generic/rte_cpuflags.h +14
-0+ +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Check CPU support for various intrinsics at runtime. +* +* @param intrinsics +* Pointer to a structure to be filled. +*/ +__rte_experimental +void +rte_cpu_get_intrinsics_support (struct rte_cpu_intrinsics *intrinsics); +/** * Enumeration of all CPU features supported */ lib/librte_eal/include/generic/rte_power_intrinsics.h +0
-0lib/librte_eal/include/generic/rte_power_intrinsics.h line 38
Quoted 7 lines
lib/librte_eal/include/generic/rte_power_intrinsics.h +4
-0@@ -32,6 +32,10 @@* checked against the expected value, and if they match, the entering of * optimized power state may be aborted. * +* @warning It is responsibility of the user to check if this function is +* supported at runtime using `rte_cpu_get_features()` API call. Failing to do +* so may result in an illegal CPU instruction error. +* - Reading this API description... what am I supposed to do in my application or driver who wants to use the new rte_power_monitor/rte_power_pause stuff?
I should call
rte_cpu_get_features(TOTO)?
This comment does not give a hint.Quoted 7 lines
I suppose the intent was to refer to the
rte_cpu_get_intrinsics_support()thing. This must be fixed.- Again, I wonder why we are exposing all this stuff. This should be hidden in the
rte_powerAPI.We're exposing all of this here because the intrinsics are *not* part of the power API but rather are generic headers within EAL. Therefore, any infrastructure checking for their support can *not* reside in the power library, but instead has to be in EAL.
The intended usage here is to call this function before calling
rte_power_monitor(), such that:
Quoted 8 lines
struct rte_cpu_intrinsics intrinsics;
rte_cpu_get_intrinsics_support(&intrinsics);
if (!intrinsics.power_monitor) {
// rte_power_monitor not supported and cannot be used
return;
}This check could be done inside the rte_power API.
Quoted 10 lines
// proceed with rte_power_monitor usageFailing to do that will result in either
-ENOTSUPon non-x86, or illegal instruction crash on x86 that doesn't have that instruction (because we encode raw opcode).I've *not* added this to the previous patches because i wanted to get this part reviewed specifically, and not mix it with other IA-specific stuff. It seems that i've succeeded in that goal, as this patch has 4 likes^W acks :)
You did not explain the need for rte_cpu_get_features() call.
Re: [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure
Quoted 110 lines · Thomas Monjalon
30/10/2020 11:09, Burakov, Anatoly:
Quoted 109 lines · David Marchand
Quoted 90 lines · Liang Ma
Quoted 11 lines
Currently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.
This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.
Quoted 6 lines
Signed-off-by Anatoly Burakov <[email protected]>Signed-off-by Liang Ma <[email protected]>Acked-by Jerin Jacob <[email protected]>Acked-by Ruifeng Wang <[email protected]>Acked-by Ray Kinsella <[email protected]>Coming late to the party, it seems crowded...
Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +2
-0@@ -13,6 +13,32 @@#include "rte_common.h" #include <errno.h> +#include <rte_compat.h> +lib/librte_eal/include/generic/rte_cpuflags.h line 27
Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +10
-0+/** +* Structure used to describe platform-specific intrinsics that may or may not +* be supported at runtime. +*/ +struct rte_cpu_intrinsics { +uint32_t power_monitor :1 ; +/**< indicates support for rte_power_monitor function */ +uint32_t power_pause :1 ; +/**< indicates support for rte_power_pause function */ +};- The
rte_powerlibrary is supposed to be built on top of cpuflags. Not the other way around.
Those capabilities should have been kept inside therte_power_API and not pollute the cpuflags API.- All of this should have come as a single patch as the previously introduced API is unusable before.
Quoted 22 lines
lib/librte_eal/include/generic/rte_cpuflags.h +14
-0+ +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Check CPU support for various intrinsics at runtime. +* +* @param intrinsics +* Pointer to a structure to be filled. +*/ +__rte_experimental +void +rte_cpu_get_intrinsics_support (struct rte_cpu_intrinsics *intrinsics); +/** * Enumeration of all CPU features supported */ lib/librte_eal/include/generic/rte_power_intrinsics.h +0
-0lib/librte_eal/include/generic/rte_power_intrinsics.h line 38
Quoted 7 lines
lib/librte_eal/include/generic/rte_power_intrinsics.h +4
-0@@ -32,6 +32,10 @@* checked against the expected value, and if they match, the entering of * optimized power state may be aborted. * +* @warning It is responsibility of the user to check if this function is +* supported at runtime using `rte_cpu_get_features()` API call. Failing to do +* so may result in an illegal CPU instruction error. +* - Reading this API description... what am I supposed to do in my application or driver who wants to use the new rte_power_monitor/rte_power_pause stuff?
I should call
rte_cpu_get_features(TOTO)?
This comment does not give a hint.Quoted 7 lines
I suppose the intent was to refer to the
rte_cpu_get_intrinsics_support()thing. This must be fixed.- Again, I wonder why we are exposing all this stuff. This should be hidden in the
rte_powerAPI.We're exposing all of this here because the intrinsics are *not* part of the power API but rather are generic headers within EAL. Therefore, any infrastructure checking for their support can *not* reside in the power library, but instead has to be in EAL.
The intended usage here is to call this function before calling
rte_power_monitor(), such that:struct rte_cpu_intrinsics intrinsics;
Quoted 8 lines
Quoted 6 lines
rte_cpu_get_intrinsics_support(&intrinsics); if (!intrinsics.power_monitor) { // rte_power_monitor not supported and cannot be used return; }This check could be done inside the
rte_powerAPI.
I'm not quite clear on exactly what you're asking here.
Do you mean the example code above? If so, code like that is already present in the power library, at the callback enable stage.
If you mean to say, i should put this check into the rte_power_monitor intrinsic, then no, i don't think it's a good idea to have this expensive check every time you call rte_power_monitor.
If you mean put this entire infrastructure into the power API - well, that kinda defeats the purpose of both having these intrinsics in generic headers and having a generic CPU feature check infrastructure that was requested of us during the review. We of course can move the intrinsic to the power library and outside of EAL, but then anything that requires UMWAIT will have to depend on the librte_power.
Please clarify exactly what changes you would like to see here, and what is your objection.
Quoted 7 lines
Quoted 7 lines
// proceed with rte_power_monitor usageFailing to do that will result in either
-ENOTSUPon non-x86, or illegal instruction crash on x86 that doesn't have that instruction (because we encode raw opcode).
Quoted 7 lines
Quoted 4 lines
I've *not* added this to the previous patches because i wanted to get this part reviewed specifically, and not mix it with other IA-specific stuff. It seems that i've succeeded in that goal, as this patch has 4 likes^W acks :)
You did not explain the need for
rte_cpu_get_features()call.
Did not explain *where*? Are you suggesting i put things about rte_power_monitor into documentation for rte_cpu_get_intrinsics_support? The documentation for rte_power_monitor already states that one should use rte_cpu_get_intrinsics_support API to check if the rte_power_monitor is supported on current machine. What else is missing?
Signature · 2 lines
Thanks, Anatoly
Re: [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure
30/10/2020 14:37, Burakov, Anatoly:
Quoted 120 lines · Thomas Monjalon
Quoted 110 lines
30/10/2020 11:09, Burakov, Anatoly:
Quoted 109 lines · David Marchand
Quoted 90 lines · Liang Ma
Quoted 11 lines
Currently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.
This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.
Quoted 6 lines
Signed-off-by Anatoly Burakov <[email protected]>Signed-off-by Liang Ma <[email protected]>Acked-by Jerin Jacob <[email protected]>Acked-by Ruifeng Wang <[email protected]>Acked-by Ray Kinsella <[email protected]>Coming late to the party, it seems crowded...
Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +2
-0@@ -13,6 +13,32 @@#include "rte_common.h" #include <errno.h> +#include <rte_compat.h> +lib/librte_eal/include/generic/rte_cpuflags.h line 27
Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +10
-0+/** +* Structure used to describe platform-specific intrinsics that may or may not +* be supported at runtime. +*/ +struct rte_cpu_intrinsics { +uint32_t power_monitor :1 ; +/**< indicates support for rte_power_monitor function */ +uint32_t power_pause :1 ; +/**< indicates support for rte_power_pause function */ +};- The
rte_powerlibrary is supposed to be built on top of cpuflags. Not the other way around.
Those capabilities should have been kept inside therte_power_API and not pollute the cpuflags API.- All of this should have come as a single patch as the previously introduced API is unusable before.
Quoted 22 lines
lib/librte_eal/include/generic/rte_cpuflags.h +14
-0+ +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Check CPU support for various intrinsics at runtime. +* +* @param intrinsics +* Pointer to a structure to be filled. +*/ +__rte_experimental +void +rte_cpu_get_intrinsics_support (struct rte_cpu_intrinsics *intrinsics); +/** * Enumeration of all CPU features supported */ lib/librte_eal/include/generic/rte_power_intrinsics.h +0
-0lib/librte_eal/include/generic/rte_power_intrinsics.h line 38
Quoted 7 lines
lib/librte_eal/include/generic/rte_power_intrinsics.h +4
-0@@ -32,6 +32,10 @@* checked against the expected value, and if they match, the entering of * optimized power state may be aborted. * +* @warning It is responsibility of the user to check if this function is +* supported at runtime using `rte_cpu_get_features()` API call. Failing to do +* so may result in an illegal CPU instruction error. +* - Reading this API description... what am I supposed to do in my application or driver who wants to use the new rte_power_monitor/rte_power_pause stuff?
I should call
rte_cpu_get_features(TOTO)?
This comment does not give a hint.Quoted 7 lines
I suppose the intent was to refer to the
rte_cpu_get_intrinsics_support()thing. This must be fixed.- Again, I wonder why we are exposing all this stuff. This should be hidden in the
rte_powerAPI.We're exposing all of this here because the intrinsics are *not* part of the power API but rather are generic headers within EAL. Therefore, any infrastructure checking for their support can *not* reside in the power library, but instead has to be in EAL.
The intended usage here is to call this function before calling
rte_power_monitor(), such that:struct rte_cpu_intrinsics intrinsics;Quoted 8 lines
Quoted 6 lines
rte_cpu_get_intrinsics_support(&intrinsics); if (!intrinsics.power_monitor) { // rte_power_monitor not supported and cannot be used return; }This check could be done inside the
rte_powerAPI.
Quoted 8 lines
I'm not quite clear on exactly what you're asking here.
Do you mean the example code above? If so, code like that is already present in the power library, at the callback enable stage.
If you mean to say, i should put this check into the
rte_power_monitorintrinsic, then no, i don't think it's a good idea to have this expensive check every time you callrte_power_monitor.
No but it can be done at initialization time.
According to what you say above, it is alread done at callback enable stage. So the app does not need to do the check?
Quoted 6 lines
If you mean put this entire infrastructure into the power API - well, that kinda defeats the purpose of both having these intrinsics in generic headers and having a generic CPU feature check infrastructure that was requested of us during the review. We of course can move the intrinsic to the power library and outside of EAL, but then anything that requires UMWAIT will have to depend on the
librte_power.
Yes the intrinsics can be in EAL if usable. But it seems DLB author cannot use what is in EAL.
Quoted 15 lines
Please clarify exactly what changes you would like to see here, and what is your objection.
Quoted 12 lines
Quoted 11 lines
// proceed with rte_power_monitor usageFailing to do that will result in either
-ENOTSUPon non-x86, or illegal instruction crash on x86 that doesn't have that instruction (because we encode raw opcode).I've *not* added this to the previous patches because i wanted to get this part reviewed specifically, and not mix it with other IA-specific stuff. It seems that i've succeeded in that goal, as this patch has 4 likes^W acks :)
Quoted 8 lines
Quoted 2 lines
You did not explain the need for
rte_cpu_get_features()call.Did not explain *where*? Are you suggesting i put things about
rte_power_monitorinto documentation forrte_cpu_get_intrinsics_support? The documentation forrte_power_monitoralready states that one should userte_cpu_get_intrinsics_supportAPI to check if therte_power_monitoris supported on current machine. What else is missing?
In your example above, you do not call rte_cpu_get_features() which is documented as required in the EAL doc.
Re: [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure
Quoted 126 lines · Thomas Monjalon
30/10/2020 14:37, Burakov, Anatoly:
Quoted 125 lines · Thomas Monjalon
Quoted 110 lines
30/10/2020 11:09, Burakov, Anatoly:
Quoted 109 lines · David Marchand
Quoted 90 lines · Liang Ma
Quoted 11 lines
Currently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.
This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.
Quoted 6 lines
Signed-off-by Anatoly Burakov <[email protected]>Signed-off-by Liang Ma <[email protected]>Acked-by Jerin Jacob <[email protected]>Acked-by Ruifeng Wang <[email protected]>Acked-by Ray Kinsella <[email protected]>Coming late to the party, it seems crowded...
Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +2
-0@@ -13,6 +13,32 @@#include "rte_common.h" #include <errno.h> +#include <rte_compat.h> +lib/librte_eal/include/generic/rte_cpuflags.h line 27
Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +10
-0+/** +* Structure used to describe platform-specific intrinsics that may or may not +* be supported at runtime. +*/ +struct rte_cpu_intrinsics { +uint32_t power_monitor :1 ; +/**< indicates support for rte_power_monitor function */ +uint32_t power_pause :1 ; +/**< indicates support for rte_power_pause function */ +};- The
rte_powerlibrary is supposed to be built on top of cpuflags. Not the other way around.
Those capabilities should have been kept inside therte_power_API and not pollute the cpuflags API.- All of this should have come as a single patch as the previously introduced API is unusable before.
Quoted 22 lines
lib/librte_eal/include/generic/rte_cpuflags.h +14
-0+ +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Check CPU support for various intrinsics at runtime. +* +* @param intrinsics +* Pointer to a structure to be filled. +*/ +__rte_experimental +void +rte_cpu_get_intrinsics_support (struct rte_cpu_intrinsics *intrinsics); +/** * Enumeration of all CPU features supported */ lib/librte_eal/include/generic/rte_power_intrinsics.h +0
-0lib/librte_eal/include/generic/rte_power_intrinsics.h line 38
Quoted 7 lines
lib/librte_eal/include/generic/rte_power_intrinsics.h +4
-0@@ -32,6 +32,10 @@* checked against the expected value, and if they match, the entering of * optimized power state may be aborted. * +* @warning It is responsibility of the user to check if this function is +* supported at runtime using `rte_cpu_get_features()` API call. Failing to do +* so may result in an illegal CPU instruction error. +* - Reading this API description... what am I supposed to do in my application or driver who wants to use the new rte_power_monitor/rte_power_pause stuff?
I should call
rte_cpu_get_features(TOTO)?
This comment does not give a hint.Quoted 7 lines
I suppose the intent was to refer to the
rte_cpu_get_intrinsics_support()thing. This must be fixed.- Again, I wonder why we are exposing all this stuff. This should be hidden in the
rte_powerAPI.We're exposing all of this here because the intrinsics are *not* part of the power API but rather are generic headers within EAL. Therefore, any infrastructure checking for their support can *not* reside in the power library, but instead has to be in EAL.
The intended usage here is to call this function before calling
rte_power_monitor(), such that:struct rte_cpu_intrinsics intrinsics;Quoted 8 lines
Quoted 6 lines
rte_cpu_get_intrinsics_support(&intrinsics); if (!intrinsics.power_monitor) { // rte_power_monitor not supported and cannot be used return; }This check could be done inside the
rte_powerAPI.I'm not quite clear on exactly what you're asking here.
Do you mean the example code above? If so, code like that is already present in the power library, at the callback enable stage.
Quoted 7 lines
Quoted 3 lines
If you mean to say, i should put this check into the
rte_power_monitorintrinsic, then no, i don't think it's a good idea to have this expensive check every time you callrte_power_monitor.No but it can be done at initialization time.
According to what you say above, it is alread done at callback enable stage. So the app does not need to do the check?
Admittedly it's a bit confusing, but please bear with me.
There are two separate issues at hand: the intrinsic itself, and the calling code. We provide both.
That means, the *calling code* should do the check. In our case, *our* calling code is the callback. However, nothing stops someone else from implementing their own scheme using our intrinsic - in that case, the user will be responsible to check if the intrinsic is supported before using it in their own code, because they won't be using our callback but will be using our intrinsic.
So, we have a check *in our calling code*. But if someone were to use the *intrinsic* directly (like DLB), they would have to add their own checks around the intrinsic usage.
Our power intrinsic is a static inline function. Are you proposing to add some sort of function pointer wrapper and make it an indirect call instead of a static inline function? (or indeed a proper function)
Quoted 10 lines
Quoted 7 lines
If you mean put this entire infrastructure into the power API - well, that kinda defeats the purpose of both having these intrinsics in generic headers and having a generic CPU feature check infrastructure that was requested of us during the review. We of course can move the intrinsic to the power library and outside of EAL, but then anything that requires UMWAIT will have to depend on the
librte_power.Yes the intrinsics can be in EAL if usable. But it seems DLB author cannot use what is in EAL.
I'll let the DLB authors clarify that themselves, but as far as i'm aware, it seems that this is not the case - while their current code wouldn't be able to use these intrinsics by search-and-replace, they will be able to use them with a couple of changes to their code that basically amounted to reimplementation of our intrinsics.
Quoted 19 lines
Quoted 19 lines
Please clarify exactly what changes you would like to see here, and what is your objection.
Quoted 14 lines
Quoted 11 lines
// proceed with rte_power_monitor usageFailing to do that will result in either
-ENOTSUPon non-x86, or illegal instruction crash on x86 that doesn't have that instruction (because we encode raw opcode).I've *not* added this to the previous patches because i wanted to get this part reviewed specifically, and not mix it with other IA-specific stuff. It seems that i've succeeded in that goal, as this patch has 4 likes^W acks :)
You did not explain the need for
rte_cpu_get_features()call.
Quoted 9 lines
Quoted 5 lines
Did not explain *where*? Are you suggesting i put things about
rte_power_monitorinto documentation forrte_cpu_get_intrinsics_support? The documentation forrte_power_monitoralready states that one should userte_cpu_get_intrinsics_supportAPI to check if therte_power_monitoris supported on current machine. What else is missing?In your example above, you do not call
rte_cpu_get_features()which is documented as required in the EAL doc.
I'm not sure i follow. This is unrelated to rte_cpu_get_features call. The rte_cpu_get_features is a CPUID check, and it was decided not to use it because the WAITPKG CPUID flag is only defined for x86 and not for other archs. This new call (rte_cpu_get_intrinsics_support) is non-arch specific, but will have an arch-specific implementation (which happens to use rte_cpu_get_features to detect support for WAITPKG). I have given the example code of how to detect support for rte_power_monitor using this new code, in the code example you just referred to.
Signature · 2 lines
Thanks, Anatoly
Re: [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure
30/10/2020 16:27, Burakov, Anatoly:
Quoted 201 lines · Thomas Monjalon
Quoted 126 lines
30/10/2020 14:37, Burakov, Anatoly:
Quoted 125 lines · Thomas Monjalon
Quoted 110 lines
30/10/2020 11:09, Burakov, Anatoly:
Quoted 109 lines · David Marchand
Quoted 90 lines · Liang Ma
Quoted 11 lines
Currently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.
This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.
Quoted 6 lines
Signed-off-by Anatoly Burakov <[email protected]>Signed-off-by Liang Ma <[email protected]>Acked-by Jerin Jacob <[email protected]>Acked-by Ruifeng Wang <[email protected]>Acked-by Ray Kinsella <[email protected]>Coming late to the party, it seems crowded...
Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +2
-0@@ -13,6 +13,32 @@#include "rte_common.h" #include <errno.h> +#include <rte_compat.h> +lib/librte_eal/include/generic/rte_cpuflags.h line 27
Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +10
-0+/** +* Structure used to describe platform-specific intrinsics that may or may not +* be supported at runtime. +*/ +struct rte_cpu_intrinsics { +uint32_t power_monitor :1 ; +/**< indicates support for rte_power_monitor function */ +uint32_t power_pause :1 ; +/**< indicates support for rte_power_pause function */ +};- The
rte_powerlibrary is supposed to be built on top of cpuflags. Not the other way around.
Those capabilities should have been kept inside therte_power_API and not pollute the cpuflags API.- All of this should have come as a single patch as the previously introduced API is unusable before.
Quoted 22 lines
lib/librte_eal/include/generic/rte_cpuflags.h +14
-0+ +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Check CPU support for various intrinsics at runtime. +* +* @param intrinsics +* Pointer to a structure to be filled. +*/ +__rte_experimental +void +rte_cpu_get_intrinsics_support (struct rte_cpu_intrinsics *intrinsics); +/** * Enumeration of all CPU features supported */ lib/librte_eal/include/generic/rte_power_intrinsics.h +0
-0lib/librte_eal/include/generic/rte_power_intrinsics.h line 38
Quoted 7 lines
lib/librte_eal/include/generic/rte_power_intrinsics.h +4
-0@@ -32,6 +32,10 @@* checked against the expected value, and if they match, the entering of * optimized power state may be aborted. * +* @warning It is responsibility of the user to check if this function is +* supported at runtime using `rte_cpu_get_features()` API call. Failing to do +* so may result in an illegal CPU instruction error. +* - Reading this API description... what am I supposed to do in my application or driver who wants to use the new rte_power_monitor/rte_power_pause stuff?
I should call
rte_cpu_get_features(TOTO)?
This comment does not give a hint.Quoted 7 lines
I suppose the intent was to refer to the
rte_cpu_get_intrinsics_support()thing. This must be fixed.- Again, I wonder why we are exposing all this stuff. This should be hidden in the
rte_powerAPI.We're exposing all of this here because the intrinsics are *not* part of the power API but rather are generic headers within EAL. Therefore, any infrastructure checking for their support can *not* reside in the power library, but instead has to be in EAL.
The intended usage here is to call this function before calling
rte_power_monitor(), such that:struct rte_cpu_intrinsics intrinsics;Quoted 8 lines
Quoted 6 lines
rte_cpu_get_intrinsics_support(&intrinsics); if (!intrinsics.power_monitor) { // rte_power_monitor not supported and cannot be used return; }This check could be done inside the
rte_powerAPI.I'm not quite clear on exactly what you're asking here.
Do you mean the example code above? If so, code like that is already present in the power library, at the callback enable stage.
Quoted 7 lines
Quoted 3 lines
If you mean to say, i should put this check into the
rte_power_monitorintrinsic, then no, i don't think it's a good idea to have this expensive check every time you callrte_power_monitor.No but it can be done at initialization time.
According to what you say above, it is alread done at callback enable stage. So the app does not need to do the check?Admittedly it's a bit confusing, but please bear with me.
There are two separate issues at hand: the intrinsic itself, and the calling code. We provide both.
That means, the *calling code* should do the check. In our case, *our* calling code is the callback. However, nothing stops someone else from implementing their own scheme using our intrinsic - in that case, the user will be responsible to check if the intrinsic is supported before using it in their own code, because they won't be using our callback but will be using our intrinsic.
So, we have a check *in our calling code*. But if someone were to use the *intrinsic* directly (like DLB), they would have to add their own checks around the intrinsic usage.
Our power intrinsic is a static inline function. Are you proposing to add some sort of function pointer wrapper and make it an indirect call instead of a static inline function? (or indeed a proper function)
Quoted 10 lines
Quoted 7 lines
If you mean put this entire infrastructure into the power API - well, that kinda defeats the purpose of both having these intrinsics in generic headers and having a generic CPU feature check infrastructure that was requested of us during the review. We of course can move the intrinsic to the power library and outside of EAL, but then anything that requires UMWAIT will have to depend on the
librte_power.Yes the intrinsics can be in EAL if usable. But it seems DLB author cannot use what is in EAL.
I'll let the DLB authors clarify that themselves, but as far as i'm aware, it seems that this is not the case - while their current code wouldn't be able to use these intrinsics by search-and-replace, they will be able to use them with a couple of changes to their code that basically amounted to reimplementation of our intrinsics.
Quoted 19 lines
Quoted 19 lines
Please clarify exactly what changes you would like to see here, and what is your objection.
Quoted 14 lines
Quoted 11 lines
// proceed with rte_power_monitor usageFailing to do that will result in either
-ENOTSUPon non-x86, or illegal instruction crash on x86 that doesn't have that instruction (because we encode raw opcode).I've *not* added this to the previous patches because i wanted to get this part reviewed specifically, and not mix it with other IA-specific stuff. It seems that i've succeeded in that goal, as this patch has 4 likes^W acks :)
You did not explain the need for
rte_cpu_get_features()call.Quoted 9 lines
Quoted 5 lines
Did not explain *where*? Are you suggesting i put things about
rte_power_monitorinto documentation forrte_cpu_get_intrinsics_support? The documentation forrte_power_monitoralready states that one should userte_cpu_get_intrinsics_supportAPI to check if therte_power_monitoris supported on current machine. What else is missing?In your example above, you do not call
rte_cpu_get_features()which is documented as required in the EAL doc.
Quoted 8 lines
I'm not sure i follow. This is unrelated to
rte_cpu_get_featurescall. Therte_cpu_get_featuresis a CPUID check, and it was decided not to use it because the WAITPKG CPUID flag is only defined for x86 and not for other archs. This new call (rte_cpu_get_intrinsics_support) is non-arch specific, but will have an arch-specific implementation (which happens to userte_cpu_get_featuresto detect support for WAITPKG). I have given the example code of how to detect support forrte_power_monitorusing this new code, in the code example you just referred to.
Please read the API again:
http://git.dpdk.org/dpdk/tree/lib/librte_eal/include/generic/rte_power_intrinsics.h
"
* @warning It is responsibility of the user to check if this function is
* supported at runtime using rte_cpu_get_features() API call.
* Failing to do so may result in an illegal CPU instruction error.
"
Why is it referring to rte_cpu_get_features?
Re: [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure
Quoted 211 lines · Thomas Monjalon
30/10/2020 16:27, Burakov, Anatoly:
Quoted 201 lines · Thomas Monjalon
Quoted 126 lines
30/10/2020 14:37, Burakov, Anatoly:
Quoted 125 lines · Thomas Monjalon
Quoted 110 lines
30/10/2020 11:09, Burakov, Anatoly:
Quoted 109 lines · David Marchand
Quoted 90 lines · Liang Ma
Quoted 11 lines
Currently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.
This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.
Quoted 6 lines
Signed-off-by Anatoly Burakov <[email protected]>Signed-off-by Liang Ma <[email protected]>Acked-by Jerin Jacob <[email protected]>Acked-by Ruifeng Wang <[email protected]>Acked-by Ray Kinsella <[email protected]>Coming late to the party, it seems crowded...
Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +2
-0@@ -13,6 +13,32 @@#include "rte_common.h" #include <errno.h> +#include <rte_compat.h> +lib/librte_eal/include/generic/rte_cpuflags.h line 27
Quoted 10 lines
lib/librte_eal/include/generic/rte_cpuflags.h +10
-0+/** +* Structure used to describe platform-specific intrinsics that may or may not +* be supported at runtime. +*/ +struct rte_cpu_intrinsics { +uint32_t power_monitor :1 ; +/**< indicates support for rte_power_monitor function */ +uint32_t power_pause :1 ; +/**< indicates support for rte_power_pause function */ +};- The
rte_powerlibrary is supposed to be built on top of cpuflags. Not the other way around.
Those capabilities should have been kept inside therte_power_API and not pollute the cpuflags API.- All of this should have come as a single patch as the previously introduced API is unusable before.
Quoted 22 lines
lib/librte_eal/include/generic/rte_cpuflags.h +14
-0+ +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Check CPU support for various intrinsics at runtime. +* +* @param intrinsics +* Pointer to a structure to be filled. +*/ +__rte_experimental +void +rte_cpu_get_intrinsics_support (struct rte_cpu_intrinsics *intrinsics); +/** * Enumeration of all CPU features supported */ lib/librte_eal/include/generic/rte_power_intrinsics.h +0
-0lib/librte_eal/include/generic/rte_power_intrinsics.h line 38
Quoted 7 lines
lib/librte_eal/include/generic/rte_power_intrinsics.h +4
-0@@ -32,6 +32,10 @@* checked against the expected value, and if they match, the entering of * optimized power state may be aborted. * +* @warning It is responsibility of the user to check if this function is +* supported at runtime using `rte_cpu_get_features()` API call. Failing to do +* so may result in an illegal CPU instruction error. +* - Reading this API description... what am I supposed to do in my application or driver who wants to use the new rte_power_monitor/rte_power_pause stuff?
I should call
rte_cpu_get_features(TOTO)?
This comment does not give a hint.Quoted 7 lines
I suppose the intent was to refer to the
rte_cpu_get_intrinsics_support()thing. This must be fixed.- Again, I wonder why we are exposing all this stuff. This should be hidden in the
rte_powerAPI.We're exposing all of this here because the intrinsics are *not* part of the power API but rather are generic headers within EAL. Therefore, any infrastructure checking for their support can *not* reside in the power library, but instead has to be in EAL.
The intended usage here is to call this function before calling
rte_power_monitor(), such that:struct rte_cpu_intrinsics intrinsics;Quoted 8 lines
Quoted 6 lines
rte_cpu_get_intrinsics_support(&intrinsics); if (!intrinsics.power_monitor) { // rte_power_monitor not supported and cannot be used return; }This check could be done inside the
rte_powerAPI.I'm not quite clear on exactly what you're asking here.
Do you mean the example code above? If so, code like that is already present in the power library, at the callback enable stage.
Quoted 7 lines
Quoted 3 lines
If you mean to say, i should put this check into the
rte_power_monitorintrinsic, then no, i don't think it's a good idea to have this expensive check every time you callrte_power_monitor.No but it can be done at initialization time.
According to what you say above, it is alread done at callback enable stage. So the app does not need to do the check?Admittedly it's a bit confusing, but please bear with me.
There are two separate issues at hand: the intrinsic itself, and the calling code. We provide both.
That means, the *calling code* should do the check. In our case, *our* calling code is the callback. However, nothing stops someone else from implementing their own scheme using our intrinsic - in that case, the user will be responsible to check if the intrinsic is supported before using it in their own code, because they won't be using our callback but will be using our intrinsic.
So, we have a check *in our calling code*. But if someone were to use the *intrinsic* directly (like DLB), they would have to add their own checks around the intrinsic usage.
Our power intrinsic is a static inline function. Are you proposing to add some sort of function pointer wrapper and make it an indirect call instead of a static inline function? (or indeed a proper function)
Quoted 10 lines
Quoted 7 lines
If you mean put this entire infrastructure into the power API - well, that kinda defeats the purpose of both having these intrinsics in generic headers and having a generic CPU feature check infrastructure that was requested of us during the review. We of course can move the intrinsic to the power library and outside of EAL, but then anything that requires UMWAIT will have to depend on the
librte_power.Yes the intrinsics can be in EAL if usable. But it seems DLB author cannot use what is in EAL.
I'll let the DLB authors clarify that themselves, but as far as i'm aware, it seems that this is not the case - while their current code wouldn't be able to use these intrinsics by search-and-replace, they will be able to use them with a couple of changes to their code that basically amounted to reimplementation of our intrinsics.
Quoted 19 lines
Quoted 19 lines
Please clarify exactly what changes you would like to see here, and what is your objection.
Quoted 14 lines
Quoted 11 lines
// proceed with rte_power_monitor usageFailing to do that will result in either
-ENOTSUPon non-x86, or illegal instruction crash on x86 that doesn't have that instruction (because we encode raw opcode).I've *not* added this to the previous patches because i wanted to get this part reviewed specifically, and not mix it with other IA-specific stuff. It seems that i've succeeded in that goal, as this patch has 4 likes^W acks :)
You did not explain the need for
rte_cpu_get_features()call.Quoted 9 lines
Quoted 5 lines
Did not explain *where*? Are you suggesting i put things about
rte_power_monitorinto documentation forrte_cpu_get_intrinsics_support? The documentation forrte_power_monitoralready states that one should userte_cpu_get_intrinsics_supportAPI to check if therte_power_monitoris supported on current machine. What else is missing?In your example above, you do not call
rte_cpu_get_features()which is documented as required in the EAL doc.Quoted 8 lines
I'm not sure i follow. This is unrelated to
rte_cpu_get_featurescall. Therte_cpu_get_featuresis a CPUID check, and it was decided not to use it because the WAITPKG CPUID flag is only defined for x86 and not for other archs. This new call (rte_cpu_get_intrinsics_support) is non-arch specific, but will have an arch-specific implementation (which happens to userte_cpu_get_featuresto detect support for WAITPKG). I have given the example code of how to detect support forrte_power_monitorusing this new code, in the code example you just referred to.
Quoted 9 lines
Please read the API again:
http://git.dpdk.org/dpdk/tree/lib/librte_eal/include/generic/rte_power_intrinsics.h
"
* @warning It is responsibility of the user to check if this function is
* supported at runtime usingrte_cpu_get_features()API call.
* Failing to do so may result in an illegal CPU instruction error.
"
Why is it referring torte_cpu_get_features?
Aw, crap. You're right, it's an artifact of earlier implementation. We'll fix this. Thanks for letting us know!
Signature · 2 lines
Thanks, Anatoly
Re: [dpdk-dev] [PATCH v10 3/9] eal: add intrinsics support check infrastructure
30/10/2020 17:36, Burakov, Anatoly:
Quoted 18 lines · Thomas Monjalon
Quoted 17 lines
30/10/2020 16:27, Burakov, Anatoly:
Quoted 16 lines · Thomas Monjalon
Quoted 15 lines
30/10/2020 14:37, Burakov, Anatoly:
Quoted 14 lines · Thomas Monjalon
Quoted 13 lines
30/10/2020 11:09, Burakov, Anatoly:
Quoted 11 lines
The intended usage here is to call this function before calling
rte_power_monitor(), such that:struct rte_cpu_intrinsics intrinsics; rte_cpu_get_intrinsics_support(&intrinsics); if (!intrinsics.power_monitor) { // rte_power_monitor not supported and cannot be used return; }
[...]
Quoted 22 lines
Quoted 13 lines
Quoted 4 lines
Quoted 3 lines
In your example above, you do not call
rte_cpu_get_features()which is documented as required in the EAL doc.Quoted 8 lines
I'm not sure i follow. This is unrelated to
rte_cpu_get_featurescall. Therte_cpu_get_featuresis a CPUID check, and it was decided not to use it because the WAITPKG CPUID flag is only defined for x86 and not for other archs. This new call (rte_cpu_get_intrinsics_support) is non-arch specific, but will have an arch-specific implementation (which happens to userte_cpu_get_featuresto detect support for WAITPKG). I have given the example code of how to detect support forrte_power_monitorusing this new code, in the code example you just referred to.Quoted 8 lines
Please read the API again:
http://git.dpdk.org/dpdk/tree/lib/librte_eal/include/generic/rte_power_intrinsics.h
"
* @warning It is responsibility of the user to check if this function is
* supported at runtime usingrte_cpu_get_features()API call.
* Failing to do so may result in an illegal CPU instruction error.
"
Why is it referring torte_cpu_get_features?
Quoted 2 lines
Aw, crap. You're right, it's an artifact of earlier implementation. We'll fix this. Thanks for letting us know!
This is what happens when everybody pushes me to merge a patch that I believe not ready but with a lot of "acked but not reviewed".
The context around this patch series is not good to allow good quality. That's why I think we should not merge any more patch on top of it except DLB PMDs and fixes in this release.
PATCH v10 4/9 ethdev: add simple power management API
In reply to Liang MaAdd a simple API to allow getting address of next RX descriptor from the PMD, as well as release notes information.
---
Notes:
v10:
- Address minor issue on comments and release notes
v8:
- Rename version map file name
v7:
- Fixed race condition (Konstantin)
- Slight rework of the structure of monitor code
- Added missing inline for wakeup
v6:
- Added wakeup mechanism for UMWAIT
- Removed memory allocation (everything is now allocated statically)
- Fixed various typos and comments
- Check for invalid queue ID
- Moved release notes to this patch
v5:
- Make error checking more robust
- Prevent initializing scaling if ACPI or PSTATE env wasn't set
- Prevent initializing UMWAIT path if PMD doesn't support get_wake_addr
- Add some debug logging
- Replace x86-specific code path to generic path using the intrinsic check
--- doc/guides/rel_notes/release_20_11.rst | 4 ++++ lib/librte_ethdev/rte_ethdev.c | 23 ++++++++++++++++++ lib/librte_ethdev/rte_ethdev.h | 33 ++++++++++++++++++++++++++ lib/librte_ethdev/rte_ethdev_driver.h | 28 ++++++++++++++++++++++ lib/librte_ethdev/version.map | 1 + 5 files changed, 89 insertions(+)
doc/guides/rel_notes/release_20_11.rst +4 -0
@@ -139,6 +139,10 @@ New Features Hairpin Tx part flow rules can be inserted explicitly. New API is added to get the hairpin peer ports list. +* **ethdev: added 1 new API for PMD power management.** + + * ``rte_eth_get_wake_addr()`` is added to get the wake up address from device. + * **Updated Broadcom bnxt driver.** Updated the Broadcom bnxt driver with new features and improvements, including:
lib/librte_ethdev/rte_ethdev.c +23 -0
@@ -5138,6 +5138,29 @@ rte_eth_tx_burst_mode_get(uint16_t port_id, uint16_t queue_id, dev->dev_ops->tx_burst_mode_get(dev, queue_id, mode)); } +int +rte_eth_get_wake_addr (uint16_t port_id,uint16_t queue_id, +volatile void **wake_addr,uint64_t *expected,uint64_t *mask, +uint8_t *data_sz) +{ +struct rte_eth_dev *dev; + +RTE_ETH_VALID_PORTID_OR_ERR_RET (port_id, -ENODEV ); + + dev = &rte_eth_devices[port_id]; + +RTE_FUNC_PTR_OR_ERR_RET (*dev->dev_ops->get_wake_addr, -ENOTSUP ); + +if (queue_id >= dev->data->nb_rx_queues) { +RTE_ETHDEV_LOG (ERR ,"Invalid RX queue_id=%u\n" , queue_id); +return -EINVAL ; + } + +return eth_err (port_id, + dev->dev_ops->get_wake_addr (dev->data->rx_queues[queue_id], + wake_addr, expected, mask, data_sz)); +} + int rte_eth_dev_set_mc_addr_list(uint16_t port_id, struct rte_ether_addr *mc_addr_set,
lib/librte_ethdev/rte_ethdev.h +33 -0
@@ -4364,6 +4364,39 @@ __rte_experimentalint rte_eth_tx_burst_mode_get (uint16_t port_id,uint16_t queue_id,struct rte_eth_burst_mode *mode); +/** +* Retrieve the wake up address for device queue or other data structure. +* Depend on the HW design, any write back mechanism can be used as a signal +* to wake up the processor. +* +* @param port_id +* The port identifier of the Ethernet device. +* @param queue_id +* The Rx queue on the Ethernet device for which information will be +* retrieved. +* @param wake_addr +* The pointer to the address which will be monitored and filled by driver. +* @param expected +* The pointer to value to be expected when target address is set. +* filled by driver. +* @param mask +* The pointer to comparison bitmask for the expected value that is filled by +* driver. +* @param data_sz +* The pointer to data size for the expected value and comparison bitmask that +* is filled by driver. +* +* @return +* - 0: Success. +* -ENOTSUP: Operation not supported. +* -EINVAL: Invalid parameters. +* -ENODEV: Invalid port ID. +*/ +__rte_experimental +int rte_eth_get_wake_addr(uint16_t port_id, uint16_t queue_id, +volatile void **wake_addr,uint64_t *expected,uint64_t *mask, +uint8_t *data_sz); +/** * Retrieve device registers and register attributes (number of registers and * register size)
lib/librte_ethdev/rte_ethdev_driver.h +28 -0
@@ -752,6 +752,32 @@ typedef int (*eth_hairpin_queue_peer_unbind_t) (struct rte_eth_dev *dev, uint16_t cur_queue, uint32_t direction);/**< @internal Unbind peer queue from the current queue. */ +/** +* @internal +* Get address of memory location whose contents will change whenever there is +* new data to be received on an RX queue. +* +* @param rxq +* Ethdev queue pointer. +* @param tail_desc_addr +* The pointer point to where the address will be stored. +* @param expected +* The pointer point to value to be expected when descriptor is set. +* @param mask +* The pointer point to comparison bitmask for the expected value. +* @param data_sz +* Data size for the expected value (can be 1, 2, 4, or 8 bytes) +* @return +* Negative errno value on error, 0 on success. +* +* @retval 0 +* Success +* @retval -EINVAL +* Invalid parameters +*/ +typedef int (*eth_get_wake_addr_t )(void *rxq,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *mask,uint8_t *data_sz); +/** * @internal A structure containing the functions exported by an Ethernet driver. */
@@ -910,6 +936,8 @@ struct eth_dev_ops {/**< Set up the connection between the pair of hairpin queues. */ eth_hairpin_queue_peer_unbind_t hairpin_queue_peer_unbind;/**< Disconnect the hairpin queues of a pair from each other. */ +eth_get_wake_addr_t get_wake_addr; +/**< Get next RX queue ring entry address. */ };/**
lib/librte_ethdev/version.map +1 -0
@@ -235,6 +235,7 @@ EXPERIMENTAL { rte_eth_fec_get_capability; rte_eth_fec_get; rte_eth_fec_set; + rte_eth_get_wake_addr; rte_flow_shared_action_create; rte_flow_shared_action_destroy; rte_flow_shared_action_query;
Signature · 1 line
2.17.1
PATCH v10 5/9 power: add PMD power management API and callback
In reply to Liang MaAdd a simple on/off switch that will enable saving power when no packets are arriving. It is based on counting the number of empty polls and, when the number reaches a certain threshold, entering an architecture-defined optimized power state that will either wait until a TSC timestamp expires, or when packets arrive.
This API mandates a core-to-single-queue mapping (that is, multiple queued per device are supported, but they have to be polled on different cores).
This design is using PMD RX callbacks.
1. UMWAIT/UMONITOR:
When a certain threshold of empty polls is reached, the core will go into a power optimized sleep while waiting on an address of next RX descriptor to be written to.
2. Pause instruction
Instead of move the core into deeper C state, this method uses the pause instruction to avoid busy polling.
3. Frequency scaling
Reuse existing DPDK power library to scale up/down core frequency depending on traffic volume.
---
Notes:
v10:
- Updated power library document
v8:
- Rename version map file name
v7:
- Fixed race condition (Konstantin)
- Slight rework of the structure of monitor code
- Added missing inline for wakeup
v6:
- Added wakeup mechanism for UMWAIT
- Removed memory allocation (everything is now allocated statically)
- Fixed various typos and comments
- Check for invalid queue ID
- Moved release notes to this patch
v5:
- Make error checking more robust
- Prevent initializing scaling if ACPI or PSTATE env wasn't set
- Prevent initializing UMWAIT path if PMD doesn't support get_wake_addr
- Add some debug logging
- Replace x86-specific code path to generic path using the intrinsic check
--- doc/guides/prog_guide/power_man.rst | 48 ++++ doc/guides/rel_notes/release_20_11.rst | 11 + lib/librte_power/meson.build | 5 +- lib/librte_power/rte_power_pmd_mgmt.c | 320 +++++++++++++++++++++++++ lib/librte_power/rte_power_pmd_mgmt.h | 92 +++++++ lib/librte_power/version.map | 4 + 6 files changed, 478 insertions(+), 2 deletions(-) create mode 100644 lib/librte_power/rte_power_pmd_mgmt.c create mode 100644 lib/librte_power/rte_power_pmd_mgmt.h
doc/guides/prog_guide/power_man.rst +48 -0
@@ -192,6 +192,51 @@ User Cases ---------- The mechanism can applied to any device which is based on polling. e.g. NIC, FPGA. +PMD Power Management API +------------------------ + +Abstract +~~~~~~~~ + +Existing power management mechanisms require developers to change application +design or change code to make use of it. The PMD power management API provides a +convenient alternative by utilizing Ethernet PMD RX callbacks, and triggering +power saving whenever empty poll count reaches a certain number. + +There are multiple power saving schemes available for developer to choose. +Although developer can configure each queue with different scheme, however, +It's strongly recommend to configure the queue within same port with same scheme. + + * UMWAIT/UMONITOR + + This power saving scheme will put the CPU into optimized power state and use + the UMWAIT/UMONITOR instructions to monitor the Ethernet PMD RX descriptor + address, and wake the CPU up whenever there's new traffic. + + * Pause + + This power saving scheme will use the `rte_pause` function to avoid busy + polling. + + * Frequency scaling + + This power saving scheme will use existing power library functionality to + scale the core frequency up/down depending on traffic volume. + + +.. note:: + + Currently, this power management API is limited to mandatory mapping of 1 + queue to 1 core (multiple queues are supported, but they must be polled from + different cores). + +API Overview for PMD Power Management +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* **Queue Enable**: Enable specific power scheme for certain queue/port/core + +* **Queue Disable**: Disable power scheme for certain queue/port/core + References ----------
@@ -200,3 +245,6 @@ References * The :doc:`../sample_app_ug/vm_power_management` chapter in the :doc:`../sample_app_ug/index` section. + +* The :doc:`../sample_app_ug/rxtx_callbacks` + chapter in the :doc:`../sample_app_ug/index` section.
doc/guides/rel_notes/release_20_11.rst +11 -0
@@ -349,6 +349,17 @@ New Features * Replaced ``--scalar`` command-line option with ``--alg=<value>``, to allow the user to select the desired classify method. +* **Add PMD power management mechanism** + + 3 new Ethernet PMD power management mechanism is added through existing + RX callback infrastructure. + + * Add power saving scheme based on UMWAIT instruction (x86 only) + * Add power saving scheme based on ``rte_pause()`` + * Add power saving scheme based on frequency scaling through the power library + * Add new EXPERIMENTAL API ``rte_power_pmd_mgmt_queue_enable()`` + * Add new EXPERIMENTAL API ``rte_power_pmd_mgmt_queue_disable()`` + Removed Items -------------
lib/librte_power/meson.build +3 -2
@@ -9,6 +9,7 @@ sources = files('rte_power.c', 'power_acpi_cpufreq.c', 'power_kvm_vm.c', 'guest_channel.c', 'rte_power_empty_poll.c', 'power_pstate_cpufreq.c', + 'rte_power_pmd_mgmt.c', 'power_common.c') -headers = files('rte_power.h','rte_power_empty_poll.h') -deps += ['timer'] +headers = files('rte_power.h','rte_power_empty_poll.h','rte_power_pmd_mgmt.h') +deps += ['timer' ,'ethdev']
lib/librte_power/rte_power_pmd_mgmt.c new file +320 -0
320 lines not on this page · raw message
lib/librte_power/rte_power_pmd_mgmt.h new file +92 -0
92 lines not on this page · raw message
lib/librte_power/version.map +4 -0
@@ -34,4 +34,8 @@ EXPERIMENTAL { rte_power_guest_channel_receive_msg; rte_power_poll_stat_fetch; rte_power_poll_stat_update; + # added in 20.11 + rte_power_pmd_mgmt_queue_enable; + rte_power_pmd_mgmt_queue_disable; + };
Signature · 1 line
2.17.1
PATCH v10 6/9 net/ixgbe: implement power management API
In reply to Liang MaImplement support for the power management API by implementing a get_wake_addr function that will return an address of an RX ring's status bit.
--- drivers/net/ixgbe/ixgbe_ethdev.c | 1 + drivers/net/ixgbe/ixgbe_rxtx.c | 25 +++++++++++++++++++++++++ drivers/net/ixgbe/ixgbe_rxtx.h | 2 ++ 3 files changed, 28 insertions(+)
drivers/net/ixgbe/ixgbe_ethdev.c +1 -0
@@ -588,6 +588,7 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = { .udp_tunnel_port_del = ixgbe_dev_udp_tunnel_port_del, .tm_ops_get = ixgbe_tm_ops_get, .tx_done_cleanup = ixgbe_dev_tx_done_cleanup, + .get_wake_addr = ixgbe_get_wake_addr, };/*
drivers/net/ixgbe/ixgbe_rxtx.c +25 -0
@@ -1367,6 +1367,31 @@ const uint32_t RTE_PTYPE_INNER_L3_IPV4_EXT | RTE_PTYPE_INNER_L4_UDP, }; +int ixgbe_get_wake_addr (void *rx_queue,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *mask,uint8_t *data_sz) +{ +volatile union ixgbe_adv_rx_desc *rxdp; +struct ixgbe_rx_queue *rxq = rx_queue; +uint16_t desc; + + desc = rxq->rx_tail; + rxdp = &rxq->rx_ring[desc]; +/* watch for changes in status bit */ + *tail_desc_addr = &rxdp->wb.upper.status_error; + +/* +* we expect the DD bit to be set to 1 if this descriptor was already +* written to. +*/ + *expected =rte_cpu_to_le_32 (IXGBE_RXDADV_STAT_DD ); + *mask =rte_cpu_to_le_32 (IXGBE_RXDADV_STAT_DD ); + +/* the registers are 32-bit */ + *data_sz =4 ; + +return 0 ; +} +/* @note: fix ixgbe_dev_supported_ptypes_get() if any change here. */ static inline uint32_t ixgbe_rxd_pkt_info_to_pkt_type (uint32_t pkt_info,uint16_t ptype_mask)
drivers/net/ixgbe/ixgbe_rxtx.h +2 -0
@@ -299,5 +299,7 @@ uint64_t ixgbe_get_tx_port_offloads(struct rte_eth_dev *dev);uint64_t ixgbe_get_rx_queue_offloads (struct rte_eth_dev *dev);uint64_t ixgbe_get_rx_port_offloads (struct rte_eth_dev *dev);uint64_t ixgbe_get_tx_queue_offloads (struct rte_eth_dev *dev); +int ixgbe_get_wake_addr (void *rx_queue,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *mask,uint8_t *data_sz);#endif /* _IXGBE_RXTX_H_ */
Signature · 1 line
2.17.1
PATCH v10 7/9 net/i40e: implement power management API
In reply to Liang MaImplement support for the power management API by implementing a get_wake_addr function that will return an address of an RX ring's status bit.
--- drivers/net/i40e/i40e_ethdev.c | 1 + drivers/net/i40e/i40e_rxtx.c | 26 ++++++++++++++++++++++++++ drivers/net/i40e/i40e_rxtx.h | 2 ++ 3 files changed, 29 insertions(+)
drivers/net/i40e/i40e_ethdev.c +1 -0
@@ -513,6 +513,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = { .mtu_set = i40e_dev_mtu_set, .tm_ops_get = i40e_tm_ops_get, .tx_done_cleanup = i40e_tx_done_cleanup, + .get_wake_addr = i40e_get_wake_addr, };/* store statistics names and its offset in stats structure */
drivers/net/i40e/i40e_rxtx.c +26 -0
@@ -72,6 +72,32 @@#define I40E_TX_OFFLOAD_NOTSUP_MASK \ (PKT_TX_OFFLOAD_MASK ^ I40E_TX_OFFLOAD_MASK) +int +i40e_get_wake_addr (void *rx_queue,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *mask,uint8_t *data_sz) +{ +struct i40e_rx_queue *rxq = rx_queue; +volatile union i40e_rx_desc *rxdp; +uint16_t desc; + + desc = rxq->rx_tail; + rxdp = &rxq->rx_ring[desc]; +/* watch for changes in status bit */ + *tail_desc_addr = &rxdp->wb.qword1.status_error_len; + +/* +* we expect the DD bit to be set to 1 if this descriptor was already +* written to. +*/ + *expected =rte_cpu_to_le_64 (1 <<I40E_RX_DESC_STATUS_DD_SHIFT ); + *mask =rte_cpu_to_le_64 (1 <<I40E_RX_DESC_STATUS_DD_SHIFT ); + +/* registers are 64-bit */ + *data_sz =8 ; + +return 0 ; +} +static inline void i40e_rxd_to_vlan_tci (struct rte_mbuf *mb,volatile union i40e_rx_desc *rxdp) {
drivers/net/i40e/i40e_rxtx.h +2 -0
@@ -248,6 +248,8 @@ uint16_t i40e_recv_scattered_pkts_vec_avx2(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);uint16_t i40e_xmit_pkts_vec_avx2 (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts); +int i40e_get_wake_addr (void *rx_queue,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *value,uint8_t *data_sz);/* For each value it means, datasheet of hardware can tell more details *
Signature · 1 line
2.17.1
PATCH v10 8/9 net/ice: implement power management API
In reply to Liang MaImplement support for the power management API by implementing a get_wake_addr function that will return an address of an RX ring's status bit.
--- drivers/net/ice/ice_ethdev.c | 1 + drivers/net/ice/ice_rxtx.c | 26 ++++++++++++++++++++++++++ drivers/net/ice/ice_rxtx.h | 2 ++ 3 files changed, 29 insertions(+)
drivers/net/ice/ice_ethdev.c +1 -0
@@ -216,6 +216,7 @@ static const struct eth_dev_ops ice_eth_dev_ops = { .udp_tunnel_port_add = ice_dev_udp_tunnel_port_add, .udp_tunnel_port_del = ice_dev_udp_tunnel_port_del, .tx_done_cleanup = ice_tx_done_cleanup, + .get_wake_addr = ice_get_wake_addr, };/* store statistics names and its offset in stats structure */
drivers/net/ice/ice_rxtx.c +26 -0
@@ -26,6 +26,32 @@ uint64_t rte_net_ice_dynflag_proto_xtr_ipv6_flow_mask;uint64_t rte_net_ice_dynflag_proto_xtr_tcp_mask;uint64_t rte_net_ice_dynflag_proto_xtr_ip_offset_mask; +int ice_get_wake_addr (void *rx_queue,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *mask,uint8_t *data_sz) +{ +volatile union ice_rx_flex_desc *rxdp; +struct ice_rx_queue *rxq = rx_queue; +uint16_t desc; + + desc = rxq->rx_tail; + rxdp = &rxq->rx_ring[desc]; +/* watch for changes in status bit */ + *tail_desc_addr = &rxdp->wb.status_error0; + +/* +* we expect the DD bit to be set to 1 if this descriptor was already +* written to. +*/ + *expected =rte_cpu_to_le_16 (1 <<ICE_RX_FLEX_DESC_STATUS0_DD_S ); + *mask =rte_cpu_to_le_16 (1 <<ICE_RX_FLEX_DESC_STATUS0_DD_S ); + +/* register is 16-bit */ + *data_sz =2 ; + +return 0 ; +} + +static inline uint8_t ice_proto_xtr_type_to_rxdid (uint8_t xtr_type) {
drivers/net/ice/ice_rxtx.h +2 -0
@@ -250,6 +250,8 @@ uint16_t ice_xmit_pkts_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);int ice_fdir_programming (struct ice_pf *pf,struct ice_fltr_desc *fdir_desc);int ice_tx_done_cleanup (void *txq,uint32_t free_cnt); +int ice_get_wake_addr (void *rx_queue,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *mask,uint8_t *data_sz);#define FDIR_PARSING_ENABLE_PER_QUEUE (ad, on) do { \ int i; \
Signature · 1 line
2.17.1
PATCH v10 9/9 examples/l3fwd-power: enable PMD power mgmt
In reply to Liang MaAdd PMD power management feature support to l3fwd-power sample app.
---
Notes:
v8:
- Add return status check for queue enable
v6:
- Fixed typos in documentation
--- .../sample_app_ug/l3_forward_power_man.rst | 13 ++++++ examples/l3fwd-power/main.c | 46 ++++++++++++++++++- 2 files changed, 58 insertions(+), 1 deletion(-)
doc/guides/sample_app_ug/l3_forward_power_man.rst +13 -0
@@ -109,6 +109,8 @@ where, * --telemetry: Telemetry mode. +* --pmd-mgmt: PMD power management mode. + See :doc:`l3_forward` for details. The L3fwd-power example reuses the L3fwd command line options.
@@ -455,3 +457,14 @@ reference cycles and accordingly busy rate is set to either 0% or The new stats ``empty_poll`` , ``full_poll`` and ``busy_percent`` can be viewed by running the script ``/usertools/dpdk-telemetry-client.py`` and selecting the menu option ``Send for global Metrics``. + +PMD power management Mode +------------------------- + +The PMD power management mode support for ``l3fwd-power`` is a standalone mode, in this mode +``l3fwd-power`` does simple l3fwding along with enable the power saving scheme on specific +port/queue/lcore. Main purpose for this mode is to demonstrate how to use the PMD power management API. + +.. code-block:: console + + ./build/examples/dpdk-l3fwd-power -l 1-3 -- --pmd-mgmt -p 0x0f --config="(0,0,2),(0,1,3)"
examples/l3fwd-power/main.c +45 -1
100 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
In reply to Liang MaPlease be more patient.
I asked some questions on v9 (ethdev API is not generic enough), and you send a v10 in the same minute you reply, without making sure I agree with your answers.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible. This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?
Regards
Liang
Quoted 7 lines · Thomas Monjalon
Please be more patient.
I asked some questions on v9 (ethdev API is not generic enough), and you send a v10 in the same minute you reply, without making sure I agree with your answers.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
Quoted 3 lines · Ma
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.
I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.
Quoted 13 lines
This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?
Regards
LiangQuoted 7 lines · Thomas Monjalon
Please be more patient.
I asked some questions on v9 (ethdev API is not generic enough), and you send a v10 in the same minute you reply, without making sure I agree with your answers.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
28/10/2020 14:49, Jerin Jacob:
Quoted 5 lines · Ma
Quoted 3 lines
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From the Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.
Quoted 8 lines
I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.
I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)
Quoted 1 line
Quoted 1 line
This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?
Being experimental is not an excuse to throw something which is not satisfying.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
Quoted 17 lines
28/10/2020 14:49, Jerin Jacob:
Quoted 4 lines · Ma
Quoted 3 lines
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From theCommunity Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.
Quoted 9 lines
I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.
I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy: cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.
Quoted 4 lines
I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)
It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?
Being experimental is not an excuse to throw something which is not satisfying.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
Quoted 26 lines · Konstantin
Quoted 18 lines
28/10/2020 14:49, Jerin Jacob:
Quoted 4 lines · Ma
Quoted 3 lines
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From theCommunity Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.
Quoted 9 lines
I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy:cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.
Quoted 8 lines
Quoted 4 lines
I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.
Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, + uint64_t *expected, uint64_t *mask, uint8_t *data_sz);
Can we make it as
typedef void (*core_sleep_t)(void *rxq)
if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.
Note: core_sleep_t can take some more arguments such as enumerated policy if something more needs to be pushed to the driver.
Thoughts?
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?
Being experimental is not an excuse to throw something which is not satisfying.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
Quoted 37 lines · Jerin Jacob
Quoted 26 lines · Konstantin
Quoted 18 lines
28/10/2020 14:49, Jerin Jacob:
Quoted 4 lines · Ma
Quoted 3 lines
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From theCommunity Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.
Quoted 9 lines
I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy:cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.Quoted 8 lines
Quoted 4 lines
I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.
Quoted 7 lines
Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, +uint64_t*expected,uint64_t*mask,uint8_t*data_sz);Can we make it as
typedef void (*core_sleep_t)(void *rxq)
How about void (*eth_core_sleep_helper_t)(void *rxq, enum scheme, void *paramter) by this way, PMD can cast the parameter accorind to the scheme.
e.g. if scheme MEM_MONITOR then cast to umwait way.
however, this will introduce another problem.
we need add PMD query callback to figure out if PMD support this scheme.
Quoted 19 lines
if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.Note:
core_sleep_tcan take some more arguments such as enumerated policy if something more needs to be pushed to the driver.Thoughts?
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?
Being experimental is not an excuse to throw something which is not satisfying.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
Quoted 44 lines · Ma
Quoted 42 lines · Jerin Jacob
Quoted 26 lines · Konstantin
Quoted 18 lines
28/10/2020 14:49, Jerin Jacob:
Quoted 4 lines · Ma
Quoted 3 lines
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From theCommunity Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.
Quoted 9 lines
I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy:cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.Quoted 8 lines
Quoted 4 lines
I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.
Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, +uint64_t*expected,uint64_t*mask,uint8_t*data_sz);
Quoted 7 lines
Quoted 2 lines
Can we make it as
typedef void (*core_sleep_t)(void *rxq)How about void (*eth_core_sleep_helper_t)(void *rxq, enum scheme, void *paramter) by this way, PMD can cast the parameter accorind to the scheme.
e.g. if schemeMEM_MONITORthen cast to umwait way.
however, this will introduce another problem.
we need add PMD query callback to figure out if PMD support this scheme.
I thought scheme/policy something that "application cares" like below not arch specifics
1) wakeup up on first packet,
2) wake me up on first packet or timeout 100 us etc
Yes. We can have query on type of the policies supported.
Quoted 19 lines
Quoted 19 lines
if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.Note:
core_sleep_tcan take some more arguments such as enumerated policy if something more needs to be pushed to the driver.Thoughts?
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?
Being experimental is not an excuse to throw something which is not satisfying.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
Quoted 53 lines · Jerin Jacob
Quoted 44 lines · Ma
Quoted 42 lines · Jerin Jacob
Quoted 26 lines · Konstantin
Quoted 18 lines
28/10/2020 14:49, Jerin Jacob:
Quoted 4 lines · Ma
Quoted 3 lines
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From theCommunity Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.
Quoted 9 lines
I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy:cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.Quoted 8 lines
Quoted 4 lines
I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.
Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, +uint64_t*expected,uint64_t*mask,uint8_t*data_sz);Quoted 7 lines
Quoted 2 lines
Can we make it as
typedef void (*core_sleep_t)(void *rxq)How about void (*eth_core_sleep_helper_t)(void *rxq, enum scheme, void *paramter) by this way, PMD can cast the parameter accorind to the scheme.
e.g. if schemeMEM_MONITORthen cast to umwait way.
however, this will introduce another problem.
we need add PMD query callback to figure out if PMD support this scheme.
Quoted 4 lines
I thought scheme/policy something that "application cares" like below not arch specifics
1) wakeup up on first packet,
2) wake me up on first packet or timeout 100 us etc
I need clarify about current design a bit. the purposed API just get target address. the API itself(include the PMD callback) will not demand the processor to idle/sleep. we use the post rx callback to do that. for ethdev layer, this API only is used to fetch target address from PMD, which make minmal impact to existing code.
Quoted 22 lines
Yes. We can have query on type of the policies supported.
Quoted 19 lines
Quoted 19 lines
if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.Note:
core_sleep_tcan take some more arguments such as enumerated policy if something more needs to be pushed to the driver.Thoughts?
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?
Being experimental is not an excuse to throw something which is not satisfying.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
Quoted 55 lines · Ma
Quoted 53 lines · Jerin Jacob
Quoted 51 lines · Ma
Quoted 37 lines · Jerin Jacob
Quoted 26 lines · Konstantin
Quoted 18 lines
28/10/2020 14:49, Jerin Jacob:
Quoted 4 lines · Ma
Quoted 3 lines
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From theCommunity Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.
Quoted 9 lines
I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy:cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.Quoted 8 lines
Quoted 4 lines
I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.
Quoted 7 lines
Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, +uint64_t*expected,uint64_t*mask,uint8_t*data_sz);Can we make it as
typedef void (*core_sleep_t)(void *rxq)How about void (*eth_core_sleep_helper_t)(void *rxq, enum scheme, void *paramter) by this way, PMD can cast the parameter accorind to the scheme.
e.g. if schemeMEM_MONITORthen cast to umwait way.
however, this will introduce another problem.
we need add PMD query callback to figure out if PMD support this scheme.
Quoted 8 lines
Quoted 4 lines
I thought scheme/policy something that "application cares" like below not arch specifics
1) wakeup up on first packet,
2) wake me up on first packet or timeout 100 us etcI need clarify about current design a bit. the purposed API just get target address. the API itself(include the PMD callback) will not demand the processor to idle/sleep. we use the post rx callback to do that. for ethdev layer, this API only is used to fetch target address from PMD, which make minmal impact to existing code.
I understand that. But if we move that logic to common code in ethdev as a set of internal
PMD helper functions(Helper functions for class of devices and/or arch) then it should be
possible. Right?
I do understand that, It will call for some rework in the code. I will leave up to ethdev maintainers
on specifics.
Quoted 23 lines
Quoted 23 lines
Yes. We can have query on type of the policies supported.
Quoted 19 lines
Quoted 19 lines
if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.Note:
core_sleep_tcan take some more arguments such as enumerated policy if something more needs to be pushed to the driver.Thoughts?
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?
Being experimental is not an excuse to throw something which is not satisfying.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
Quoted 65 lines · Jerin Jacob
Quoted 55 lines · Ma
Quoted 53 lines · Jerin Jacob
Quoted 51 lines · Ma
Quoted 37 lines · Jerin Jacob
Quoted 26 lines · Konstantin
Quoted 18 lines
28/10/2020 14:49, Jerin Jacob:
Quoted 4 lines · Ma
Quoted 3 lines
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From theCommunity Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.
Quoted 9 lines
I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy:cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.Quoted 8 lines
Quoted 4 lines
I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.
Quoted 7 lines
Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, +uint64_t*expected,uint64_t*mask,uint8_t*data_sz);Can we make it as
typedef void (*core_sleep_t)(void *rxq)How about void (*eth_core_sleep_helper_t)(void *rxq, enum scheme, void *paramter) by this way, PMD can cast the parameter accorind to the scheme.
e.g. if schemeMEM_MONITORthen cast to umwait way.
however, this will introduce another problem.
we need add PMD query callback to figure out if PMD support this scheme.Quoted 8 lines
Quoted 4 lines
I thought scheme/policy something that "application cares" like below not arch specifics
1) wakeup up on first packet,
2) wake me up on first packet or timeout 100 us etcI need clarify about current design a bit. the purposed API just get target address. the API itself(include the PMD callback) will not demand the processor to idle/sleep. we use the post rx callback to do that. for ethdev layer, this API only is used to fetch target address from PMD, which make minmal impact to existing code.
Quoted 5 lines
I understand that. But if we move that logic to common code in ethdev as a set of internal
PMD helper functions(Helper functions for class of devices and/or arch) then it should be
possible. Right?
Although that's possible, but power manangment logic will be duplicated in all related PMD. Advantage of current design is decouple power mgmt logic with PMD, just need minmal help from PMD. there are 3 scheme already which are implemented as post rx callback. Put all power mgmt logic into PMD is way too heavy. Also current design is very easy to extend. Any new scheme has no impact to ethdev layer unless it need some kind of help from PMD. we have another 2 scheme, PAUSE and Freq Scale dont not need any PMD help.
Quoted 28 lines
I do understand that, It will call for some rework in the code. I will leave up to ethdev maintainers
on specifics.Quoted 23 lines
Quoted 23 lines
Yes. We can have query on type of the policies supported.
Quoted 19 lines
Quoted 19 lines
if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.Note:
core_sleep_tcan take some more arguments such as enumerated policy if something more needs to be pushed to the driver.Thoughts?
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?
Being experimental is not an excuse to throw something which is not satisfying.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
In reply to Jerin JacobQuoted 39 lines
Quoted 25 lines
Quoted 17 lines
28/10/2020 14:49, Jerin Jacob:
Quoted 4 lines · Ma
Quoted 3 lines
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From theCommunity Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.
Quoted 9 lines
I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy:cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.Quoted 8 lines
Quoted 4 lines
I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.
Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, +uint64_t*expected,uint64_t*mask,uint8_t*data_sz);
Quoted 6 lines
Can we make it as
typedef void (*core_sleep_t)(void *rxq)if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.
Ok I see, thanks for explanation.
From my perspective main disadvantage of such approach -
it can't be extended easily.
If/when will have an ability for core to sleep/wake-up on multiple events (multiple addresses) will have to either rework that API again.
Quoted 15 lines
Note:
core_sleep_tcan take some more arguments such as enumerated policy if something more needs to be pushed to the driver.Thoughts?
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?
Being experimental is not an excuse to throw something which is not satisfying.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
Quoted 47 lines · Konstantin
Quoted 40 lines
Quoted 26 lines
Quoted 18 lines
28/10/2020 14:49, Jerin Jacob:
Quoted 4 lines · Ma
Quoted 3 lines
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From theCommunity Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.
Quoted 9 lines
I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy:cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.Quoted 8 lines
Quoted 4 lines
I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.
Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, +uint64_t*expected,uint64_t*mask,uint8_t*data_sz);Quoted 6 lines
Can we make it as
typedef void (*core_sleep_t)(void *rxq)if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.
Quoted 5 lines
Ok I see, thanks for explanation.
From my perspective main disadvantage of such approach -
it can't be extended easily.
If/when will have an ability for core to sleep/wake-up on multiple events (multiple addresses) will have to either rework that API again.
I think, we can enumerate the policies and pass the associated structures as input to the driver.
Quoted 16 lines
Quoted 16 lines
Note:
core_sleep_tcan take some more arguments such as enumerated policy if something more needs to be pushed to the driver.Thoughts?
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?
Being experimental is not an excuse to throw something which is not satisfying.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
Quoted 64 lines
-----Original Message-----
From: Jerin Jacob <[email protected]>
Sent: Wednesday, October 28, 2020 3:40 PM
To: Ananyev, Konstantin <[email protected]>
Cc: Thomas Monjalon <[email protected]>; Ma, Liang J <[email protected]>; dpdk-dev <[email protected]>; Ruifeng Wang (Arm Technology China) <[email protected]>; Wang, Haiyue <[email protected]>; Richardson, Bruce <[email protected]>; Hunt, David <[email protected]>; Neil Horman <[email protected]>; McDaniel, Timothy <[email protected]>; Eads, Gage <[email protected]>; Marcin Wojtas <[email protected]>; Guy Tzalik <[email protected]>; Ajit Khaparde <[email protected]>; Harman Kalra <[email protected]>; John Daley <[email protected]>; Wei Hu (Xavier <[email protected]>; Ziyang Xuan <[email protected]>; [email protected]; Yong Wang <[email protected]>; [email protected]
Subject: Re: [PATCH v10 0/9] Add PMD power mgmtQuoted 47 lines · Konstantin
Quoted 46 lines
Quoted 26 lines
Quoted 18 lines
28/10/2020 14:49, Jerin Jacob:
Quoted 4 lines · Ma
Quoted 3 lines
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From theCommunity Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.
Quoted 9 lines
I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy:cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.Quoted 8 lines
Quoted 4 lines
I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.
Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, +uint64_t*expected,uint64_t*mask,uint8_t*data_sz);Can we make it as
typedef void (*core_sleep_t)(void *rxq)if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.
Quoted 8 lines
Quoted 5 lines
Ok I see, thanks for explanation.
From my perspective main disadvantage of such approach -
it can't be extended easily.
If/when will have an ability for core to sleep/wake-up on multiple events (multiple addresses) will have to either rework that API again.I think, we can enumerate the policies and pass the associated structures as input to the driver.
What I am trying to say: with that API we will not be able to wait for events from multiple devices (HW queues).
I.E. something like that:
get_wake_addr(port=X, ..., &addr[0], ...);get_wake_addr(port=Y,..., &addr[1],...);wait_on_multi(addr, 2);
wouldn't be possible.
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Note:
core_sleep_tcan take some more arguments such as enumerated policy if something more needs to be pushed to the driver.Thoughts?
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?
Being experimental is not an excuse to throw something which is not satisfying.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
Quoted 78 lines · Konstantin
Quoted 65 lines
-----Original Message-----
From: Jerin Jacob <[email protected]>
Sent: Wednesday, October 28, 2020 3:40 PM
To: Ananyev, Konstantin <[email protected]>
Cc: Thomas Monjalon <[email protected]>; Ma, Liang J <[email protected]>; dpdk-dev <[email protected]>; Ruifeng Wang (Arm Technology China) <[email protected]>; Wang, Haiyue <[email protected]>; Richardson, Bruce <[email protected]>; Hunt, David <[email protected]>; Neil Horman <[email protected]>; McDaniel, Timothy <[email protected]>; Eads, Gage <[email protected]>; Marcin Wojtas <[email protected]>; Guy Tzalik <[email protected]>; Ajit Khaparde <[email protected]>; Harman Kalra <[email protected]>; John Daley <[email protected]>; Wei Hu (Xavier <[email protected]>; Ziyang Xuan <[email protected]>; [email protected]; Yong Wang <[email protected]>; [email protected]
Subject: Re: [PATCH v10 0/9] Add PMD power mgmtQuoted 47 lines · Konstantin
Quoted 46 lines
Quoted 26 lines
Quoted 18 lines
28/10/2020 14:49, Jerin Jacob:
Quoted 4 lines · Ma
Quoted 3 lines
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From theCommunity Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.
Quoted 9 lines
I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy:cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.Quoted 8 lines
Quoted 4 lines
I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.
Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, +uint64_t*expected,uint64_t*mask,uint8_t*data_sz);Can we make it as
typedef void (*core_sleep_t)(void *rxq)if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.Quoted 8 lines
Quoted 5 lines
Ok I see, thanks for explanation.
From my perspective main disadvantage of such approach -
it can't be extended easily.
If/when will have an ability for core to sleep/wake-up on multiple events (multiple addresses) will have to either rework that API again.I think, we can enumerate the policies and pass the associated structures as input to the driver.
What I am trying to say: with that API we will not be able to wait for events from multiple devices (HW queues).
I.E. something like that:
Quoted 5 lines
get_wake_addr(port=X, ..., &addr[0], ...);get_wake_addr(port=Y,..., &addr[1],...);wait_on_multi(addr, 2);wouldn't be possible.
I see. But the current implementation dictates the only queue bound to a core. Right?
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Note:
core_sleep_tcan take some more arguments such as enumerated policy if something more needs to be pushed to the driver.Thoughts?
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?
Being experimental is not an excuse to throw something which is not satisfying.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
Quoted 85 lines
Quoted 13 lines · Konstantin
Quoted 13 lines
-----Original Message-----
From: Jerin Jacob <[email protected]>
Sent: Wednesday, October 28, 2020 3:40 PM
To: Ananyev, Konstantin <[email protected]>
Cc: Thomas Monjalon <[email protected]>; Ma, Liang J <[email protected]>; dpdk-dev <[email protected]>; Ruifeng Wang (Arm Technology China) <[email protected]>; Wang, Haiyue <[email protected]>; Richardson, Bruce <[email protected]>; Hunt, David <[email protected]>; Neil Horman <[email protected]>; McDaniel, Timothy <[email protected]>; Eads, Gage <[email protected]>; Marcin Wojtas <[email protected]>; Guy Tzalik <[email protected]>; Ajit Khaparde <[email protected]>; Harman Kalra <[email protected]>; John Daley <[email protected]>; Wei Hu (Xavier <[email protected]>; Ziyang Xuan <[email protected]>; [email protected];Yong
Quoted 8 lines
Quoted 8 lines
Wang <[email protected]>; [email protected]
Subject: Re: [PATCH v10 0/9] Add PMD power mgmtQuoted 3 lines · Konstantin
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
28/10/2020 14:49, Jerin Jacob:
Quoted 4 lines · Ma
Quoted 3 lines
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. Fromthe
Quoted 52 lines
Quoted 39 lines
Quoted 39 lines
Quoted 38 lines
Quoted 18 lines
Quoted 10 lines
Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.
Quoted 9 lines
I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy:cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.Quoted 8 lines
Quoted 4 lines
I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.
Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, +uint64_t*expected,uint64_t*mask,uint8_t*data_sz);Can we make it as
typedef void (*core_sleep_t)(void *rxq)if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.Quoted 8 lines
Quoted 5 lines
Ok I see, thanks for explanation.
From my perspective main disadvantage of such approach -
it can't be extended easily.
If/when will have an ability for core to sleep/wake-up on multiple events (multiple addresses) will have to either rework that API again.I think, we can enumerate the policies and pass the associated structures as input to the driver.
What I am trying to say: with that API we will not be able to wait for events from multiple devices (HW queues).
I.E. something like that:
Quoted 8 lines
Quoted 5 lines
get_wake_addr(port=X, ..., &addr[0], ...);get_wake_addr(port=Y,..., &addr[1],...);wait_on_multi(addr, 2);wouldn't be possible.
I see. But the current implementation dictates the only queue bound to a core. Right?
Yes, current implementation of rte_power_monitor() supports only single address. Though proposed API for both ethdev (get_wake_addr) and power(rte_power_pmd_mgmt_queue_enable) don't dictate
one to one mapping as the only possible usage model.
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Note:
core_sleep_tcan take some more arguments such as enumerated policy if something more needs to be pushed to the driver.Thoughts?
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?
Being experimental is not an excuse to throw something which is not satisfying.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
In reply to Jerin JacobQuoted 59 lines · Jerin Jacob
Quoted 57 lines · Konstantin
Quoted 44 lines
Quoted 44 lines
Quoted 43 lines
Quoted 23 lines
Quoted 15 lines
28/10/2020 14:49, Jerin Jacob:
Quoted 4 lines · Ma
Quoted 3 lines
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From theCommunity Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.
Quoted 9 lines
I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy:cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.Quoted 8 lines
Quoted 4 lines
I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.
Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, +uint64_t*expected,uint64_t*mask,uint8_t*data_sz);Can we make it as
typedef void (*core_sleep_t)(void *rxq)if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.Quoted 8 lines
Quoted 5 lines
Ok I see, thanks for explanation.
From my perspective main disadvantage of such approach -
it can't be extended easily.
If/when will have an ability for core to sleep/wake-up on multiple events (multiple addresses) will have to either rework that API again.I think, we can enumerate the policies and pass the associated structures as input to the driver.
What I am trying to say: with that API we will not be able to wait for events from multiple devices (HW queues).
I.E. something like that:
Quoted 8 lines
Quoted 5 lines
get_wake_addr(port=X, ..., &addr[0], ...);get_wake_addr(port=Y,..., &addr[1],...);wait_on_multi(addr, 2);wouldn't be possible.
I see. But the current implementation dictates the only queue bound to a core. Right?
Current implementation only support 1:1 queue/core mapping is because of the limitation of umwait/umonitor which can not work with multiple address range. However, for other scheme like PASUE/Freq Scale have no such limitation. The proposed API itself doesn't limit the 1:1 queue/core mapping.
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Note:
core_sleep_tcan take some more arguments such as enumerated policy if something more needs to be pushed to the driver.Thoughts?
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?
Being experimental is not an excuse to throw something which is not satisfying.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
Quoted 111 lines
-----Original Message-----
From: Ma, Liang J <[email protected]>
Sent: Wednesday, October 28, 2020 11:48 AM
To: Jerin Jacob <[email protected]>
Cc: Ananyev, Konstantin <[email protected]>; Thomas Monjalon <[email protected]>; dpdk-dev <[email protected]>; Ruifeng Wang (Arm Technology China) <[email protected]>; Wang, Haiyue <[email protected]>; Richardson, Bruce <[email protected]>; Hunt, David <[email protected]>; Neil Horman <[email protected]>; McDaniel, Timothy <[email protected]>; Eads, Gage <[email protected]>; Marcin Wojtas <[email protected]>; Guy Tzalik <[email protected]>; Ajit Khaparde <[email protected]>; Harman Kalra <[email protected]>; John Daley <[email protected]>; Wei Hu (Xavier <[email protected]>; Ziyang Xuan <[email protected]>; [email protected]; Yong Wang <[email protected]>; [email protected]
Subject: Re: [PATCH v10 0/9] Add PMD power mgmtQuoted 4 lines · Jerin Jacob
Quoted 2 lines · Konstantin
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
28/10/2020 14:49, Jerin Jacob:
Quoted 1 line
On Wed, Oct 28, 2020 at 7:05 PM Liang, Ma
Quoted 3 lines · <[email protected]> wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don'tthink I can solve the issue of a generic API on my own. From the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Community Call last week Jerin also said that a generic was
investigated but that a single solution wasn't feasible.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etcby
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects?
or similar
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
thoughts on abstraction.
I think there is probably some sort of misunderstanding. This API is not about providing acync notification when next packet
arrives.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
This is about to putting core to sleep till some event (or timeout)
happens.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
From my perspective the closest analogy:
cond_timedwait(). So we need PMD to tell us what will be the address of the conditionvariable
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
we should sleep on.
Quoted 4 lines
I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen. Second problem: it is probably not generic enough (if we understandit well)
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.
Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, +uint64_t*expected,uint64_t*mask,uint8_t*data_sz);Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Can we make it as
typedef void (*core_sleep_t)(void *rxq)if we do such abstraction and "move the polling on memory by
HW/CPU"
Quoted 16 lines
Quoted 16 lines
Quoted 11 lines
Quoted 8 lines
Quoted 2 lines
to the driver using a helper function then I can think of abstracting in some way in all PMDs.
Ok I see, thanks for explanation.
From my perspective main disadvantage of such approach -
it can't be extended easily.
If/when will have an ability for core to sleep/wake-up on multiple events (multiple addresses) will have to either rework that API again.I think, we can enumerate the policies and pass the associated structures as input to the driver.
What I am trying to say: with that API we will not be able to wait for events from multiple devices (HW queues).
I.E. something like that:Quoted 8 lines
Quoted 5 lines
get_wake_addr(port=X, ..., &addr[0], ...);get_wake_addr(port=Y,..., &addr[1],...);wait_on_multi(addr, 2);wouldn't be possible.
I see. But the current implementation dictates the only queue bound to a core. Right?
Current implementation only support 1:1 queue/core mapping is because of the limitation of umwait/umonitor which can not work with multiple address range. However, for other scheme like PASUE/Freq Scale have no such limitation.
The proposed API itself doesn't limit the 1:1 queue/core mapping.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Note:
core_sleep_tcan take some more arguments such asenumerated
Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
policy if something more needs to be pushed to the driver.
Thoughts?
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
This API is experimental and other vendor support can be added
as needed. If there are any other open issue let me know?
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Being experimental is not an excuse to throw something which is not satisfying.
It would be nice if the low level definition of the UMWAIT and UMONOTOR instructions were split out into their own inline function or macro so that any PMD could use the intrinsic without being tied to ethdev or any of the other logic associated with this patch set. This would be similar to rte_wmb, and so on
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
On 28 Oct 09:54, McDaniel, Timothy wrote:
<snip>
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
It would be nice if the low level definition of the UMWAIT and UMONOTOR instructions were split out into their own inline function or macro so that any PMD could use the intrinsic without being tied to ethdev or any of the other logic associated with this patch set. This would be similar to
rte_wmb, and so on
Current patches split the intrinsics out. and already part of eal lib. any other PMD can re-use it easily.
Quoted 2 lines
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
In reply to Liang, MaQuoted 65 lines · Ma
Quoted 63 lines · Jerin Jacob
Quoted 61 lines · Konstantin
Quoted 44 lines
Quoted 44 lines
Quoted 43 lines
Quoted 23 lines
Quoted 15 lines
28/10/2020 14:49, Jerin Jacob:
Quoted 4 lines · Ma
Quoted 3 lines
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own. From theCommunity Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.
Quoted 9 lines
I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy:cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.Quoted 8 lines
Quoted 4 lines
I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.
Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, +uint64_t*expected,uint64_t*mask,uint8_t*data_sz);Can we make it as
typedef void (*core_sleep_t)(void *rxq)if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.Quoted 8 lines
Quoted 5 lines
Ok I see, thanks for explanation.
From my perspective main disadvantage of such approach -
it can't be extended easily.
If/when will have an ability for core to sleep/wake-up on multiple events (multiple addresses) will have to either rework that API again.I think, we can enumerate the policies and pass the associated structures as input to the driver.
What I am trying to say: with that API we will not be able to wait for events from multiple devices (HW queues).
I.E. something like that:
get_wake_addr(port=X, ..., &addr[0], ...);get_wake_addr(port=Y,..., &addr[1],...);wait_on_multi(addr, 2);
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
wouldn't be possible.
I see. But the current implementation dictates the only queue bound to a core. Right?
Current implementation only support 1:1 queue/core mapping is because of the limitation of umwait/umonitor which can not work with multiple address range. However, for other scheme like PASUE/Freq Scale have no such limitation. The proposed API itself doesn't limit the 1:1 queue/core mapping.
The PMD would not know if it is 1:1 queue/core or any other shared scheme. So the intelligence and decision making is best left to the application. I think PMD and the underlying hardware does not need to know what kind of power management scheme is implemented.
IMHO the original API which provides the address, value and mask should suffice. Any other callback or handshake between PMD and application may be an overkill.
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Note:
core_sleep_tcan take some more arguments such as enumerated policy if something more needs to be pushed to the driver.Thoughts?
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?
Being experimental is not an excuse to throw something which is not satisfying.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
Quoted 91 lines
-----Original Message-----
From: Ajit Khaparde <[email protected]>
Sent: Wednesday, October 28, 2020 5:02 PM
To: Ma, Liang J <[email protected]>
Cc: Jerin Jacob <[email protected]>; Ananyev, Konstantin <[email protected]>; Thomas Monjalon <[email protected]>; dpdk-dev <[email protected]>; Ruifeng Wang (Arm Technology China) <[email protected]>; Wang, Haiyue <[email protected]>; Richardson, Bruce <[email protected]>; Hunt, David <[email protected]>; Neil Horman <[email protected]>; McDaniel, Timothy <[email protected]>; Eads, Gage <[email protected]>; Marcin Wojtas <[email protected]>; Guy Tzalik <[email protected]>; Harman Kalra <[email protected]>; John Daley <[email protected]>; Wei Hu (Xavier <[email protected]>; Ziyang Xuan <[email protected]>; [email protected]; Yong Wang <[email protected]>; [email protected]
Subject: Re: [PATCH v10 0/9] Add PMD power mgmtQuoted 9 lines · Ma
Quoted 7 lines · Jerin Jacob
Quoted 5 lines · Konstantin
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
28/10/2020 14:49, Jerin Jacob:
Quoted 4 lines · Ma
Quoted 3 lines
Hi Thomas,
I think I addressed all of the questions in relation to V9. I don't think I can solve the issue of a generic API on my own.From the
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 39 lines
Quoted 39 lines
Quoted 38 lines
Quoted 18 lines
Quoted 10 lines
Community Call last week Jerin also said that a generic was investigated but that a single solution wasn't feasible.
Quoted 9 lines
I think, From the architecture point of view, the specific functionally of UMONITOR may not be abstracted.
But from the ethdev callback point of view, Can it be abstracted in such a way that packet notification available through checking interrupt status register or ring descriptor location, etc by the driver. Use that callback as a notification mechanism rather than defining a memory-based scheme that UMONITOR expects? or similar thoughts on abstraction.I think there is probably some sort of misunderstanding.
This API is not about providing acync notification when next packet arrives. This is about to putting core to sleep till some event (or timeout) happens. From my perspective the closest analogy:cond_timedwait().
So we need PMD to tell us what will be the address of the condition variable we should sleep on.Quoted 8 lines
Quoted 4 lines
I agree with Jerin.
The ethdev API is the blocking problem.
First problem: it is not well explained in doxygen.
Second problem: it is probably not generic enough (if we understand it well)It is an address to sleep(/wakeup) on, plus expected value. Honestly, I can't think-up of anything even more generic then that. If you guys have something particular in mind - please share.
Current PMD callback:
typedef int (*eth_get_wake_addr_t)(void *rxq, volatile void **tail_desc_addr, +uint64_t*expected,uint64_t*mask,uint8_t*data_sz);Can we make it as
typedef void (*core_sleep_t)(void *rxq)if we do such abstraction and "move the polling on memory by HW/CPU" to the driver using a helper function then
I can think of abstracting in some way in all PMDs.Quoted 8 lines
Quoted 5 lines
Ok I see, thanks for explanation.
From my perspective main disadvantage of such approach -
it can't be extended easily.
If/when will have an ability for core to sleep/wake-up on multiple events (multiple addresses) will have to either rework that API again.I think, we can enumerate the policies and pass the associated structures as input to the driver.
What I am trying to say: with that API we will not be able to wait for events from multiple devices (HW queues).
I.E. something like that:
get_wake_addr(port=X, ..., &addr[0], ...);get_wake_addr(port=Y,..., &addr[1],...);wait_on_multi(addr, 2);Quoted 8 lines
Quoted 4 lines
Quoted 1 line
wouldn't be possible.
I see. But the current implementation dictates the only queue bound to a core. Right?
Current implementation only support 1:1 queue/core mapping is because of the limitation of umwait/umonitor which can not work with multiple address range. However, for other scheme like PASUE/Freq Scale have no such limitation. The proposed API itself doesn't limit the 1:1 queue/core mapping.
Quoted 4 lines
The PMD would not know if it is 1:1 queue/core or any other shared scheme. So the intelligence and decision making is best left to the application. I think PMD and the underlying hardware does not need to know what kind of power management scheme is implemented.
Yep, good point. 100% agree.
Quoted 24 lines
IMHO the original API which provides the address, value and mask should suffice. Any other callback or handshake between PMD and application may be an overkill.
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Note:
core_sleep_tcan take some more arguments such as enumerated policy if something more needs to be pushed to the driver.Thoughts?
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
This API is experimental and other vendor support can be added as needed. If there are any other open issue let me know?
Being experimental is not an excuse to throw something which is not satisfying.
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
In reply to Liang MaQuoted 8 lines · Liang Ma
This patchset proposes a simple API for Ethernet drivers to cause the CPU to enter a power-optimized state while waiting for packets to arrive, along with a set of generic intrinsics that facilitate that. This is achieved through cooperation with the NIC driver that will allow us to know address of wake up event, and wait for writes on it.
Is the wake event the same as ring status or interrupt status register?
So in a way the PMD is passing the address of the next ring descriptor? So that instead of the PMD polling it, the application peeks at it and when ready asks the PMD to actually process the packet(s)?
Quoted 20 lines
On IA, this is achieved through using UMONITOR/UMWAIT instructions. They are used in their raw opcode form because there is no widespread compiler support for them yet. Still, the API is made generic enough to hopefully support other architectures, if they happen to implement similar instructions.
To achieve power savings, there is a very simple mechanism
used: we're counting empty polls, and if a certain threshold is reached, we get the address of next RX ring descriptor from the NIC driver, arm the monitoring hardware, and enter a power-optimized state. We will then wake up when either a timeout happens, or a write happens (or generally whenever CPU feels like waking up - this is platform- specific), and proceed as normal. The empty poll counter is reset whenever we actually get packets, so we only go to sleep when we know nothing is going on. The mechanism is generic which can be used for any write back descriptor.
Quoted 5 lines
Why are we putting it into ethdev as opposed to leaving this up to the application? Our customers specifically requested a way to do it wit minimal changes to the application code. The current approach allows to just flip a switch and automatically have power savings.
The application still has to know address of wake up event. Right? And then it will need the logic to count empty polls and the threshold? This will be done by application or something else?
Quoted 65 lines
- Only 1:1 core to queue mapping is supported, meaning that each lcore must at most handle RX on a single queue
- Support 3 type policies. UMWAIT/PAUSE/Frequency_Scale
- Power management is enabled per-queue
- The API doesn't extend to other device typesLiang Ma (9):
eal: add new x86 cpuid support for WAITPKG
eal: add power management intrinsics
eal: add intrinsics support check infrastructure
ethdev: add simple power management API
power: add PMD power management API and callback net/ixgbe: implement power management API net/i40e: implement power management API net/ice: implement power management API examples/l3fwd-power: enable PMD power mgmtdoc/guides/prog_guide/power_man.rst | 48 +++ doc/guides/rel_notes/release_20_11.rst | 15 + .../sample_app_ug/l3_forward_power_man.rst | 13 + drivers/net/i40e/i40e_ethdev.c | 1 + drivers/net/i40e/i40e_rxtx.c | 26 ++ drivers/net/i40e/i40e_rxtx.h | 2 + drivers/net/ice/ice_ethdev.c | 1 + drivers/net/ice/ice_rxtx.c | 26 ++ drivers/net/ice/ice_rxtx.h | 2 + drivers/net/ixgbe/ixgbe_ethdev.c | 1 + drivers/net/ixgbe/ixgbe_rxtx.c | 25 ++ drivers/net/ixgbe/ixgbe_rxtx.h | 2 + examples/l3fwd-power/main.c | 46 ++- lib/librte_eal/arm/include/meson.build | 1 + .../arm/include/rte_power_intrinsics.h | 60 ++++ lib/librte_eal/arm/rte_cpuflags.c | 6 + lib/librte_eal/include/generic/rte_cpuflags.h | 26 ++ .../include/generic/rte_power_intrinsics.h | 123 +++++++ lib/librte_eal/include/meson.build | 1 + lib/librte_eal/ppc/include/meson.build | 1 + .../ppc/include/rte_power_intrinsics.h | 60 ++++ lib/librte_eal/ppc/rte_cpuflags.c | 7 + lib/librte_eal/version.map | 1 + lib/librte_eal/x86/include/meson.build | 1 + lib/librte_eal/x86/include/rte_cpuflags.h | 1 + .../x86/include/rte_power_intrinsics.h | 135 ++++++++ lib/librte_eal/x86/rte_cpuflags.c | 14 + lib/librte_ethdev/rte_ethdev.c | 23 ++ lib/librte_ethdev/rte_ethdev.h | 33 ++ lib/librte_ethdev/rte_ethdev_driver.h | 28 ++ lib/librte_ethdev/version.map | 1 + lib/librte_power/meson.build | 5 +- lib/librte_power/rte_power_pmd_mgmt.c | 320 ++++++++++++++++++ lib/librte_power/rte_power_pmd_mgmt.h | 92 +++++ lib/librte_power/version.map | 4 + 35 files changed, 1148 insertions(+), 3 deletions(-) create mode 100644 lib/librte_eal/arm/include/rte_power_intrinsics.h create mode 100644 lib/librte_eal/include/generic/rte_power_intrinsics.h create mode 100644 lib/librte_eal/ppc/include/rte_power_intrinsics.h create mode 100644 lib/librte_eal/x86/include/rte_power_intrinsics.h create mode 100644 lib/librte_power/rte_power_pmd_mgmt.c create mode 100644 lib/librte_power/rte_power_pmd_mgmt.h--
2.17.1
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
Quoted 10 lines · Ajit Khaparde
Quoted 8 lines · Liang Ma
This patchset proposes a simple API for Ethernet drivers to cause the CPU to enter a power-optimized state while waiting for packets to arrive, along with a set of generic intrinsics that facilitate that. This is achieved through cooperation with the NIC driver that will allow us to know address of wake up event, and wait for writes on it.
Is the wake event the same as ring status or interrupt status register?
the wake event is any change happen to the monitoring address range. can be a ring descriptor(e.g. Done Bits) but not limiti to that.
Quoted 4 lines
So in a way the PMD is passing the address of the next ring descriptor? So that instead of the PMD polling it, the application peeks at it and when ready asks the PMD to actually process the packet(s)?
NO, after PMD pssing the address, processor will monitoring and wait on the address range(aka, move to a sub-state). any change to the address range caused by other source(another processor, NIC etc) will wake up the processor then start polling again.
Quoted 21 lines
Quoted 21 lines
On IA, this is achieved through using UMONITOR/UMWAIT instructions. They are used in their raw opcode form because there is no widespread compiler support for them yet. Still, the API is made generic enough to hopefully support other architectures, if they happen to implement similar instructions.
To achieve power savings, there is a very simple mechanism
used: we're counting empty polls, and if a certain threshold is reached, we get the address of next RX ring descriptor from the NIC driver, arm the monitoring hardware, and enter a power-optimized state. We will then wake up when either a timeout happens, or a write happens (or generally whenever CPU feels like waking up - this is platform- specific), and proceed as normal. The empty poll counter is reset whenever we actually get packets, so we only go to sleep when we know nothing is going on. The mechanism is generic which can be used for any write back descriptor.
Quoted 8 lines
Quoted 5 lines
Why are we putting it into ethdev as opposed to leaving this up to the application? Our customers specifically requested a way to do it wit minimal changes to the application code. The current approach allows to just flip a switch and automatically have power savings.
The application still has to know address of wake up event. Right? And then it will need the logic to count empty polls and the threshold? This will be done by application or something else?
the empty poll counting is done by the power library callback function which is included in the patch 5. APP has no need change any thing beside the initilization code(call the enable/disable API). please Ref the patch 9 which demostrate how to use it with l3fwd-power.
Quoted 66 lines
Quoted 66 lines
- Only 1:1 core to queue mapping is supported, meaning that each lcore must at most handle RX on a single queue
- Support 3 type policies. UMWAIT/PAUSE/Frequency_Scale
- Power management is enabled per-queue
- The API doesn't extend to other device typesLiang Ma (9):
eal: add new x86 cpuid support for WAITPKG
eal: add power management intrinsics
eal: add intrinsics support check infrastructure
ethdev: add simple power management API
power: add PMD power management API and callback net/ixgbe: implement power management API net/i40e: implement power management API net/ice: implement power management API examples/l3fwd-power: enable PMD power mgmtdoc/guides/prog_guide/power_man.rst | 48 +++ doc/guides/rel_notes/release_20_11.rst | 15 + .../sample_app_ug/l3_forward_power_man.rst | 13 + drivers/net/i40e/i40e_ethdev.c | 1 + drivers/net/i40e/i40e_rxtx.c | 26 ++ drivers/net/i40e/i40e_rxtx.h | 2 + drivers/net/ice/ice_ethdev.c | 1 + drivers/net/ice/ice_rxtx.c | 26 ++ drivers/net/ice/ice_rxtx.h | 2 + drivers/net/ixgbe/ixgbe_ethdev.c | 1 + drivers/net/ixgbe/ixgbe_rxtx.c | 25 ++ drivers/net/ixgbe/ixgbe_rxtx.h | 2 + examples/l3fwd-power/main.c | 46 ++- lib/librte_eal/arm/include/meson.build | 1 + .../arm/include/rte_power_intrinsics.h | 60 ++++ lib/librte_eal/arm/rte_cpuflags.c | 6 + lib/librte_eal/include/generic/rte_cpuflags.h | 26 ++ .../include/generic/rte_power_intrinsics.h | 123 +++++++ lib/librte_eal/include/meson.build | 1 + lib/librte_eal/ppc/include/meson.build | 1 + .../ppc/include/rte_power_intrinsics.h | 60 ++++ lib/librte_eal/ppc/rte_cpuflags.c | 7 + lib/librte_eal/version.map | 1 + lib/librte_eal/x86/include/meson.build | 1 + lib/librte_eal/x86/include/rte_cpuflags.h | 1 + .../x86/include/rte_power_intrinsics.h | 135 ++++++++ lib/librte_eal/x86/rte_cpuflags.c | 14 + lib/librte_ethdev/rte_ethdev.c | 23 ++ lib/librte_ethdev/rte_ethdev.h | 33 ++ lib/librte_ethdev/rte_ethdev_driver.h | 28 ++ lib/librte_ethdev/version.map | 1 + lib/librte_power/meson.build | 5 +- lib/librte_power/rte_power_pmd_mgmt.c | 320 ++++++++++++++++++ lib/librte_power/rte_power_pmd_mgmt.h | 92 +++++ lib/librte_power/version.map | 4 + 35 files changed, 1148 insertions(+), 3 deletions(-) create mode 100644 lib/librte_eal/arm/include/rte_power_intrinsics.h create mode 100644 lib/librte_eal/include/generic/rte_power_intrinsics.h create mode 100644 lib/librte_eal/ppc/include/rte_power_intrinsics.h create mode 100644 lib/librte_eal/x86/include/rte_power_intrinsics.h create mode 100644 lib/librte_power/rte_power_pmd_mgmt.c create mode 100644 lib/librte_power/rte_power_pmd_mgmt.h--
2.17.1
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
In reply to Liang Ma27/10/2020 15:59, Liang Ma:
Quoted 4 lines
Liang Ma (9):
eal: add new x86 cpuid support for WAITPKG
eal: add power management intrinsics
eal: add intrinsics support check infrastructure
EAL patches applied, thanks.
Quoted 1 line
ethdev: add simple power management API
Waiting for doxygen reword which may help to get acks for ethdev API.
Quoted 5 lines
power: add PMD power management API and callback net/ixgbe: implement power management API net/i40e: implement power management API net/ice: implement power management API examples/l3fwd-power: enable PMD power mgmt
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
Quoted 7 lines · Thomas Monjalon
27/10/2020 15:59, Liang Ma:
Quoted 4 lines
Liang Ma (9):
eal: add new x86 cpuid support for WAITPKG
eal: add power management intrinsics
eal: add intrinsics support check infrastructureEAL patches applied, thanks.
thanks!
Quoted 4 lines
Quoted 2 lines
ethdev: add simple power management API
Waiting for doxygen reword which may help to get acks for ethdev API.
we are working on this. I will send out v11 today.
Quoted 9 lines
Quoted 6 lines
power: add PMD power management API and callback net/ixgbe: implement power management API net/i40e: implement power management API net/ice: implement power management API examples/l3fwd-power: enable PMD power mgmt
Re: [dpdk-dev] [PATCH v10 0/9] Add PMD power mgmt
30/10/2020 10:36, Liang, Ma:
Quoted 7 lines · Thomas Monjalon
Quoted 6 lines
27/10/2020 15:59, Liang Ma:
Quoted 4 lines
Liang Ma (9):
eal: add new x86 cpuid support for WAITPKG
eal: add power management intrinsics
eal: add intrinsics support check infrastructure
Quoted 7 lines
Quoted 1 line
EAL patches applied, thanks.
thanks!
Quoted 4 lines
Quoted 2 lines
ethdev: add simple power management API
Waiting for doxygen reword which may help to get acks for ethdev API.
we are working on this. I will send out v11 today.
David sent some interesting questions about the EAL API. Maybe some explanations are missing, and we could improve the API. Please reply, I would like to solve the EAL questions quickly.
PATCH v8 2/10 eal: add power management intrinsics
In reply to Anatoly BurakovAdd two new power management intrinsics, and provide an implementation in eal/x86 based on UMONITOR/UMWAIT instructions. The instructions are implemented as raw byte opcodes because there is not yet widespread compiler support for these instructions.
The power management instructions provide an architecture-specific function to either wait until a specified TSC timestamp is reached, or optionally wait until either a TSC timestamp is reached or a memory location is written to. The monitor function also provides an optional comparison, to avoid sleeping when the expected write has already happened, and no more writes are expected.
For more details, please refer to Intel(R) 64 and IA-32 Architectures Software Developer's Manual, Volume 2.
--- lib/librte_eal/arm/include/meson.build | 1 + .../arm/include/rte_power_intrinsics.h | 60 ++++++++ .../include/generic/rte_power_intrinsics.h | 111 ++++++++++++++ lib/librte_eal/include/meson.build | 1 + lib/librte_eal/ppc/include/meson.build | 1 + .../ppc/include/rte_power_intrinsics.h | 60 ++++++++ lib/librte_eal/x86/include/meson.build | 1 + .../x86/include/rte_power_intrinsics.h | 135 ++++++++++++++++++ 8 files changed, 370 insertions(+) create mode 100644 lib/librte_eal/arm/include/rte_power_intrinsics.h create mode 100644 lib/librte_eal/include/generic/rte_power_intrinsics.h create mode 100644 lib/librte_eal/ppc/include/rte_power_intrinsics.h create mode 100644 lib/librte_eal/x86/include/rte_power_intrinsics.h
lib/librte_eal/arm/include/meson.build +1 -0
@@ -20,6 +20,7 @@ arch_headers = files( 'rte_pause_32.h', 'rte_pause_64.h', 'rte_pause.h', + 'rte_power_intrinsics.h', 'rte_prefetch_32.h', 'rte_prefetch_64.h', 'rte_prefetch.h',
lib/librte_eal/arm/include/rte_power_intrinsics.h new file +60 -0
60 lines not on this page · raw message
lib/librte_eal/include/generic/rte_power_intrinsics.h new file +111 -0
111 lines not on this page · raw message
lib/librte_eal/include/meson.build +1 -0
@@ -60,6 +60,7 @@ generic_headers = files( 'generic/rte_memcpy.h', 'generic/rte_pause.h', 'generic/rte_prefetch.h', + 'generic/rte_power_intrinsics.h', 'generic/rte_rwlock.h', 'generic/rte_spinlock.h', 'generic/rte_ticketlock.h',
lib/librte_eal/ppc/include/meson.build +1 -0
@@ -10,6 +10,7 @@ arch_headers = files( 'rte_io.h', 'rte_memcpy.h', 'rte_pause.h', + 'rte_power_intrinsics.h', 'rte_prefetch.h', 'rte_rwlock.h', 'rte_spinlock.h',
lib/librte_eal/ppc/include/rte_power_intrinsics.h new file +60 -0
60 lines not on this page · raw message
lib/librte_eal/x86/include/meson.build +1 -0
@@ -13,6 +13,7 @@ arch_headers = files( 'rte_io.h', 'rte_memcpy.h', 'rte_prefetch.h', + 'rte_power_intrinsics.h', 'rte_pause.h', 'rte_rtm.h', 'rte_rwlock.h',
lib/librte_eal/x86/include/rte_power_intrinsics.h new file +135 -0
135 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v8 3/10 eal: add intrinsics support check infrastructure
In reply to Anatoly BurakovCurrently, it is not possible to check support for intrinsics that are platform-specific, cannot be abstracted in a generic way, or do not have support on all architectures. The CPUID flags can be used to some extent, but they are only defined for their platform, while intrinsics will be available to all code as they are in generic headers.
This patch introduces infrastructure to check support for certain platform-specific intrinsics, and adds support for checking support for IA power management-related intrinsics for UMWAIT/UMONITOR and TPAUSE.
---
Notes:
v6:
- Fix the comments
v8:
- Rename eal version.map
--- lib/librte_eal/arm/rte_cpuflags.c | 6 +++++ lib/librte_eal/include/generic/rte_cpuflags.h | 26 +++++++++++++++++++ .../include/generic/rte_power_intrinsics.h | 12 +++++++++ lib/librte_eal/ppc/rte_cpuflags.c | 7 +++++ lib/librte_eal/version.map | 1 + lib/librte_eal/x86/rte_cpuflags.c | 12 +++++++++ 6 files changed, 64 insertions(+)
lib/librte_eal/arm/rte_cpuflags.c +6 -0
@@ -151,3 +151,9 @@ rte_cpu_get_flag_name(enum rte_cpu_flag_t feature)return NULL ;return rte_cpu_feature_table[feature].name; } + +void +rte_cpu_get_intrinsics_support (struct rte_cpu_intrinsics *intrinsics) +{ +memset (intrinsics,0 ,sizeof (*intrinsics)); +}
lib/librte_eal/include/generic/rte_cpuflags.h +26 -0
@@ -13,6 +13,32 @@#include "rte_common.h" #include <errno.h> +#include <rte_compat.h> + +/** +* Structure used to describe platform-specific intrinsics that may or may not +* be supported at runtime. +*/ +struct rte_cpu_intrinsics { +uint32_t power_monitor :1 ; +/**< indicates support for rte_power_monitor function */ +uint32_t power_pause :1 ; +/**< indicates support for rte_power_pause function */ +}; + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Check CPU support for various intrinsics at runtime. +* +* @param intrinsics +* Pointer to a structure to be filled. +*/ +__rte_experimental +void +rte_cpu_get_intrinsics_support (struct rte_cpu_intrinsics *intrinsics); +/** * Enumeration of all CPU features supported */
lib/librte_eal/include/generic/rte_power_intrinsics.h +12 -0
@@ -32,6 +32,10 @@* checked against the expected value, and if they match, the entering of * optimized power state may be aborted. * +* @warning It is responsibility of the user to check if this function is +* supported at runtime using `rte_cpu_get_features()` API call. Failing to do +* so may result in an illegal CPU instruction error. +* * @param p * Address to monitor for changes. * @param expected_value
@@ -69,6 +73,10 @@ static inline void rte_power_monitor(const volatile void *p,* This call will also lock a spinlock on entering sleep, and release it on * waking up the CPU. * +* @warning It is responsibility of the user to check if this function is +* supported at runtime using `rte_cpu_get_features()` API call. Failing to do +* so may result in an illegal CPU instruction error. +* * @param p * Address to monitor for changes. * @param expected_value
@@ -101,6 +109,10 @@ static inline void rte_power_monitor_sync(const volatile void *p,* Enter an architecture-defined optimized power state until a certain TSC * timestamp is reached. * +* @warning It is responsibility of the user to check if this function is +* supported at runtime using `rte_cpu_get_features()` API call. Failing to do +* so may result in an illegal CPU instruction error. +* * @param tsc_timestamp * Maximum TSC timestamp to wait for. Note that the wait behavior is * architecture-dependent.
lib/librte_eal/ppc/rte_cpuflags.c +7 -0
@@ -8,6 +8,7 @@#include <elf.h> #include <fcntl.h> #include <assert.h> +#include <string.h> #include <unistd.h> /* Symbolic values for the entries in the auxiliary table */
@@ -108,3 +109,9 @@ rte_cpu_get_flag_name(enum rte_cpu_flag_t feature)return NULL ;return rte_cpu_feature_table[feature].name; } + +void +rte_cpu_get_intrinsics_support (struct rte_cpu_intrinsics *intrinsics) +{ +memset (intrinsics,0 ,sizeof (*intrinsics)); +}
lib/librte_eal/version.map +1 -0
@@ -402,6 +402,7 @@ EXPERIMENTAL { rte_service_lcore_may_be_active; rte_vect_get_max_simd_bitwidth; rte_vect_set_max_simd_bitwidth; + rte_cpu_get_intrinsics_support; }; INTERNAL {
lib/librte_eal/x86/rte_cpuflags.c +12 -0
@@ -7,6 +7,7 @@#include <stdio.h> #include <errno.h> #include <stdint.h> +#include <string.h> #include "rte_cpuid.h"
@@ -179,3 +180,14 @@ rte_cpu_get_flag_name(enum rte_cpu_flag_t feature)return NULL ;return rte_cpu_feature_table[feature].name; } + +void +rte_cpu_get_intrinsics_support (struct rte_cpu_intrinsics *intrinsics) +{ +memset (intrinsics,0 ,sizeof (*intrinsics)); + +if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_WAITPKG )) { + intrinsics->power_monitor =1 ; + intrinsics->power_pause =1 ; + } +}
Signature · 1 line
2.17.1
PATCH v8 4/10 ethdev: add simple power management API
In reply to Anatoly BurakovAdd a simple API to allow getting address of next RX descriptor from the PMD, as well as release notes information.
---
Notes:
v8:
- Rename version map file name.
v7:
- Fixed queue ID validation
- Fixed documentation
v6:
- Rebase on top of latest main
- Ensure the API checks queue ID (Konstantin)
- Removed accidental inclusion of unrelated release notes
v5:
- Bring function format in line with other functions in the file
- Ensure the API is supported by the driver before calling it (Konstantin)
--- doc/guides/rel_notes/release_20_11.rst | 5 +++++ lib/librte_ethdev/rte_ethdev.c | 23 +++++++++++++++++++++ lib/librte_ethdev/rte_ethdev.h | 28 ++++++++++++++++++++++++++ lib/librte_ethdev/rte_ethdev_driver.h | 28 ++++++++++++++++++++++++++ lib/librte_ethdev/version.map | 1 + 5 files changed, 85 insertions(+)
doc/guides/rel_notes/release_20_11.rst +5 -0
@@ -139,6 +139,11 @@ New Features Hairpin Tx part flow rules can be inserted explicitly. New API is added to get the hairpin peer ports list. +* **ethdev: add 1 new EXPERIMENTAL API for PMD power management.** + + * ``rte_eth_get_wake_addr()`` + * add new eth_dev_ops ``get_wake_addr`` + * **Updated Broadcom bnxt driver.** Updated the Broadcom bnxt driver with new features and improvements, including:
lib/librte_ethdev/rte_ethdev.c +23 -0
@@ -5138,6 +5138,29 @@ rte_eth_tx_burst_mode_get(uint16_t port_id, uint16_t queue_id, dev->dev_ops->tx_burst_mode_get(dev, queue_id, mode)); } +int +rte_eth_get_wake_addr (uint16_t port_id,uint16_t queue_id, +volatile void **wake_addr,uint64_t *expected,uint64_t *mask, +uint8_t *data_sz) +{ +struct rte_eth_dev *dev; + +RTE_ETH_VALID_PORTID_OR_ERR_RET (port_id, -ENODEV ); + + dev = &rte_eth_devices[port_id]; + +RTE_FUNC_PTR_OR_ERR_RET (*dev->dev_ops->get_wake_addr, -ENOTSUP ); + +if (queue_id >= dev->data->nb_rx_queues) { +RTE_ETHDEV_LOG (ERR ,"Invalid RX queue_id=%u\n" , queue_id); +return -EINVAL ; + } + +return eth_err (port_id, + dev->dev_ops->get_wake_addr (dev->data->rx_queues[queue_id], + wake_addr, expected, mask, data_sz)); +} + int rte_eth_dev_set_mc_addr_list(uint16_t port_id, struct rte_ether_addr *mc_addr_set,
lib/librte_ethdev/rte_ethdev.h +28 -0
@@ -4364,6 +4364,34 @@ __rte_experimentalint rte_eth_tx_burst_mode_get (uint16_t port_id,uint16_t queue_id,struct rte_eth_burst_mode *mode); +/** +* Retrieve the wake up address for the receive queue. +* +* @param port_id +* The port identifier of the Ethernet device. +* @param queue_id +* The Rx queue on the Ethernet device for which information will be +* retrieved. +* @param wake_addr +* The pointer to the address which will be monitored. +* @param expected +* The pointer to value to be expected when descriptor is set. +* @param mask +* The pointer to comparison bitmask for the expected value. +* @param data_sz +* The pointer to data size for the expected value and comparison bitmask. +* +* @return +* - 0: Success. +* -ENOTSUP: Operation not supported. +* -EINVAL: Invalid parameters. +* -ENODEV: Invalid port ID. +*/ +__rte_experimental +int rte_eth_get_wake_addr(uint16_t port_id, uint16_t queue_id, +volatile void **wake_addr,uint64_t *expected,uint64_t *mask, +uint8_t *data_sz); +/** * Retrieve device registers and register attributes (number of registers and * register size)
lib/librte_ethdev/rte_ethdev_driver.h +28 -0
@@ -752,6 +752,32 @@ typedef int (*eth_hairpin_queue_peer_unbind_t) (struct rte_eth_dev *dev, uint16_t cur_queue, uint32_t direction);/**< @internal Unbind peer queue from the current queue. */ +/** +* @internal +* Get address of memory location whose contents will change whenever there is +* new data to be received on an RX queue. +* +* @param rxq +* Ethdev queue pointer. +* @param tail_desc_addr +* The pointer point to where the address will be stored. +* @param expected +* The pointer point to value to be expected when descriptor is set. +* @param mask +* The pointer point to comparison bitmask for the expected value. +* @param data_sz +* Data size for the expected value (can be 1, 2, 4, or 8 bytes) +* @return +* Negative errno value on error, 0 on success. +* +* @retval 0 +* Success +* @retval -EINVAL +* Invalid parameters +*/ +typedef int (*eth_get_wake_addr_t )(void *rxq,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *mask,uint8_t *data_sz); +/** * @internal A structure containing the functions exported by an Ethernet driver. */
@@ -910,6 +936,8 @@ struct eth_dev_ops {/**< Set up the connection between the pair of hairpin queues. */ eth_hairpin_queue_peer_unbind_t hairpin_queue_peer_unbind;/**< Disconnect the hairpin queues of a pair from each other. */ +eth_get_wake_addr_t get_wake_addr; +/**< Get next RX queue ring entry address. */ };/**
lib/librte_ethdev/version.map +1 -0
@@ -244,6 +244,7 @@ EXPERIMENTAL { rte_flow_get_restore_info; rte_flow_tunnel_action_decap_release; rte_flow_tunnel_item_release; + rte_eth_get_wake_addr; }; INTERNAL {
Signature · 1 line
2.17.1
PATCH v8 5/10 power: add PMD power management API and callback
In reply to Anatoly BurakovAdd a simple on/off switch that will enable saving power when no packets are arriving. It is based on counting the number of empty polls and, when the number reaches a certain threshold, entering an architecture-defined optimized power state that will either wait until a TSC timestamp expires, or when packets arrive.
This API mandates a core-to-single-queue mapping (that is, multiple queued per device are supported, but they have to be polled on different cores).
This design is using PMD RX callbacks.
1. UMWAIT/UMONITOR:
When a certain threshold of empty polls is reached, the core will go into a power optimized sleep while waiting on an address of next RX descriptor to be written to.
2. Pause instruction
Instead of move the core into deeper C state, this method uses the pause instruction to avoid busy polling.
3. Frequency scaling
Reuse existing DPDK power library to scale up/down core frequency depending on traffic volume.
---
Notes:
v8:
- Rename version map file name
v7:
- Fixed race condition (Konstantin)
- Slight rework of the structure of monitor code
- Added missing inline for wakeup
v6:
- Added wakeup mechanism for UMWAIT
- Removed memory allocation (everything is now allocated statically)
- Fixed various typos and comments
- Check for invalid queue ID
- Moved release notes to this patch
v5:
- Make error checking more robust
- Prevent initializing scaling if ACPI or PSTATE env wasn't set
- Prevent initializing UMWAIT path if PMD doesn't support get_wake_addr
- Add some debug logging
- Replace x86-specific code path to generic path using the intrinsic check
--- doc/guides/rel_notes/release_20_11.rst | 11 + lib/librte_power/meson.build | 5 +- lib/librte_power/rte_power_pmd_mgmt.c | 320 +++++++++++++++++++++++++ lib/librte_power/rte_power_pmd_mgmt.h | 92 +++++++ lib/librte_power/version.map | 4 + 5 files changed, 430 insertions(+), 2 deletions(-) create mode 100644 lib/librte_power/rte_power_pmd_mgmt.c create mode 100644 lib/librte_power/rte_power_pmd_mgmt.h
doc/guides/rel_notes/release_20_11.rst +11 -0
@@ -350,6 +350,17 @@ New Features * Replaced ``--scalar`` command-line option with ``--alg=<value>``, to allow the user to select the desired classify method. +* **Add PMD power management mechanism** + + 3 new Ethernet PMD power management mechanism is added through existing + RX callback infrastructure. + + * Add power saving scheme based on UMWAIT instruction (x86 only) + * Add power saving scheme based on ``rte_pause()`` + * Add power saving scheme based on frequency scaling through the power library + * Add new EXPERIMENTAL API ``rte_power_pmd_mgmt_queue_enable()`` + * Add new EXPERIMENTAL API ``rte_power_pmd_mgmt_queue_disable()`` + Removed Items -------------
lib/librte_power/meson.build +3 -2
@@ -9,6 +9,7 @@ sources = files('rte_power.c', 'power_acpi_cpufreq.c', 'power_kvm_vm.c', 'guest_channel.c', 'rte_power_empty_poll.c', 'power_pstate_cpufreq.c', + 'rte_power_pmd_mgmt.c', 'power_common.c') -headers = files('rte_power.h','rte_power_empty_poll.h') -deps += ['timer'] +headers = files('rte_power.h','rte_power_empty_poll.h','rte_power_pmd_mgmt.h') +deps += ['timer' ,'ethdev']
lib/librte_power/rte_power_pmd_mgmt.c new file +320 -0
320 lines not on this page · raw message
lib/librte_power/rte_power_pmd_mgmt.h new file +92 -0
92 lines not on this page · raw message
lib/librte_power/version.map +4 -0
@@ -34,4 +34,8 @@ EXPERIMENTAL { rte_power_guest_channel_receive_msg; rte_power_poll_stat_fetch; rte_power_poll_stat_update; + # added in 20.11 + rte_power_pmd_mgmt_queue_enable; + rte_power_pmd_mgmt_queue_disable; + };
Signature · 1 line
2.17.1
PATCH v8 6/10 net/ixgbe: implement power management API
In reply to Anatoly BurakovImplement support for the power management API by implementing a get_wake_addr function that will return an address of an RX ring's status bit.
--- drivers/net/ixgbe/ixgbe_ethdev.c | 1 + drivers/net/ixgbe/ixgbe_rxtx.c | 25 +++++++++++++++++++++++++ drivers/net/ixgbe/ixgbe_rxtx.h | 2 ++ 3 files changed, 28 insertions(+)
drivers/net/ixgbe/ixgbe_ethdev.c +1 -0
@@ -588,6 +588,7 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = { .udp_tunnel_port_del = ixgbe_dev_udp_tunnel_port_del, .tm_ops_get = ixgbe_tm_ops_get, .tx_done_cleanup = ixgbe_dev_tx_done_cleanup, + .get_wake_addr = ixgbe_get_wake_addr, };/*
drivers/net/ixgbe/ixgbe_rxtx.c +25 -0
@@ -1367,6 +1367,31 @@ const uint32_t RTE_PTYPE_INNER_L3_IPV4_EXT | RTE_PTYPE_INNER_L4_UDP, }; +int ixgbe_get_wake_addr (void *rx_queue,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *mask,uint8_t *data_sz) +{ +volatile union ixgbe_adv_rx_desc *rxdp; +struct ixgbe_rx_queue *rxq = rx_queue; +uint16_t desc; + + desc = rxq->rx_tail; + rxdp = &rxq->rx_ring[desc]; +/* watch for changes in status bit */ + *tail_desc_addr = &rxdp->wb.upper.status_error; + +/* +* we expect the DD bit to be set to 1 if this descriptor was already +* written to. +*/ + *expected =rte_cpu_to_le_32 (IXGBE_RXDADV_STAT_DD ); + *mask =rte_cpu_to_le_32 (IXGBE_RXDADV_STAT_DD ); + +/* the registers are 32-bit */ + *data_sz =4 ; + +return 0 ; +} +/* @note: fix ixgbe_dev_supported_ptypes_get() if any change here. */ static inline uint32_t ixgbe_rxd_pkt_info_to_pkt_type (uint32_t pkt_info,uint16_t ptype_mask)
drivers/net/ixgbe/ixgbe_rxtx.h +2 -0
@@ -299,5 +299,7 @@ uint64_t ixgbe_get_tx_port_offloads(struct rte_eth_dev *dev);uint64_t ixgbe_get_rx_queue_offloads (struct rte_eth_dev *dev);uint64_t ixgbe_get_rx_port_offloads (struct rte_eth_dev *dev);uint64_t ixgbe_get_tx_queue_offloads (struct rte_eth_dev *dev); +int ixgbe_get_wake_addr (void *rx_queue,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *mask,uint8_t *data_sz);#endif /* _IXGBE_RXTX_H_ */
Signature · 1 line
2.17.1
PATCH v8 7/10 net/i40e: implement power management API
In reply to Anatoly BurakovImplement support for the power management API by implementing a get_wake_addr function that will return an address of an RX ring's status bit.
--- drivers/net/i40e/i40e_ethdev.c | 1 + drivers/net/i40e/i40e_rxtx.c | 26 ++++++++++++++++++++++++++ drivers/net/i40e/i40e_rxtx.h | 2 ++ 3 files changed, 29 insertions(+)
drivers/net/i40e/i40e_ethdev.c +1 -0
@@ -513,6 +513,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = { .mtu_set = i40e_dev_mtu_set, .tm_ops_get = i40e_tm_ops_get, .tx_done_cleanup = i40e_tx_done_cleanup, + .get_wake_addr = i40e_get_wake_addr, };/* store statistics names and its offset in stats structure */
drivers/net/i40e/i40e_rxtx.c +26 -0
@@ -72,6 +72,32 @@#define I40E_TX_OFFLOAD_NOTSUP_MASK \ (PKT_TX_OFFLOAD_MASK ^ I40E_TX_OFFLOAD_MASK) +int +i40e_get_wake_addr (void *rx_queue,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *mask,uint8_t *data_sz) +{ +struct i40e_rx_queue *rxq = rx_queue; +volatile union i40e_rx_desc *rxdp; +uint16_t desc; + + desc = rxq->rx_tail; + rxdp = &rxq->rx_ring[desc]; +/* watch for changes in status bit */ + *tail_desc_addr = &rxdp->wb.qword1.status_error_len; + +/* +* we expect the DD bit to be set to 1 if this descriptor was already +* written to. +*/ + *expected =rte_cpu_to_le_64 (1 <<I40E_RX_DESC_STATUS_DD_SHIFT ); + *mask =rte_cpu_to_le_64 (1 <<I40E_RX_DESC_STATUS_DD_SHIFT ); + +/* registers are 64-bit */ + *data_sz =8 ; + +return 0 ; +} +static inline void i40e_rxd_to_vlan_tci (struct rte_mbuf *mb,volatile union i40e_rx_desc *rxdp) {
drivers/net/i40e/i40e_rxtx.h +2 -0
@@ -248,6 +248,8 @@ uint16_t i40e_recv_scattered_pkts_vec_avx2(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);uint16_t i40e_xmit_pkts_vec_avx2 (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts); +int i40e_get_wake_addr (void *rx_queue,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *value,uint8_t *data_sz);/* For each value it means, datasheet of hardware can tell more details *
Signature · 1 line
2.17.1
PATCH v8 8/10 net/ice: implement power management API
In reply to Anatoly BurakovImplement support for the power management API by implementing a get_wake_addr function that will return an address of an RX ring's status bit.
--- drivers/net/ice/ice_ethdev.c | 1 + drivers/net/ice/ice_rxtx.c | 26 ++++++++++++++++++++++++++ drivers/net/ice/ice_rxtx.h | 2 ++ 3 files changed, 29 insertions(+)
drivers/net/ice/ice_ethdev.c +1 -0
@@ -216,6 +216,7 @@ static const struct eth_dev_ops ice_eth_dev_ops = { .udp_tunnel_port_add = ice_dev_udp_tunnel_port_add, .udp_tunnel_port_del = ice_dev_udp_tunnel_port_del, .tx_done_cleanup = ice_tx_done_cleanup, + .get_wake_addr = ice_get_wake_addr, };/* store statistics names and its offset in stats structure */
drivers/net/ice/ice_rxtx.c +26 -0
@@ -26,6 +26,32 @@ uint64_t rte_net_ice_dynflag_proto_xtr_ipv6_flow_mask;uint64_t rte_net_ice_dynflag_proto_xtr_tcp_mask;uint64_t rte_net_ice_dynflag_proto_xtr_ip_offset_mask; +int ice_get_wake_addr (void *rx_queue,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *mask,uint8_t *data_sz) +{ +volatile union ice_rx_flex_desc *rxdp; +struct ice_rx_queue *rxq = rx_queue; +uint16_t desc; + + desc = rxq->rx_tail; + rxdp = &rxq->rx_ring[desc]; +/* watch for changes in status bit */ + *tail_desc_addr = &rxdp->wb.status_error0; + +/* +* we expect the DD bit to be set to 1 if this descriptor was already +* written to. +*/ + *expected =rte_cpu_to_le_16 (1 <<ICE_RX_FLEX_DESC_STATUS0_DD_S ); + *mask =rte_cpu_to_le_16 (1 <<ICE_RX_FLEX_DESC_STATUS0_DD_S ); + +/* register is 16-bit */ + *data_sz =2 ; + +return 0 ; +} + +static inline uint8_t ice_proto_xtr_type_to_rxdid (uint8_t xtr_type) {
drivers/net/ice/ice_rxtx.h +2 -0
@@ -250,6 +250,8 @@ uint16_t ice_xmit_pkts_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);int ice_fdir_programming (struct ice_pf *pf,struct ice_fltr_desc *fdir_desc);int ice_tx_done_cleanup (void *txq,uint32_t free_cnt); +int ice_get_wake_addr (void *rx_queue,volatile void **tail_desc_addr, +uint64_t *expected,uint64_t *mask,uint8_t *data_sz);#define FDIR_PARSING_ENABLE_PER_QUEUE (ad, on) do { \ int i; \
Signature · 1 line
2.17.1
PATCH v8 9/10 examples/l3fwd-power: enable PMD power mgmt
In reply to Anatoly BurakovAdd PMD power management feature support to l3fwd-power sample app.
---
Notes:
v8:
- Add return status check for queue enable
v6:
- Fixed typos in documentation
--- .../sample_app_ug/l3_forward_power_man.rst | 13 ++++++ examples/l3fwd-power/main.c | 46 ++++++++++++++++++- 2 files changed, 58 insertions(+), 1 deletion(-)
doc/guides/sample_app_ug/l3_forward_power_man.rst +13 -0
@@ -109,6 +109,8 @@ where, * --telemetry: Telemetry mode. +* --pmd-mgmt: PMD power management mode. + See :doc:`l3_forward` for details. The L3fwd-power example reuses the L3fwd command line options.
@@ -455,3 +457,14 @@ reference cycles and accordingly busy rate is set to either 0% or The new stats ``empty_poll`` , ``full_poll`` and ``busy_percent`` can be viewed by running the script ``/usertools/dpdk-telemetry-client.py`` and selecting the menu option ``Send for global Metrics``. + +PMD power management Mode +------------------------- + +The PMD power management mode support for ``l3fwd-power`` is a standalone mode, in this mode +``l3fwd-power`` does simple l3fwding along with enable the power saving scheme on specific +port/queue/lcore. Main purpose for this mode is to demonstrate how to use the PMD power management API. + +.. code-block:: console + + ./build/examples/dpdk-l3fwd-power -l 1-3 -- --pmd-mgmt -p 0x0f --config="(0,0,2),(0,1,3)"
examples/l3fwd-power/main.c +45 -1
100 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v8 10/10 doc: update programmer's guide for power library
In reply to Anatoly BurakovUpdate programmer's guide to document PMD power management usage.
--- doc/guides/prog_guide/power_man.rst | 42 +++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)
doc/guides/prog_guide/power_man.rst +42 -0
@@ -192,6 +192,45 @@ User Cases ---------- The mechanism can applied to any device which is based on polling. e.g. NIC, FPGA. +PMD Power Management API +------------------------ + +Abstract +~~~~~~~~ +Existing power management mechanisms require developers to change application +design or change code to make use of it. The PMD power management API provides a +convenient alternative by utilizing Ethernet PMD RX callbacks, and triggering +power saving whenever empty poll count reaches a certain number. + + * UMWAIT/UMONITOR + + This power saving scheme will put the CPU into optimized power state and use + the UMWAIT/UMONITOR instructions to monitor the Ethernet PMD RX descriptor + address, and wake the CPU up whenever there's new traffic. + + * Pause + + This power saving scheme will use the `rte_pause` function to avoid busy + polling. + + * Frequency scaling + + This power saving scheme will use existing power library functionality to + scale the core frequency up/down depending on traffic volume. + + +.. note:: + + Currently, this power management API is limited to mandatory mapping of 1 + queue to 1 core (multiple queues are supported, but they must be polled from + different cores). + +API Overview for PMD Power Management +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +* **Queue Enable**: Enable specific power scheme for certain queue/port/core + +* **Queue Disable**: Disable power scheme for certain queue/port/core + References ----------
@@ -200,3 +239,6 @@ References * The :doc:`../sample_app_ug/vm_power_management` chapter in the :doc:`../sample_app_ug/index` section. + +* The :doc:`../sample_app_ug/rxtx_callbacks` + chapter in the :doc:`../sample_app_ug/index` section.
Signature · 1 line
2.17.1