virtio-net: clarify coalescing parameters settings
PATCH v5 1/1 virtio-net: clarify coalescing parameters settings
The device can set any initial coalescing parameters (0 or non-zero) for the receive/send queue before the setting command is executed, not just 0, enhancing device performance even without DIM enabled.
So we need to clarify descriptions that don't fit the behavior.
---v4->v5:
- Add RB tags from Parav and Jason. Thanks!
v3->v4:
- Doesn't force the device to remember more stuff. @Parav
v2->v3:
- Clarify description to be more generic. @Parav
v1->v2:
- Update description. @Jason
device-types/net/description.tex | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
device-types/net/description.tex +11 -3
@@ -1803,6 +1803,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi for an enabled transmit/receive virtqueue whose index is \field{vq_index}. \end{enumerate} +If the VIRTIO_NET_F_NOTF_COAL or VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, +the device may apply any coalescing parameters to each transmit/receive virtqueue +before the driver successfully performs one of the VIRTIO_NET_CTRL_NOTF_COAL set commands. + +The driver can query the coalescing parameters of any enabled transmit/receive +virtqueue using the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command, before or after any +VIRTIO_NET_CTRL_NOTF_COAL set command is done. + The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class. If coalescing parameters are being set, the device applies the last coalescing parameters set for a
@@ -1885,17 +1893,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue. Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0. +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, the device MAY initialize them to any values. Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0. +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, the device MAY initialize them to any values. The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort: the device MAY generate notifications more or less frequently than specified. A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. -Upon reset, a device MUST initialize all coalescing parameters to 0. +Upon reset, a device MAY set any coalescing parameters for all transmit and receive virtqueues. \paragraph{Device Statistics}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Statistics}
Signature · 1 line
2.32.0.3.g01195cf9f
Quoted 11 lines · Heng Qi
The device can set any initial coalescing parameters (0 or non-zero) for the receive/send queue before the setting command is executed, not just 0, enhancing device performance even without DIM enabled.
So we need to clarify descriptions that don't fit the behavior.
Suggested-by Jason Wang <[email protected]>Signed-off-by Heng Qi <[email protected]>Reviewed-by Parav Pandit <[email protected]>Reviewed-by Jason Wang <[email protected]>
Hi, Cornelia!
I'm wondering if our voting infrastructure is already working properly, and if so, can we turn on voting for this update patch?
Thanks a lot!
Quoted 60 lines
---
v4->v5:
- Add RB tags from Parav and Jason. Thanks!
v3->v4:
- Doesn't force the device to remember more stuff. @Parav
v2->v3:
- Clarify description to be more generic. @Parav
v1->v2:
- Update description. @Jasondevice-types/net/description.tex | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)device-types/net/description.tex +11
-3@@ -1803,6 +1803,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi for an enabled transmit/receive virtqueue whose index is \field{vq_index}. \end{enumerate} +If the VIRTIO_NET_F_NOTF_COAL or VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, +the device may apply any coalescing parameters to each transmit/receive virtqueue +before the driver successfully performs one of the VIRTIO_NET_CTRL_NOTF_COAL set commands. + +The driver can query the coalescing parameters of any enabled transmit/receive +virtqueue using the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command, before or after any +VIRTIO_NET_CTRL_NOTF_COAL set command is done. + The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class. If coalescing parameters are being set, the device applies the last coalescing parameters set for a@@ -1885,17 +1893,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue. Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0. +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, the device MAY initialize them to any values. Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0. +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, the device MAY initialize them to any values. The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort: the device MAY generate notifications more or less frequently than specified. A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. -Upon reset, a device MUST initialize all coalescing parameters to 0. +Upon reset, a device MAY set any coalescing parameters for all transmit and receive virtqueues. \paragraph{Device Statistics}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Statistics}Signature · 1 line
2.32.0.3.g01195cf9f
Quoted 13 lines · Heng Qi
Quoted 11 lines · Heng Qi
The device can set any initial coalescing parameters (0 or non-zero) for the receive/send queue before the setting command is executed, not just 0, enhancing device performance even without DIM enabled.
So we need to clarify descriptions that don't fit the behavior.
Suggested-by Jason Wang <[email protected]>Signed-off-by Heng Qi <[email protected]>Reviewed-by Parav Pandit <[email protected]>Reviewed-by Jason Wang <[email protected]>
Quoted 4 lines
Hi, Cornelia!
I'm wondering if our voting infrastructure is already working properly, and if so, can we turn on voting for this update patch?
Do I need to repost a patch with the subject prefix [Request for vote]?
Quoted 64 lines
Thanks a lot!
Quoted 60 lines
---
v4->v5:
- Add RB tags from Parav and Jason. Thanks!
v3->v4:
- Doesn't force the device to remember more stuff. @Parav
v2->v3:
- Clarify description to be more generic. @Parav
v1->v2:
- Update description. @Jasondevice-types/net/description.tex | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)device-types/net/description.tex +11
-3@@ -1803,6 +1803,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi for an enabled transmit/receive virtqueue whose index is \field{vq_index}. \end{enumerate} +If the VIRTIO_NET_F_NOTF_COAL or VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, +the device may apply any coalescing parameters to each transmit/receive virtqueue +before the driver successfully performs one of the VIRTIO_NET_CTRL_NOTF_COAL set commands. + +The driver can query the coalescing parameters of any enabled transmit/receive +virtqueue using the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command, before or after any +VIRTIO_NET_CTRL_NOTF_COAL set command is done. + The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class. If coalescing parameters are being set, the device applies the last coalescing parameters set for a@@ -1885,17 +1893,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue. Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0. +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, the device MAY initialize them to any values. Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0. +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, the device MAY initialize them to any values. The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort: the device MAY generate notifications more or less frequently than specified. A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. -Upon reset, a device MUST initialize all coalescing parameters to 0. +Upon reset, a device MAY set any coalescing parameters for all transmit and receive virtqueues. \paragraph{Device Statistics}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Statistics}Signature · 1 line
2.32.0.3.g01195cf9f
Quoted 14 lines · Heng Qi
Quoted 13 lines · Heng Qi
Quoted 11 lines · Heng Qi
The device can set any initial coalescing parameters (0 or non-zero) for the receive/send queue before the setting command is executed, not just 0, enhancing device performance even without DIM enabled.
So we need to clarify descriptions that don't fit the behavior.
Suggested-by Jason Wang <[email protected]>Signed-off-by Heng Qi <[email protected]>Reviewed-by Parav Pandit <[email protected]>Reviewed-by Jason Wang <[email protected]>
Quoted 4 lines
Quoted 4 lines
Hi, Cornelia!
I'm wondering if our voting infrastructure is already working properly, and if so, can we turn on voting for this update patch?
Oh, seems I forgot to send that mail out -- platform still has issues :(
Quoted 2 lines
Do I need to repost a patch with the subject prefix [Request for vote]?
This one is already sitting on my list, no need to do so (and we haven't actually finalized what it should look like, I'm still waiting for more feedback before proposing a patch.)
Quoted 5 lines · On Tue, 28 May 2024 12:47:02 +0800 Heng Qi <[email protected]> wrote:
The device can set any initial coalescing parameters (0 or non-zero) for the receive/send queue before the setting command is executed, not just 0, enhancing device performance even without DIM enabled.
So we need to clarify descriptions that don't fit the behavior.
Sorry I'm late to the party -- again! Just for my understanding: how/why is this a clarification and not just a (basically incompatible) change?
I mean if I read this correctly, before the driver had the guaranty that if the parameters are not set by the driver, negotiating the feature does not introduce any coalescing. After this in theory the device could just pick some max value and potentially introduce maximal latency in certain scenarios.
I understand that it is probably in the best interest of the devices to not pick stupid defaults. But it is also probably in the best interest of the driver to set those params, and if the driver is going to set its values, the devices defaults are moot unless we assume that those may end up being used by the driver as a hint when deciding which parameter values to choose.
What am I missing here?
Regards,
Halil
Quoted 11 lines · Halil Pasic
Quoted 5 lines · On Tue, 28 May 2024 12:47:02 +0800 Heng Qi <[email protected]> wrote:
The device can set any initial coalescing parameters (0 or non-zero) for the receive/send queue before the setting command is executed, not just 0, enhancing device performance even without DIM enabled.
So we need to clarify descriptions that don't fit the behavior.
Sorry I'm late to the party -- again! Just for my understanding: how/why is this a clarification and not just a (basically incompatible) change?
In my opinion, "clarification" means that something may have been described incorrectly before, and we now need to discuss, explain clearly, and correct the possibly incorrect description.
Quoted 6 lines
I mean if I read this correctly, before the driver had the guaranty that if the parameters are not set by the driver, negotiating the feature does not introduce any coalescing. After this in theory the device could just pick some max value and potentially introduce maximal latency in certain scenarios.
"maximum latency" also means "throughput improvement".
Quoted 7 lines
I understand that it is probably in the best interest of the devices to not pick stupid defaults. But it is also probably in the best interest of the driver to set those params, and if the driver is going to set its values, the devices defaults are moot unless we assume that those may end up being used by the driver as a hint when deciding which parameter values to choose.
"Any values" is compatible with "0 for max-usecs and max-frames", and the device can choose "no coalescing".
"No coalescing" means "latency friendly", but it also means "a lot of interruptions and throughput unfriendly". If the device chooses a stupid maximum value, it is his choice (spec should give more devices choices instead of forcing them to choose "0" which is not the best practice). We can't talk about performance for drivers when the devices tend to choose any "stupid" designs.
We need relaxe the restrictions and makes the spec more reasonable.
Thanks.
Quoted 4 lines
What am I missing here?
Regards,
Halil
Quoted 8 lines · On Sat, 8 Jun 2024 10:34:22 +0800 Heng Qi <[email protected]> wrote:
Quoted 7 lines · Halil Pasic
Quoted 4 lines · On Tue, 28 May 2024 12:47:02 +0800 Heng Qi <[email protected]> wrote:
The device can set any initial coalescing parameters (0 or non-zero) for the receive/send queue before the setting command is executed, not just 0, enhancing device performance even without DIM enabled.
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
So we need to clarify descriptions that don't fit the behavior.
Sorry I'm late to the party -- again! Just for my understanding: how/why is this a clarification and not just a (basically incompatible) change?
In my opinion, "clarification" means that something may have been described incorrectly before, and we now need to discuss, explain clearly, and correct the possibly incorrect description.
I figure the difference in perceived semantics of the word "clarification" is at the root of my confusion. Let us have a look at
https://dictionary.cambridge.org/de/worterbuch/englisch/clarification
According to my understanding a "clarification", while an improvement in ease of understanding and/or decrease of ambiguity (possibly to no ambiguity at all) implies that what receiving a clarification is not outright wrong.
When rectifying something that is outright incorrect or wrong, I would refer to that with words like "correction", "fix", "erratum" or "corrigendum".
Quoted 9 lines
Quoted 6 lines
I mean if I read this correctly, before the driver had the guaranty that if the parameters are not set by the driver, negotiating the feature does not introduce any coalescing. After this in theory the device could just pick some max value and potentially introduce maximal latency in certain scenarios.
"maximum latency" also means "throughput improvement".
Under certain assumptions. But not necessarily. Again my concern is mostly the type of change. The virtio standard maintain a revision history appendix, and I would like to avoid the nature of this change being misrepresented there. If Connie and/or Michael think it is worth fixing, I believe it can be fixed with an editorial change.
AFAIU VIRTIO_NET_F_NOTF_COAL and VIRTIO_NET_F_VQ_NOTF_COAL are about to land with virtio-1.3, i.e. there is no released/standardized virtio version where the "initialize to 0" is released. In that sense it looks like we are still on time to change this. But I am not 100% certain. In any case I don't think this as a huge impact and I'm fine going ahead with the change.
Quoted 8 lines
Quoted 7 lines
I understand that it is probably in the best interest of the devices to not pick stupid defaults. But it is also probably in the best interest of the driver to set those params, and if the driver is going to set its values, the devices defaults are moot unless we assume that those may end up being used by the driver as a hint when deciding which parameter values to choose.
Quoted 5 lines
"Any values" is compatible with "0 for max-usecs and max-frames", and the device can choose "no coalescing".
"No coalescing" means "latency friendly", but it also means "a lot of interruptions and throughput unfriendly".
To what extent does virtio's "normal" notification suppression (VIRTIO_F_EVENT_IDX) would alleviate that in practice? At least in theory the interruption suppression could save us there right?
Quoted 7 lines
If the device chooses a stupid
maximum value, it is his choice (spec should give more devices choices instead of forcing them to choose "0" which is not the best practice). We can't talk about performance for drivers when the devices tend to choose any "stupid" designs.We need relaxe the restrictions and makes the spec more reasonable.
Hm, I see Linux virtio-net changes have landed with v6.0 and if I read those correctly the driver -- contrary to my initial expectation -- negotiates the feature, but does not set the parameters explicitly and thus keeps the defaults (until userspace decides to set the parameters). So it does matter whether the defaults are guaranteed to be 0 or not, and if not it does matter what defaults are chosen by the device.
One could even argue that those patches have been reviewed under the assumption that the device needs to use 0 as the default parameter value.
Well no strong opinions here. If the community is fine with it, I'm fine with it as well.
Thank you!
Regards,
Halil
Quoted 50 lines · Halil Pasic
Quoted 8 lines · On Sat, 8 Jun 2024 10:34:22 +0800 Heng Qi <[email protected]> wrote:
Quoted 7 lines · Halil Pasic
Quoted 4 lines · On Tue, 28 May 2024 12:47:02 +0800 Heng Qi <[email protected]> wrote:
The device can set any initial coalescing parameters (0 or non-zero) for the receive/send queue before the setting command is executed, not just 0, enhancing device performance even without DIM enabled.
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
So we need to clarify descriptions that don't fit the behavior.
Sorry I'm late to the party -- again! Just for my understanding: how/why is this a clarification and not just a (basically incompatible) change?
In my opinion, "clarification" means that something may have been described incorrectly before, and we now need to discuss, explain clearly, and correct the possibly incorrect description.
I figure the difference in perceived semantics of the word "clarification" is at the root of my confusion. Let us have a look at
https://dictionary.cambridge.org/de/worterbuch/englisch/clarificationAccording to my understanding a "clarification", while an improvement in ease of understanding and/or decrease of ambiguity (possibly to no ambiguity at all) implies that what receiving a clarification is not outright wrong.
When rectifying something that is outright incorrect or wrong, I would refer to that with words like "correction", "fix", "erratum" or "corrigendum".
Quoted 9 lines
Quoted 6 lines
I mean if I read this correctly, before the driver had the guaranty that if the parameters are not set by the driver, negotiating the feature does not introduce any coalescing. After this in theory the device could just pick some max value and potentially introduce maximal latency in certain scenarios.
"maximum latency" also means "throughput improvement".
Under certain assumptions. But not necessarily. Again my concern is mostly the type of change. The virtio standard maintain a revision history appendix, and I would like to avoid the nature of this change being misrepresented there. If Connie and/or Michael think it is worth fixing, I believe it can be fixed with an editorial change.
Quoted 6 lines
AFAIU
VIRTIO_NET_F_NOTF_COALandVIRTIO_NET_F_VQ_NOTF_COALare about to land with virtio-1.3, i.e. there is no released/standardized virtio version where the "initialize to 0" is released. In that sense it looks like we are still on time to change this. But I am not 100% certain. In any case I don't think this as a huge impact and I'm fine going ahead with the change.
Sorry for the late reply, I'm on vacation.
I agree with this, and I prefer to release this patch as a correction for
virtio1.3 instead of a new patch for virtio1.4, because if devices support moderation coalescing based on virtio1.3, and after virtio1.4 is released, these devices need to be updated again for a
more reasonable coalescing parameters.
Cornelia and Michael, what do you think?
Quoted 12 lines
Quoted 12 lines
Quoted 8 lines
I understand that it is probably in the best interest of the devices to not pick stupid defaults. But it is also probably in the best interest of the driver to set those params, and if the driver is going to set its values, the devices defaults are moot unless we assume that those may end up being used by the driver as a hint when deciding which parameter values to choose.
"Any values" is compatible with "0 for max-usecs and max-frames", and the device can choose "no coalescing".
Quoted 7 lines
Quoted 2 lines
"No coalescing" means "latency friendly", but it also means "a lot of interruptions and throughput unfriendly".
To what extent does virtio's "normal" notification suppression (
VIRTIO_F_EVENT_IDX) would alleviate that in practice? At least in theory the interruption suppression could save us there right?
Interruption suppression is useful, but many scenes still have a lot of interruptions.
Quoted 6 lines
Quoted 6 lines
If the device chooses a stupid
maximum value, it is his choice (spec should give more devices choices instead of forcing them to choose "0" which is not the best practice). We can't talk about performance for drivers when the devices tend to choose any "stupid" designs.
Quoted 6 lines
Quoted 2 lines
We need relaxe the restrictions and makes the spec more reasonable.
Hm, I see Linux virtio-net changes have landed with v6.0 and if I read those correctly the driver -- contrary to my initial expectation -- negotiates the feature, but does not set the parameters explicitly and
How does the driver know what parameters to set? The parameters should be exposed by each device.
Quoted 3 lines
thus keeps the defaults (until userspace decides to set the parameters). So it does matter whether the defaults are guaranteed to be 0 or not, and if not it does matter what defaults are chosen by the device.
Didn't follow this. More below.
Quoted 3 lines
One could even argue that those patches have been reviewed under the assumption that the device needs to use 0 as the default parameter value.
The default value should not be explicitly specified in the spec, because one size does not fit multiple devices. The source of this problem is that we are missing fields like default_{rx, tx}_coalesicng_params that indicate the device capabilities. No?
Thanks.
Quoted 8 lines
Well no strong opinions here. If the community is fine with it, I'm fine with it as well.
Thank you!
Regards,
Halil
Quoted 58 lines · Heng Qi
Quoted 50 lines · Halil Pasic
Quoted 8 lines · On Sat, 8 Jun 2024 10:34:22 +0800 Heng Qi <[email protected]> wrote:
Quoted 7 lines · Halil Pasic
Quoted 4 lines · On Tue, 28 May 2024 12:47:02 +0800 Heng Qi <[email protected]> wrote:
The device can set any initial coalescing parameters (0 or non-zero) for the receive/send queue before the setting command is executed, not just 0, enhancing device performance even without DIM enabled.
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
So we need to clarify descriptions that don't fit the behavior.
Sorry I'm late to the party -- again! Just for my understanding: how/why is this a clarification and not just a (basically incompatible) change?
In my opinion, "clarification" means that something may have been described incorrectly before, and we now need to discuss, explain clearly, and correct the possibly incorrect description.
I figure the difference in perceived semantics of the word "clarification" is at the root of my confusion. Let us have a look at
https://dictionary.cambridge.org/de/worterbuch/englisch/clarificationAccording to my understanding a "clarification", while an improvement in ease of understanding and/or decrease of ambiguity (possibly to no ambiguity at all) implies that what receiving a clarification is not outright wrong.
When rectifying something that is outright incorrect or wrong, I would refer to that with words like "correction", "fix", "erratum" or "corrigendum".
Quoted 9 lines
Quoted 6 lines
I mean if I read this correctly, before the driver had the guaranty that if the parameters are not set by the driver, negotiating the feature does not introduce any coalescing. After this in theory the device could just pick some max value and potentially introduce maximal latency in certain scenarios.
"maximum latency" also means "throughput improvement".
Under certain assumptions. But not necessarily. Again my concern is mostly the type of change. The virtio standard maintain a revision history appendix, and I would like to avoid the nature of this change being misrepresented there. If Connie and/or Michael think it is worth fixing, I believe it can be fixed with an editorial change.
Quoted 6 lines
AFAIU
VIRTIO_NET_F_NOTF_COALandVIRTIO_NET_F_VQ_NOTF_COALare about to land with virtio-1.3, i.e. there is no released/standardized virtio version where the "initialize to 0" is released. In that sense it looks like we are still on time to change this. But I am not 100% certain. In any case I don't think this as a huge impact and I'm fine going ahead with the change.
Quoted 8 lines
Sorry for the late reply, I'm on vacation.
I agree with this, and I prefer to release this patch as a correction for
virtio1.3 instead of a new patch for virtio1.4, because if devices support moderation coalescing based on virtio1.3, and after virtio1.4 is released, these devices need to be updated again for a
more reasonable coalescing parameters.Cornelia and Michael, what do you think?
The TC is just voting to start the public review process. We can defer that by a couple more weeks if there is a known issue to address.
For that I expect we want a final patch and a couple of acks on list from TC members by end of the ballot, June 13.
Quoted 64 lines
Quoted 12 lines
Quoted 12 lines
Quoted 8 lines
I understand that it is probably in the best interest of the devices to not pick stupid defaults. But it is also probably in the best interest of the driver to set those params, and if the driver is going to set its values, the devices defaults are moot unless we assume that those may end up being used by the driver as a hint when deciding which parameter values to choose.
"Any values" is compatible with "0 for max-usecs and max-frames", and the device can choose "no coalescing".
Quoted 7 lines
Quoted 2 lines
"No coalescing" means "latency friendly", but it also means "a lot of interruptions and throughput unfriendly".
To what extent does virtio's "normal" notification suppression (
VIRTIO_F_EVENT_IDX) would alleviate that in practice? At least in theory the interruption suppression could save us there right?Interruption suppression is useful, but many scenes still have a lot of interruptions.
Quoted 6 lines
Quoted 6 lines
If the device chooses a stupid
maximum value, it is his choice (spec should give more devices choices instead of forcing them to choose "0" which is not the best practice). We can't talk about performance for drivers when the devices tend to choose any "stupid" designs.Quoted 6 lines
Quoted 2 lines
We need relaxe the restrictions and makes the spec more reasonable.
Hm, I see Linux virtio-net changes have landed with v6.0 and if I read those correctly the driver -- contrary to my initial expectation -- negotiates the feature, but does not set the parameters explicitly and
How does the driver know what parameters to set? The parameters should be exposed by each device.
Quoted 3 lines
thus keeps the defaults (until userspace decides to set the parameters). So it does matter whether the defaults are guaranteed to be 0 or not, and if not it does matter what defaults are chosen by the device.
Didn't follow this. More below.
Quoted 3 lines
One could even argue that those patches have been reviewed under the assumption that the device needs to use 0 as the default parameter value.
The default value should not be explicitly specified in the spec, because one size does not fit multiple devices. The source of this problem is that we are missing fields like default_{rx, tx}_coalesicng_params that indicate the device capabilities. No?
Thanks.
Quoted 8 lines
Well no strong opinions here. If the community is fine with it, I'm fine with it as well.
Thank you!
Regards,
Halil
Quoted 70 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Monday, June 10, 2024 8:20 PMQuoted 1 line · Heng Qi
On Mon, 10 Jun 2024 14:46:02 +0200, Halil Pasic <[email protected]>
Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 1 line · On Sat, 8 Jun 2024 10:34:22 +0800 Heng Qi <[email protected]> wrote:
On Fri, 7 Jun 2024 22:02:46 +0200, Halil Pasic <[email protected]>
Quoted 6 lines · wrote:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines · On Tue, 28 May 2024 12:47:02 +0800 Heng Qi <[email protected]> wrote:
The device can set any initial coalescing parameters (0 or non-zero) for the receive/send queue before the setting command is executed, not just 0, enhancing device performance even
without DIM enabled.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
So we need to clarify descriptions that don't fit the behavior.
Sorry I'm late to the party -- again! Just for my understanding:
how/why is this a clarification and not just a (basically incompatible)change?
Quoted 31 lines
Quoted 31 lines
Quoted 5 lines
In my opinion, "clarification" means that something may have been described incorrectly before, and we now need to discuss, explain clearly, and correct the possibly incorrect description.
I figure the difference in perceived semantics of the word "clarification" is at the root of my confusion. Let us have a look at
https://dictionary.cambridge.org/de/worterbuch/englisch/clarificatio
nAccording to my understanding a "clarification", while an improvement in ease of understanding and/or decrease of ambiguity (possibly to no ambiguity at all) implies that what receiving a clarification is not outright wrong.
When rectifying something that is outright incorrect or wrong, I would refer to that with words like "correction", "fix", "erratum" or "corrigendum".
Quoted 9 lines
Quoted 6 lines
I mean if I read this correctly, before the driver had the guaranty that if the parameters are not set by the driver, negotiating the feature does not introduce any coalescing. After this in theory the device could just pick some max value and potentially introduce maximal latency in certain scenarios.
"maximum latency" also means "throughput improvement".
Quoted 7 lines
Quoted 7 lines
Under certain assumptions. But not necessarily. Again my concern is mostly the type of change. The virtio standard maintain a revision history appendix, and I would like to avoid the nature of this change being misrepresented there. If Connie and/or Michael think it is worth fixing, I believe it can be fixed with an editorial change.
AFAIU
VIRTIO_NET_F_NOTF_COALandVIRTIO_NET_F_VQ_NOTF_COALare about
Quoted 6 lines
Quoted 5 lines
to land with virtio-1.3, i.e. there is no released/standardized virtio version where the "initialize to 0" is released. In that sense it looks like we are still on time to change this. But I am not 100% certain. In any case I don't think this as a huge impact and I'm fine going ahead with the change.
Quoted 8 lines
Quoted 7 lines
Sorry for the late reply, I'm on vacation.
I agree with this, and I prefer to release this patch as a correction for
virtio1.3 instead of a new patch for virtio1.4, because if devices support moderation coalescing based on virtio1.3, and after virtio1.4 is released, these devices need to be updated again for a more reasonablecoalescing parameters.
Heng's patch is bringing only the relaxation to the device that it can have nonzero value on reset.
I don't have strong opinion either way.
Either approach looks fine to me as it does not seem to break any functionality for driver and device. It only improves the performance when driver has not set them.
'Clarify' to 'correct' or 'fix' subject line would have been obviously better as Pasic Halil mentioned, but probably while merging the patch, it can be corrected.
Quoted 78 lines
Quoted 2 lines
Cornelia and Michael, what do you think?
The TC is just voting to start the public review process.
We can defer that by a couple more weeks if there is a known issue to address.For that I expect we want a final patch and a couple of acks on list from TC members by end of the ballot, June 13.
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 8 lines
I understand that it is probably in the best interest of the devices to not pick stupid defaults. But it is also probably in the best interest of the driver to set those params, and if the driver is going to set its values, the devices defaults are moot unless we assume that those may end up being used by the driver as a hint when deciding which parameter values to choose.
"Any values" is compatible with "0 for max-usecs and max-frames", and the device can choose "no coalescing".
"No coalescing" means "latency friendly", but it also means "a lot of interruptions and throughput unfriendly".
Quoted 13 lines
Quoted 4 lines
To what extent does virtio's "normal" notification suppression (
VIRTIO_F_EVENT_IDX) would alleviate that in practice? At least in theory the interruption suppression could save us there right?Interruption suppression is useful, but many scenes still have a lot of interruptions.
Quoted 5 lines
Quoted 5 lines
If the device chooses a stupid
maximum value, it is his choice (spec should give more devices choices instead of forcing them to choose "0" which is not the best practice). We can't talk about performance for drivers when thedevices tend to choose any "stupid" designs.
Quoted 37 lines
Quoted 8 lines
Quoted 3 lines
We need relaxe the restrictions and makes the spec more reasonable.
Hm, I see Linux virtio-net changes have landed with v6.0 and if I read those correctly the driver -- contrary to my initial expectation -- negotiates the feature, but does not set the parameters explicitly and
How does the driver know what parameters to set? The parameters should be exposed by each device.
Quoted 3 lines
thus keeps the defaults (until userspace decides to set the parameters). So it does matter whether the defaults are guaranteed to be 0 or not, and if not it does matter what defaults are chosen by the device.
Didn't follow this. More below.
Quoted 3 lines
One could even argue that those patches have been reviewed under the assumption that the device needs to use 0 as the default parameter value.
The default value should not be explicitly specified in the spec, because one size does not fit multiple devices. The source of this problem is that we are missing fields like default_{rx, tx}_coalesicng_params that indicate the device capabilities. No?
Thanks.
Quoted 8 lines
Well no strong opinions here. If the community is fine with it, I'm fine with it as well.
Thank you!
Regards,
Halil
Quoted 66 lines · Michael S. Tsirkin
Quoted 65 lines · Heng Qi
Quoted 50 lines · Halil Pasic
Quoted 8 lines · On Sat, 8 Jun 2024 10:34:22 +0800 Heng Qi <[email protected]> wrote:
Quoted 7 lines · Halil Pasic
Quoted 4 lines · On Tue, 28 May 2024 12:47:02 +0800 Heng Qi <[email protected]> wrote:
The device can set any initial coalescing parameters (0 or non-zero) for the receive/send queue before the setting command is executed, not just 0, enhancing device performance even without DIM enabled.
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
So we need to clarify descriptions that don't fit the behavior.
Sorry I'm late to the party -- again! Just for my understanding: how/why is this a clarification and not just a (basically incompatible) change?
In my opinion, "clarification" means that something may have been described incorrectly before, and we now need to discuss, explain clearly, and correct the possibly incorrect description.
I figure the difference in perceived semantics of the word "clarification" is at the root of my confusion. Let us have a look at
https://dictionary.cambridge.org/de/worterbuch/englisch/clarificationAccording to my understanding a "clarification", while an improvement in ease of understanding and/or decrease of ambiguity (possibly to no ambiguity at all) implies that what receiving a clarification is not outright wrong.
When rectifying something that is outright incorrect or wrong, I would refer to that with words like "correction", "fix", "erratum" or "corrigendum".
Quoted 9 lines
Quoted 6 lines
I mean if I read this correctly, before the driver had the guaranty that if the parameters are not set by the driver, negotiating the feature does not introduce any coalescing. After this in theory the device could just pick some max value and potentially introduce maximal latency in certain scenarios.
"maximum latency" also means "throughput improvement".
Under certain assumptions. But not necessarily. Again my concern is mostly the type of change. The virtio standard maintain a revision history appendix, and I would like to avoid the nature of this change being misrepresented there. If Connie and/or Michael think it is worth fixing, I believe it can be fixed with an editorial change.
Quoted 6 lines
AFAIU
VIRTIO_NET_F_NOTF_COALandVIRTIO_NET_F_VQ_NOTF_COALare about to land with virtio-1.3, i.e. there is no released/standardized virtio version where the "initialize to 0" is released. In that sense it looks like we are still on time to change this. But I am not 100% certain. In any case I don't think this as a huge impact and I'm fine going ahead with the change.Sorry for the late reply, I'm on vacation.
I agree with this, and I prefer to release this patch as a correction for
virtio1.3 instead of a new patch for virtio1.4, because if devices support moderation coalescing based on virtio1.3, and after virtio1.4 is released, these devices need to be updated again for a
more reasonable coalescing parameters.
Quoted 8 lines
Quoted 1 line
Cornelia and Michael, what do you think?
The TC is just voting to start the public review process. We can defer that by a couple more weeks if there is a known issue to address.
For that I expect we want a final patch and a couple of acks on list from TC members by end of the ballot, June 13.
FTR, I'd be fine with that if there's a consensus, but I'll not be around to handle this personally. I also have no objection to proceeding with the process.
Quoted 1 line · On Mon, 10 Jun 2024 21:35:45 +0800 Heng Qi <[email protected]> wrote:
On Mon, 10 Jun 2024 14:46:02 +0200, Halil Pasic <[email protected]> wrote:
[..]
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
If the device chooses a stupid
maximum value, it is his choice (spec should give more devices choices instead of forcing them to choose "0" which is not the best practice). We can't talk about performance for drivers when the devices tend to choose any "stupid" designs.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
We need relaxe the restrictions and makes the spec more reasonable.
Hm, I see Linux virtio-net changes have landed with v6.0 and if I read those correctly the driver -- contrary to my initial expectation -- negotiates the feature, but does not set the parameters explicitly and
How does the driver know what parameters to set?
Disclaimer: I'm not very knowledgeable when it comes to networking and NICs. Please be patient with me.
If I understood that properly have previously stated that it is basically a trade-off between "latency friendly" (downside: overhead and not so throughput friendly) and "throughput friendly" (downside: not so latency friendly). And that makes sense to me.
So I would think, the answer to the question what is the best trade-off should also depend on the workload.
Now as far as I understand although we call the parameters max_usecs and max_packets the notification condition is dictated by those two values. I.e. there won't be a notification unless the compound condition is met.
When there is no traffic 0 and 0 looks like reasonable values to me: I want the first of possibly many packets ASAP. Depending on the actual load, maybe one could employ some sort of a heuristics to keep good balance -- maybe based on a frequency of interrupts. Maybe DIM is actually exactly what I have in mind.
You seem more knowledgeable on the topic. How is this usually done? How are the optimal values correlated with device characteristics?
Quoted 2 lines
The parameters should be
exposed by each device.
I would like to better understand why. My intention is not to question the correctness of this statement, but to gain a better understanding.
Quoted 7 lines
Quoted 4 lines
thus keeps the defaults (until userspace decides to set the parameters). So it does matter whether the defaults are guaranteed to be 0 or not, and if not it does matter what defaults are chosen by the device.
Didn't follow this. More below.
Maybe my statement was wrong. So let me make a question out of it. What entity or entities do we expect to change the parameter values, and when or under which conditions do we expect them to change the parameter values?
Quoted 6 lines
Quoted 3 lines
One could even argue that those patches have been reviewed under the assumption that the device needs to use 0 as the default parameter value.
The default value should not be explicitly specified in the spec, because one size does not fit multiple devices.
This ties in to my previous question about the relationship between device characteristics and the optimal values for max_usecs and max_packets.
Quoted 3 lines
The source of this problem is that we are
missing fields like default_{rx, tx}_coalesicng_params that indicate the device capabilities. No?
I don't think so! In my understanding is that with your proposal after a reset, the OS/driver has no problem obtaining the defaults that indicate the device capabilities. But I would really like to better understand that device capabilities part.
And yes another way around this would have been say:
* let us introduce those defaults fields, e.g. to the config space
* make the driver read those values as a part of the initialization
* and set those values as the initial parameters.
But frankly I see no benefit of that over what you propose here.
Regards,
Halil
Quoted 20 lines · Halil Pasic
Quoted 1 line · On Mon, 10 Jun 2024 21:35:45 +0800 Heng Qi <[email protected]> wrote:
On Mon, 10 Jun 2024 14:46:02 +0200, Halil Pasic <[email protected]> wrote:
[..]
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
If the device chooses a stupid
maximum value, it is his choice (spec should give more devices choices instead of forcing them to choose "0" which is not the best practice). We can't talk about performance for drivers when the devices tend to choose any "stupid" designs.Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
We need relaxe the restrictions and makes the spec more reasonable.
Hm, I see Linux virtio-net changes have landed with v6.0 and if I read those correctly the driver -- contrary to my initial expectation -- negotiates the feature, but does not set the parameters explicitly and
How does the driver know what parameters to set?
Quoted 8 lines
Disclaimer: I'm not very knowledgeable when it comes to networking and NICs. Please be patient with me.
If I understood that properly have previously stated that it is basically a trade-off between "latency friendly" (downside: overhead and not so throughput friendly) and "throughput friendly" (downside: not so latency friendly). And that makes sense to me.
Yes, I am correcting the description about the trade-off thing.
Quoted 2 lines
So I would think, the answer to the question what is the best trade-off should also depend on the workload.
device-side dim or driver-side dim to solve this problem. But, when the device is reset, the driver may not enable dim, the device or driver needs to have a static coalescing parameters (0 or non-zero) for the trade-off.
(don't trust dim too much now, I'm doing some updates to optimize dim.)
Quoted 4 lines
Now as far as I understand although we call the parameters
max_usecsandmax_packetsthe notification condition is dictated by those two values. I.e. there won't be a notification unless the compound condition is met.
Right, just one of the conditions needs to be met.
Quoted 6 lines
When there is no traffic 0 and 0 looks like reasonable values to me: I want the first of possibly many packets ASAP. Depending on the actual load, maybe one could employ some sort of a heuristics to keep good balance -- maybe based on a frequency of interrupts. Maybe DIM is actually exactly what I have in mind.
This patch is not to solve the scenario where dim exists, but what the static value of the coalescing parameter is when the device is reset. (please check some hhistorical discussion of this topic)
Quoted 3 lines
You seem more knowledgeable on the topic. How is this usually done? How are the optimal values correlated with device characteristics?
NICs such as mlx, ena, ice, and HiSilicon all have a non-zero static coalescing value. Although they all support netdim, the static value is still useful when dim is disabled.
As for the specific value, I think it is best to notify the driver through the device itself, such as adding capability fileds or this:
https://lore.kernel.org/all/[email protected]/
Quoted 12 lines
Quoted 3 lines
The parameters should be
exposed by each device.I would like to better understand why. My intention is not to question the correctness of this statement, but to gain a better understanding.
Quoted 5 lines
Quoted 4 lines
thus keeps the defaults (until userspace decides to set the parameters). So it does matter whether the defaults are guaranteed to be 0 or not, and if not it does matter what defaults are chosen by the device.
Quoted 7 lines
Quoted 2 lines
Didn't follow this. More below.
Maybe my statement was wrong. So let me make a question out of it. What entity or entities do we expect to change the parameter values, and when or under which conditions do we expect them to change the parameter values?
Changed by the driver when the load changes or user..
Quoted 11 lines
Quoted 7 lines
Quoted 4 lines
One could even argue that those patches have been reviewed under the assumption that the device needs to use 0 as the default parameter value.
The default value should not be explicitly specified in the spec, because one size does not fit multiple devices.
This ties in to my previous question about the relationship between device characteristics and the optimal values for
max_usecsandmax_packets.
Device characteristics and optimal max_usecs/frames are preferably communicated via device capability, but at least should not be forced to a specific value by the driver.
Quoted 5 lines
Quoted 4 lines
The source of this problem is that we are
missing fields like default_{rx, tx}_coalesicng_params that indicate the device capabilities. No?
Quoted 7 lines
I don't think so! In my understanding is that with your proposal after a reset, the OS/driver has no problem obtaining the defaults that indicate the device capabilities. But I would really like to better understand that device capabilities part.
And yes another way around this would have been say:
* let us introduce those defaults fields, e.g. to the config space
Do you want to solve it by adding another new feature? Otherwise, how do you solve the compatibility problem?
Quoted 1 line
* make the driver read those values as a part of the initialization
The spec is in conflict with this, because the driver can only read 0 from devices that conform to the spec. I submitted this patch to achieve this goal:
https://lore.kernel.org/all/[email protected]/
Quoted 1 line
* and set those values as the initial parameters.
I don't support this, the device doesn't want to accept any initial unreasonable settings.
Thanks.
Quoted 4 lines
But frankly I see no benefit of that over what you propose here.
Regards,
Halil
Quoted 24 lines · Heng Qi
Quoted 23 lines · Halil Pasic
Quoted 1 line · On Mon, 10 Jun 2024 21:35:45 +0800 Heng Qi <[email protected]> wrote:
On Mon, 10 Jun 2024 14:46:02 +0200, Halil Pasic <[email protected]> wrote:
[..]
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
If the device chooses a stupid
maximum value, it is his choice (spec should give more devices choices instead of forcing them to choose "0" which is not the best practice). We can't talk about performance for drivers when the devices tend to choose any "stupid" designs.Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
We need relaxe the restrictions and makes the spec more reasonable.
Hm, I see Linux virtio-net changes have landed with v6.0 and if I read those correctly the driver -- contrary to my initial expectation -- negotiates the feature, but does not set the parameters explicitly and
How does the driver know what parameters to set?
Disclaimer: I'm not very knowledgeable when it comes to networking and NICs. Please be patient with me.
Quoted 7 lines
Quoted 5 lines
If I understood that properly have previously stated that it is basically a trade-off between "latency friendly" (downside: overhead and not so throughput friendly) and "throughput friendly" (downside: not so latency friendly). And that makes sense to me.
Yes, I am correcting the description about the trade-off thing.
It does not look like this thing is ready yet.
How we can we make progress with
the realease but sure we don't make backwards compat a pain?
Ideas?
Quoted 112 lines
Quoted 2 lines
So I would think, the answer to the question what is the best trade-off should also depend on the workload.
device-side dim or driver-side dim to solve this problem. But, when the device is reset, the driver may not enable dim, the device or driver needs to have a static coalescing parameters (0 or non-zero) for the trade-off.
(don't trust dim too much now, I'm doing some updates to optimize dim.)
Quoted 4 lines
Now as far as I understand although we call the parameters
max_usecsandmax_packetsthe notification condition is dictated by those two values. I.e. there won't be a notification unless the compound condition is met.Right, just one of the conditions needs to be met.
Quoted 6 lines
When there is no traffic 0 and 0 looks like reasonable values to me: I want the first of possibly many packets ASAP. Depending on the actual load, maybe one could employ some sort of a heuristics to keep good balance -- maybe based on a frequency of interrupts. Maybe DIM is actually exactly what I have in mind.
This patch is not to solve the scenario where dim exists, but what the static value of the coalescing parameter is when the device is reset. (please check some hhistorical discussion of this topic)
Quoted 3 lines
You seem more knowledgeable on the topic. How is this usually done? How are the optimal values correlated with device characteristics?
NICs such as mlx, ena, ice, and HiSilicon all have a non-zero static coalescing value. Although they all support netdim, the static value is still useful when dim is disabled.
As for the specific value, I think it is best to notify the driver through the device itself, such as adding capability fileds or this:
https://lore.kernel.org/all/[email protected]/Quoted 12 lines
Quoted 3 lines
The parameters should be
exposed by each device.I would like to better understand why. My intention is not to question the correctness of this statement, but to gain a better understanding.
Quoted 5 lines
Quoted 4 lines
thus keeps the defaults (until userspace decides to set the parameters). So it does matter whether the defaults are guaranteed to be 0 or not, and if not it does matter what defaults are chosen by the device.
Quoted 7 lines
Quoted 2 lines
Didn't follow this. More below.
Maybe my statement was wrong. So let me make a question out of it. What entity or entities do we expect to change the parameter values, and when or under which conditions do we expect them to change the parameter values?
Changed by the driver when the load changes or user..
Quoted 11 lines
Quoted 7 lines
Quoted 4 lines
One could even argue that those patches have been reviewed under the assumption that the device needs to use 0 as the default parameter value.
The default value should not be explicitly specified in the spec, because one size does not fit multiple devices.
This ties in to my previous question about the relationship between device characteristics and the optimal values for
max_usecsandmax_packets.Device characteristics and optimal max_usecs/frames are preferably communicated via device capability, but at least should not be forced to a specific value by the driver.
Quoted 5 lines
Quoted 4 lines
The source of this problem is that we are
missing fields like default_{rx, tx}_coalesicng_params that indicate the device capabilities. No?Quoted 7 lines
I don't think so! In my understanding is that with your proposal after a reset, the OS/driver has no problem obtaining the defaults that indicate the device capabilities. But I would really like to better understand that device capabilities part.
And yes another way around this would have been say:
* let us introduce those defaults fields, e.g. to the config spaceDo you want to solve it by adding another new feature? Otherwise, how do you solve the compatibility problem?
Quoted 1 line
* make the driver read those values as a part of the initialization
The spec is in conflict with this, because the driver can only read 0 from devices that conform to the spec. I submitted this patch to achieve this goal:
https://lore.kernel.org/all/[email protected]/Quoted 1 line
* and set those values as the initial parameters.
I don't support this, the device doesn't want to accept any initial unreasonable settings.
Thanks.
Quoted 4 lines
But frankly I see no benefit of that over what you propose here.
Regards,
Halil
Quoted 11 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 11, 2024 10:00 PMHow we can we make progress with
the realease but sure we don't make backwards compat a pain?Ideas?
There is no functional break with this relaxation.
Device set some non-zero defaults and driver didn't modify them.... Anything broken? Unlikely.
And device/driver has better performance, is that a problem? Unlikely.
Quoted 9 lines · Parav Pandit
Quoted 9 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 11, 2024 10:00 PMHow we can we make progress with
the realease but sure we don't make backwards compat a pain?
Quoted 7 lines
Quoted 2 lines
Ideas?
There is no functional break with this relaxation.
Device set some non-zero defaults and driver didn't modify them.... Anything broken? Unlikely.
And device/driver has better performance, is that a problem? Unlikely.
Yes, it is possible. Driver can cache values it sets and never query device with get.
Before anything is set, driver will report incorrect values. What will break as a result? Hard to predict.
Given the patch is big, I am inclined to say it just should use a feature bit.
Signature · 1 line
MST
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Parav Pandit
Quoted 9 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 11, 2024 10:00 PMHow we can we make progress with
the realease but sure we don't make backwards compat a pain?
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Ideas?
There is no functional break with this relaxation.
Device set some non-zero defaults and driver didn't modify them.... Anything broken? Unlikely.
And device/driver has better performance, is that a problem? Unlikely.
Sorry, my vacation just ended.
Quoted 3 lines
Yes, it is possible. Driver can cache values it sets and never query device with get.
Don't we already have a lot of behaviors to drive queries from devices? RSS context, device stats.
Quoted 1 line
Before anything is set, driver will report incorrect values.
Devices that are widely supported and supported by good practices should have any initialization value. Just reporting 0 is incorrect value. Although the spec now says so.
Quoted 6 lines
What will break as a result? Hard to predict.
Given the patch is big, I am inclined to say it just should use a feature bit.
This doesn't seem to break anything in my opinion, we just told the device that now you can set more values.
Using new feature bits does not seem necessary.
Thanks.
Quoted 3 lines
Signature · 1 line
MST
Quoted 6 lines · Heng Qi
Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines · Parav Pandit
Quoted 3 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 11, 2024 10:00 PM
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
How we can we make progress with
the realease but sure we don't make backwards compat a pain?Ideas?
There is no functional break with this relaxation. Device set some non-zero defaults and driver didn't modify them.... Anything broken? Unlikely.
Generally it's inappropriate to leave this decision making to the device for what would be the best / most performant default config, as the device is generally considered agnostic to the guest load... Unless the device is specially hard wired to some fixed guest setup that users couldn't change, it doesn't seem logical that the device could derive the best or most performant config on driver's behalf. What if the guest wants best latency for its load but the device just blindlessly guess the guest might prefer throughput friendly that it miserably uses latency impacting non-zero default? Could this device side change for the default config regress boot time performance (which may need best latency over throughput)?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
And device/driver has better performance, is that a problem? Unlikely.
Even for rare case with a hard wired setup, the way to tackle the very problem using device's default is still quite questionable. Usually the mgmt software or network config utility should be equipped with some default value if need be. And we know the guest has the best position to impose the best / most performant config for its own load. What is the issue or use case that this initial config couldn't be applied by the guest mgmt software ahead but has to resort to the device to load some default (which is odd and irrelevant to any guest load), before the interface is brought up for operation i.e. performing I/O?
Quoted 11 lines
Sorry, my vacation just ended.
Quoted 2 lines
Yes, it is possible. Driver can cache values it sets and never query device with get.
Don't we already have a lot of behaviors to drive queries from devices? RSS context, device stats.
Quoted 1 line
Before anything is set, driver will report incorrect values.
Devices that are widely supported and supported by good practices should have any initialization value. Just reporting 0 is incorrect value. Although the spec now says so.
I don't have an aligned view here, sorry. As I recall having 0 as the default is just to keep device started in a state where coalescing is disabled, so it's backward compatible and consistent with a non-coalescing supporting driver - such that it won't yield surprising effect (for e.g. regressed latency) inadvertently after user's getting driver software upgraded. Unlike the other virtio-net features that could 100% improve performance in all aspect, this coalescing feature is more of a performance tuning knob that may improve performance metrics (such as cpu usage or throughput) of one dimension while demoting the others (such as latency, jitter or connection rate) from the equation. That said, there's not a single and fixed set of default config that device could supply which is able to satisfy all kind of guest load. Rather than rely on the device to offer a matching default for driver (which I think it's technically wrong), I'd lean to having guest software or network utility to apply the initial config for the guest, where they should have best knowledge for the specific guest workload than what device could do.
Quoted 9 lines
Quoted 7 lines
What will break as a result? Hard to predict.
Given the patch is big, I am inclined to say it just should use a feature bit.
This doesn't seem to break anything in my opinion, we just told the device that now you can set more values.
Another thing this could break is live migration between devices with varied default. How do you make sure the guest doesn't rely on some default from the source device, while on the destination it just doesn't get the same default coalescing value? To get rid of this side effect the guest would still need to apply the initial config for its own, anyway... Which eventually would render this proposal with arbitrary default rather pointless.
Thanks,
-Siwei
Quoted 8 lines
Using new feature bits does not seem necessary.
Thanks.
Quoted 3 lines
Signature · 1 line
MST
Quoted 12 lines · Si-Wei Liu
Quoted 9 lines · Heng Qi
Quoted 8 lines · Michael S. Tsirkin
Quoted 7 lines · Parav Pandit
Quoted 7 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 11, 2024 10:00 PMHow we can we make progress with
the realease but sure we don't make backwards compat a pain?
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Ideas?
There is no functional break with this relaxation. Device set some non-zero defaults and driver didn't modify them.... Anything broken? Unlikely.
Generally it's inappropriate to leave this decision making to the device for what would be the best / most performant default config, as the device is generally considered agnostic to the guest load...
Instead, the performance of the virtual machine and the driver depends heavily on how the device is implemented, just as we have proposed various ways to offload the data queue in the device to the hardware. The reason why most devices use software to simulate ctrlq instead of using hardware offload is that the driver has no requirements for the performance of ctrlq before, that is, the device implementation is responsible for and meets the driver's performance requirements.
Quoted 7 lines
Unless the
device is specially hard wired to some fixed guest setup that users couldn't change, it doesn't seem logical that the device could derive the best or most performant config on driver's behalf. What if the guest wants best latency for its load but the device just blindlessly guess the guest might prefer throughput friendly that it miserably uses latency impacting non-zero default?
The device does not want to guess and cannot guess. This patch does not force the device to choose a non-zero value, but relaxes it to allow the device to choose 0 or non-zero, which is very friendly to virtual machines with different performance requirements, right?
Quoted 3 lines
Could this device side change for
the default config regress boot time performance (which may need best latency over throughput)?
Don't make these assumptions, what if the driver needs better throughput?
Quoted 12 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
And device/driver has better performance, is that a problem? Unlikely.
Even for rare case with a hard wired setup, the way to tackle the very problem using device's default is still quite questionable. Usually the mgmt software or network config utility should be equipped with some default value if need be. And we know the guest has the best position to impose the best / most performant config for its own load. What is the issue or use case that this initial config couldn't be applied by the guest mgmt software ahead but has to resort to the device to load some default (which is odd and irrelevant to any guest load), before the interface is brought up for operation i.e. performing I/O?
Use cases are everywhere, Alibaba Cloud, MLX and all other modern network cards have a default value that is not 0.
(0 is actually a kind of default value)
Quoted 21 lines
Quoted 5 lines
Sorry, my vacation just ended.
Quoted 2 lines
Yes, it is possible. Driver can cache values it sets and never query device with get.
Quoted 7 lines
Don't we already have a lot of behaviors to drive queries from devices? RSS context, device stats.
Quoted 1 line
Before anything is set, driver will report incorrect values.
Devices that are widely supported and supported by good practices should have any initialization value. Just reporting 0 is incorrect value. Although the spec now says so.
I don't have an aligned view here, sorry. As I recall having 0 as the default is just to keep device started in a state where coalescing is disabled, so it's backward compatible and consistent with a non-coalescing supporting driver - such that it won't yield surprising effect (for e.g. regressed latency) inadvertently after user's getting driver software upgraded. Unlike the other virtio-net features that could 100% improve performance in all aspect, this coalescing feature is more of a performance tuning knob that may improve performance metrics (such as cpu usage or throughput) of one dimension while demoting the
Quoted 8 lines
others (such as latency, jitter or connection rate) from the equation. That said, there's not a single and fixed set of default config that device could supply which is able to satisfy all kind of guest load. Rather than rely on the device to offer a matching default for driver (which I think it's technically wrong), I'd lean to having guest software or network utility to apply the initial config for the guest, where they should have best knowledge for the specific guest workload than what device could do.
Before this feature, a good device implementation should also support coalescing (of course we don't necessarily assume it has coalescing). In addition, virtual machines that tend to favor latency and throughput exist. If the device supported by the manufacturer needs to provide a low-latency virtual machine, please continue to keep the default value of 0.
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
What will break as a result? Hard to predict.
Given the patch is big, I am inclined to say it just should use a feature bit.
This doesn't seem to break anything in my opinion, we just told the device that now you can set more values.
Quoted 7 lines
Another thing this could break is live migration between devices with varied default. How do you make sure the guest doesn't rely on some default from the source device, while on the destination it just doesn't get the same default coalescingjjj value? To get rid of this side effect the guest would still need to apply the initial config for its own, anyway... Which eventually would render this proposal with arbitrary default rather pointless.
I don't quite understand why this would affect hot migration, the values would be migrated over.
Thanks.
Quoted 13 lines
Thanks,
-SiweiQuoted 8 lines
Using new feature bits does not seem necessary.
Thanks.
Quoted 3 lines
Signature · 1 line
MST
Quoted 18 lines · Heng Qi
Quoted 9 lines · Si-Wei Liu
Quoted 7 lines · Heng Qi
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Parav Pandit
Quoted 5 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 11, 2024 10:00 PM
How we can we make progress with
the realease but sure we don't make backwards compat a pain?Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Ideas?
There is no functional break with this relaxation. Device set some non-zero defaults and driver didn't modify them.... Anything broken? Unlikely.
Generally it's inappropriate to leave this decision making to the device for what would be the best / most performant default config, as the device is generally considered agnostic to the guest load...
Quoted 7 lines
Instead, the performance of the virtual machine and the driver depends heavily on how the device is implemented, just as we have proposed various ways to offload the data queue in the device to the hardware. The reason why most devices use software to simulate ctrlq instead of using hardware offload is that the driver has no requirements for the performance of ctrlq before, that is, the device implementation is responsible for and meets the driver's performance requirements.
I am not sure I follow the arguments of ctrlq being s/w or h/w, I thought that it was for the debate why we need coalescing on hardware offload device in the first place, instead of reusing event index or similar s/w based notification suppression. I don't doubt the value of coalescing, but how's it relevant to the default value disposition? Generally the default config disposition is not considered to be part of device implementation, especially when it comes to the situation where device can't easily figure out the specific workload to occur in the guest, and there's no perfect single default value that could meet every single performance metric across the board. This is a typical tuning knob left up to the user to adjust, why the device or driver needs to set or load the initial value? The driver just needs to start with certain value, be it 0 or non-zero, which guest user can override at any point of time, depending on his/her need, and that's it! I guess I still don't understand your user case here, why device / driver default is of such importance.
Quoted 11 lines
Quoted 7 lines
Unless the
device is specially hard wired to some fixed guest setup that users couldn't change, it doesn't seem logical that the device could derive the best or most performant config on driver's behalf. What if the guest wants best latency for its load but the device just blindlessly guess the guest might prefer throughput friendly that it miserably uses latency impacting non-zero default?The device does not want to guess and cannot guess. This patch does not force the device to choose a non-zero value, but relaxes it to allow the device to choose 0 or non-zero, which is very friendly to virtual machines with different performance requirements, right?
I don't understand the friendly part - do you imply your VM users are on kinda fixed wired setup that they cannot change these coalescing parameters after driver is loaded? Can the owner of the VM in control apply certain initial config for the coalescing parameters to the VM image? Or is it the problem of the guest driver that doesn't yet expose coalescing parameters to the end user? Otherwise I would think that guest user should be able to set parameters accordingly that would best fit the specific performance requirement of their own. How the device could even help here? I don't feel there's a lot of value to grant the device or host admin the flexibility to policy the *best* config on guest user's behalf, to be honest. And you seem to admit the fact that the default doesn't really matter, be it 0 or non-zero.
Quoted 5 lines
Quoted 4 lines
Could this device side change for
the default config regress boot time performance (which may need best latency over throughput)?Don't make these assumptions, what if the driver needs better throughput?
There's a misconception here: what we think that driver may need in terms on performance does actually reflect what guest user would like to have. Driver cannot read guest user's mind to make the decision, either.
In history there was drastic change in the Linux virtio-net driver that ever changed the default disposition for XPS (and RPS as well) from throughput and long-lived connection oriented to concurrency and short-lived oriented, which regressed a lot of existing setups that expects sustained throughput and packet rate after kernel upgrade. Although the occurrence of such drastic change for default disposition is not so welcome (that is one of the reasons why I valued consistent initial value and back compatible behavior), I don't see people yelling at virtio spec for less flexibility of offering the default disposition, given that the guest user can override the config any time with their own tooling or script, there's no problem at all for them to just set the corresponding config back explicitly to what it was before.
Quoted 12 lines
Quoted 12 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
And device/driver has better performance, is that a problem? Unlikely.
Even for rare case with a hard wired setup, the way to tackle the very problem using device's default is still quite questionable. Usually the mgmt software or network config utility should be equipped with some default value if need be. And we know the guest has the best position to impose the best / most performant config for its own load. What is the issue or use case that this initial config couldn't be applied by the guest mgmt software ahead but has to resort to the device to load some default (which is odd and irrelevant to any guest load), before the interface is brought up for operation i.e. performing I/O?
Quoted 2 lines
Use cases are everywhere, Alibaba Cloud, MLX and all other modern network cards have a default value that is not 0.
You seems to be referencing your own setup basically, and the question is still left answered - why the initial config can't be done through the mgmt software or network config utility within the guest?
Quoted 30 lines
(0 is actually a kind of default value)
Quoted 20 lines
Quoted 11 lines
Sorry, my vacation just ended.
Quoted 2 lines
Yes, it is possible. Driver can cache values it sets and never query device with get.
Don't we already have a lot of behaviors to drive queries from devices? RSS context, device stats.
Quoted 1 line
Before anything is set, driver will report incorrect values.
Devices that are widely supported and supported by good practices should have any initialization value. Just reporting 0 is incorrect value. Although the spec now says so.
I don't have an aligned view here, sorry. As I recall having 0 as the default is just to keep device started in a state where coalescing is disabled, so it's backward compatible and consistent with a non-coalescing supporting driver - such that it won't yield surprising effect (for e.g. regressed latency) inadvertently after user's getting driver software upgraded. Unlike the other virtio-net features that could 100% improve performance in all aspect, this coalescing feature is more of a performance tuning knob that may improve performance metrics (such as cpu usage or throughput) of one dimension while demoting the
Quoted 8 lines
others (such as latency, jitter or connection rate) from the equation. That said, there's not a single and fixed set of default config that device could supply which is able to satisfy all kind of guest load. Rather than rely on the device to offer a matching default for driver (which I think it's technically wrong), I'd lean to having guest software or network utility to apply the initial config for the guest, where they should have best knowledge for the specific guest workload than what device could do.
Quoted 2 lines
Before this feature, a good device implementation should also support coalescing (of course we don't necessarily assume it has coalescing).
Again, I don't doubt the value of supporting coalescing.
Quoted 4 lines
In addition, virtual
machines that tend to favor latency and throughput exist. If the device supported by the manufacturer needs to provide a low-latency virtual machine, please continue to keep the default value of 0.
No, that's not what I was asking. There's no such requirement for any vendor to provide a low-latency or high throughput VM. The more general use case is - the setup for real world workload might just be too complex that end users would prefer low-latency on some virtual NIC or even some specific queues, while the other queues of a virtual NIC, or reset virtual NIC might have very different dispositions. Due to the needs and dynamics of workload scaling up & down, they might have more or less queues or virtual NICs to configure, so these disposition would need to be readjusted at any point of time, for which there's no easy way for device to adapt to easily. The guest user should have best knowledge for the specific guest workload and setup than what device could/should offer.
Quoted 15 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
What will break as a result? Hard to predict.
Given the patch is big, I am inclined to say it just should use a feature bit.
This doesn't seem to break anything in my opinion, we just told the device that now you can set more values.
Quoted 7 lines
Another thing this could break is live migration between devices with varied default. How do you make sure the guest doesn't rely on some default from the source device, while on the destination it just doesn't get the same default coalescingjjj value? To get rid of this side effect the guest would still need to apply the initial config for its own, anyway... Which eventually would render this proposal with arbitrary default rather pointless.
Quoted 2 lines
I don't quite understand why this would affect hot migration, the values would be migrated over.
Then I don't see this has been discussed (wouldn't the initial value be part of virtio device state to migrate over?) in the thread or described in the proposed text. And your proposed per-queue coalescing parameters (default plus current value) would have to be described too as part of virtio-net device state, so that people don't misunderstand your proposal.
Thanks,
-Siwei
Quoted 12 lines
Thanks.
Quoted 10 lines
Thanks,
-SiweiQuoted 7 lines
Using new feature bits does not seem necessary.
Thanks.
Quoted 3 lines
Signature · 1 line
MST
Quoted 28 lines · Si-Wei Liu
Quoted 18 lines · Heng Qi
Quoted 9 lines · Si-Wei Liu
Quoted 7 lines · Heng Qi
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Parav Pandit
Quoted 5 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 11, 2024 10:00 PM
How we can we make progress with
the realease but sure we don't make backwards compat a pain?Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Ideas?
There is no functional break with this relaxation. Device set some non-zero defaults and driver didn't modify them.... Anything broken? Unlikely.
Generally it's inappropriate to leave this decision making to the device for what would be the best / most performant default config, as the device is generally considered agnostic to the guest load...
Quoted 7 lines
Instead, the performance of the virtual machine and the driver depends heavily on how the device is implemented, just as we have proposed various ways to offload the data queue in the device to the hardware. The reason why most devices use software to simulate ctrlq instead of using hardware offload is that the driver has no requirements for the performance of ctrlq before, that is, the device implementation is responsible for and meets the driver's performance requirements.
Quoted 16 lines
I am not sure I follow the arguments of ctrlq being s/w or h/w, I thought that it was for the debate why we need coalescing on hardware offload device in the first place, instead of reusing event index or similar s/w based notification suppression. I don't doubt the value of coalescing, but how's it relevant to the default value disposition? Generally the default config disposition is not considered to be part of device implementation, especially when it comes to the situation where device can't easily figure out the specific workload to occur in the guest, and there's no perfect single default value that could meet every single performance metric across the board. This is a typical tuning knob left up to the user to adjust, why the device or driver needs to set or load the initial value? The driver just needs to start with certain value, be it 0 or non-zero, which guest user can override at any point of time, depending on his/her need, and that's it! I guess I still don't understand your user case here, why device / driver default is of such importance.
I've explained that, and I understand your argument is why default value is needed, and users should be able to adjust them, right?
The default value is working when the user doesn't adjust them. It's not practical to rely entirely on user adjustment, and for devices that serve hundreds of thousands of customers, the device implementation has to be comprehensive.
Thanks.
Quoted 155 lines
Quoted 8 lines
Quoted 8 lines
Unless the
device is specially hard wired to some fixed guest setup that users couldn't change, it doesn't seem logical that the device could derive the best or most performant config on driver's behalf. What if the guest wants best latency for its load but the device just blindlessly guess the guest might prefer throughput friendly that it miserably uses latency impacting non-zero default?Quoted 4 lines
The device does not want to guess and cannot guess. This patch does not force the device to choose a non-zero value, but relaxes it to allow the device to choose 0 or non-zero, which is very friendly to virtual machines with different performance requirements, right?
I don't understand the friendly part - do you imply your VM users are on kinda fixed wired setup that they cannot change these coalescing parameters after driver is loaded? Can the owner of the VM in control apply certain initial config for the coalescing parameters to the VM image? Or is it the problem of the guest driver that doesn't yet expose coalescing parameters to the end user? Otherwise I would think that guest user should be able to set parameters accordingly that would best fit the specific performance requirement of their own. How the device could even help here? I don't feel there's a lot of value to grant the device or host admin the flexibility to policy the *best* config on guest user's behalf, to be honest. And you seem to admit the fact that the default doesn't really matter, be it 0 or non-zero.
Quoted 5 lines
Quoted 4 lines
Could this device side change for
the default config regress boot time performance (which may need best latency over throughput)?Don't make these assumptions, what if the driver needs better throughput?
There's a misconception here: what we think that driver may need in terms on performance does actually reflect what guest user would like to have. Driver cannot read guest user's mind to make the decision, either.
In history there was drastic change in the Linux virtio-net driver that ever changed the default disposition for XPS (and RPS as well) from throughput and long-lived connection oriented to concurrency and short-lived oriented, which regressed a lot of existing setups that expects sustained throughput and packet rate after kernel upgrade. Although the occurrence of such drastic change for default disposition is not so welcome (that is one of the reasons why I valued consistent initial value and back compatible behavior), I don't see people yelling at virtio spec for less flexibility of offering the default disposition, given that the guest user can override the config any time with their own tooling or script, there's no problem at all for them to just set the corresponding config back explicitly to what it was before.
Quoted 12 lines
Quoted 12 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
And device/driver has better performance, is that a problem? Unlikely.
Even for rare case with a hard wired setup, the way to tackle the very problem using device's default is still quite questionable. Usually the mgmt software or network config utility should be equipped with some default value if need be. And we know the guest has the best position to impose the best / most performant config for its own load. What is the issue or use case that this initial config couldn't be applied by the guest mgmt software ahead but has to resort to the device to load some default (which is odd and irrelevant to any guest load), before the interface is brought up for operation i.e. performing I/O?
Quoted 2 lines
Use cases are everywhere, Alibaba Cloud, MLX and all other modern network cards have a default value that is not 0.
You seems to be referencing your own setup basically, and the question is still left answered - why the initial config can't be done through the mgmt software or network config utility within the guest?
Quoted 30 lines
(0 is actually a kind of default value)
Quoted 20 lines
Quoted 11 lines
Sorry, my vacation just ended.
Quoted 2 lines
Yes, it is possible. Driver can cache values it sets and never query device with get.
Don't we already have a lot of behaviors to drive queries from devices? RSS context, device stats.
Quoted 1 line
Before anything is set, driver will report incorrect values.
Devices that are widely supported and supported by good practices should have any initialization value. Just reporting 0 is incorrect value. Although the spec now says so.
I don't have an aligned view here, sorry. As I recall having 0 as the default is just to keep device started in a state where coalescing is disabled, so it's backward compatible and consistent with a non-coalescing supporting driver - such that it won't yield surprising effect (for e.g. regressed latency) inadvertently after user's getting driver software upgraded. Unlike the other virtio-net features that could 100% improve performance in all aspect, this coalescing feature is more of a performance tuning knob that may improve performance metrics (such as cpu usage or throughput) of one dimension while demoting the
Quoted 8 lines
others (such as latency, jitter or connection rate) from the equation. That said, there's not a single and fixed set of default config that device could supply which is able to satisfy all kind of guest load. Rather than rely on the device to offer a matching default for driver (which I think it's technically wrong), I'd lean to having guest software or network utility to apply the initial config for the guest, where they should have best knowledge for the specific guest workload than what device could do.
Quoted 2 lines
Before this feature, a good device implementation should also support coalescing (of course we don't necessarily assume it has coalescing).
Again, I don't doubt the value of supporting coalescing.
Quoted 4 lines
In addition, virtual
machines that tend to favor latency and throughput exist. If the device supported by the manufacturer needs to provide a low-latency virtual machine, please continue to keep the default value of 0.No, that's not what I was asking. There's no such requirement for any vendor to provide a low-latency or high throughput VM. The more general use case is - the setup for real world workload might just be too complex that end users would prefer low-latency on some virtual NIC or even some specific queues, while the other queues of a virtual NIC, or reset virtual NIC might have very different dispositions. Due to the needs and dynamics of workload scaling up & down, they might have more or less queues or virtual NICs to configure, so these disposition would need to be readjusted at any point of time, for which there's no easy way for device to adapt to easily. The guest user should have best knowledge for the specific guest workload and setup than what device could/should offer.
Quoted 15 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
What will break as a result? Hard to predict.
Given the patch is big, I am inclined to say it just should use a feature bit.
This doesn't seem to break anything in my opinion, we just told the device that now you can set more values.
Quoted 7 lines
Another thing this could break is live migration between devices with varied default. How do you make sure the guest doesn't rely on some default from the source device, while on the destination it just doesn't get the same default coalescingjjj value? To get rid of this side effect the guest would still need to apply the initial config for its own, anyway... Which eventually would render this proposal with arbitrary default rather pointless.
Quoted 2 lines
I don't quite understand why this would affect hot migration, the values would be migrated over.
Then I don't see this has been discussed (wouldn't the initial value be part of virtio device state to migrate over?) in the thread or described in the proposed text. And your proposed per-queue coalescing parameters (default plus current value) would have to be described too as part of virtio-net device state, so that people don't misunderstand your proposal.
Thanks,
-SiweiQuoted 12 lines
Thanks.
Quoted 10 lines
Thanks,
-SiweiQuoted 7 lines
Using new feature bits does not seem necessary.
Thanks.
Quoted 3 lines
Signature · 1 line
MST
Quoted 43 lines · Heng Qi
Quoted 26 lines · Si-Wei Liu
Quoted 17 lines · Heng Qi
Quoted 8 lines · Si-Wei Liu
Quoted 7 lines · Heng Qi
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Parav Pandit
Quoted 5 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 11, 2024 10:00 PM
How we can we make progress with
the realease but sure we don't make backwards compat a pain?Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Ideas?
There is no functional break with this relaxation. Device set some non-zero defaults and driver didn't modify them.... Anything broken? Unlikely.
Generally it's inappropriate to leave this decision making to the device for what would be the best / most performant default config, as the device is generally considered agnostic to the guest load...
Quoted 7 lines
Instead, the performance of the virtual machine and the driver depends heavily on how the device is implemented, just as we have proposed various ways to offload the data queue in the device to the hardware. The reason why most devices use software to simulate ctrlq instead of using hardware offload is that the driver has no requirements for the performance of ctrlq before, that is, the device implementation is responsible for and meets the driver's performance requirements.
Quoted 16 lines
I am not sure I follow the arguments of ctrlq being s/w or h/w, I thought that it was for the debate why we need coalescing on hardware offload device in the first place, instead of reusing event index or similar s/w based notification suppression. I don't doubt the value of coalescing, but how's it relevant to the default value disposition? Generally the default config disposition is not considered to be part of device implementation, especially when it comes to the situation where device can't easily figure out the specific workload to occur in the guest, and there's no perfect single default value that could meet every single performance metric across the board. This is a typical tuning knob left up to the user to adjust, why the device or driver needs to set or load the initial value? The driver just needs to start with certain value, be it 0 or non-zero, which guest user can override at any point of time, depending on his/her need, and that's it! I guess I still don't understand your user case here, why device / driver default is of such importance.
Quoted 2 lines
I've explained that, and I understand your argument is why default value is needed, and users should be able to adjust them, right?
Sounds about right. You'll soon realize that there's no perfect default that could work with everyone - in that it's just a static value to begin with, no matter whatever initial value the device comes up with, one user or another will come over to you and complain that what is loaded from the device doesn't match the workload they have, so those users are still expected to adjust manually tweaking for their own. As long as it's a tunable that guest user can control and override anytime, I don't feel it too much different what initial config the device would start with.
Quoted 2 lines
The default value is working when the user doesn't adjust them. It's not practical to rely entirely on user adjustment,
That's where the adaptive interrupt moderation framework (e.g. DIM for Linux) could come to the play, I think?
Quoted 3 lines
and for devices
that serve hundreds of thousands of customers, the device implementation has to be comprehensive.
Would you mind elaborating the technical difficulty for why the device implementation has to be comprehensive to serve hundreds or thousands of customers (i.e. adaptive moderation in the device), and is there better way to improve the current interface (ctrlq v.s. doorbell/MMIO register) that is hardware offload implementation wise. I feel people reading the thread without understanding the full background would become even more confused as to why it's relevant to your proposal, to me it's completely distinct use case or problem area that we are talking about.
Thanks,
-Siwei
Quoted 152 lines
Thanks.
Quoted 149 lines
Quoted 11 lines
Quoted 7 lines
Unless the
device is specially hard wired to some fixed guest setup that users couldn't change, it doesn't seem logical that the device could derive the best or most performant config on driver's behalf. What if the guest wants best latency for its load but the device just blindlessly guess the guest might prefer throughput friendly that it miserably uses latency impacting non-zero default?The device does not want to guess and cannot guess. This patch does not force the device to choose a non-zero value, but relaxes it to allow the device to choose 0 or non-zero, which is very friendly to virtual machines with different performance requirements, right?
I don't understand the friendly part - do you imply your VM users are on kinda fixed wired setup that they cannot change these coalescing parameters after driver is loaded? Can the owner of the VM in control apply certain initial config for the coalescing parameters to the VM image? Or is it the problem of the guest driver that doesn't yet expose coalescing parameters to the end user? Otherwise I would think that guest user should be able to set parameters accordingly that would best fit the specific performance requirement of their own. How the device could even help here? I don't feel there's a lot of value to grant the device or host admin the flexibility to policy the *best* config on guest user's behalf, to be honest. And you seem to admit the fact that the default doesn't really matter, be it 0 or non-zero.
Quoted 4 lines
Quoted 3 lines
Could this device side change for
the default config regress boot time performance (which may need best latency over throughput)?Don't make these assumptions, what if the driver needs better throughput?
There's a misconception here: what we think that driver may need in terms on performance does actually reflect what guest user would like to have. Driver cannot read guest user's mind to make the decision, either.
In history there was drastic change in the Linux virtio-net driver that ever changed the default disposition for XPS (and RPS as well) from throughput and long-lived connection oriented to concurrency and short-lived oriented, which regressed a lot of existing setups that expects sustained throughput and packet rate after kernel upgrade. Although the occurrence of such drastic change for default disposition is not so welcome (that is one of the reasons why I valued consistent initial value and back compatible behavior), I don't see people yelling at virtio spec for less flexibility of offering the default disposition, given that the guest user can override the config any time with their own tooling or script, there's no problem at all for them to just set the corresponding config back explicitly to what it was before.
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
And device/driver has better performance, is that a problem? Unlikely.
Even for rare case with a hard wired setup, the way to tackle the very problem using device's default is still quite questionable. Usually the mgmt software or network config utility should be equipped with some default value if need be. And we know the guest has the best position to impose the best / most performant config for its own load. What is the issue or use case that this initial config couldn't be applied by the guest mgmt software ahead but has to resort to the device to load some default (which is odd and irrelevant to any guest load), before the interface is brought up for operation i.e. performing I/O?
Quoted 2 lines
Use cases are everywhere, Alibaba Cloud, MLX and all other modern network cards have a default value that is not 0.
You seems to be referencing your own setup basically, and the question is still left answered - why the initial config can't be done through the mgmt software or network config utility within the guest?
Quoted 30 lines
(0 is actually a kind of default value)
Quoted 20 lines
Quoted 11 lines
Sorry, my vacation just ended.
Quoted 2 lines
Yes, it is possible. Driver can cache values it sets and never query device with get.
Don't we already have a lot of behaviors to drive queries from devices? RSS context, device stats.
Quoted 1 line
Before anything is set, driver will report incorrect values.
Devices that are widely supported and supported by good practices should have any initialization value. Just reporting 0 is incorrect value. Although the spec now says so.
I don't have an aligned view here, sorry. As I recall having 0 as the default is just to keep device started in a state where coalescing is disabled, so it's backward compatible and consistent with a non-coalescing supporting driver - such that it won't yield surprising effect (for e.g. regressed latency) inadvertently after user's getting driver software upgraded. Unlike the other virtio-net features that could 100% improve performance in all aspect, this coalescing feature is more of a performance tuning knob that may improve performance metrics (such as cpu usage or throughput) of one dimension while demoting the
Quoted 8 lines
others (such as latency, jitter or connection rate) from the equation. That said, there's not a single and fixed set of default config that device could supply which is able to satisfy all kind of guest load. Rather than rely on the device to offer a matching default for driver (which I think it's technically wrong), I'd lean to having guest software or network utility to apply the initial config for the guest, where they should have best knowledge for the specific guest workload than what device could do.
Quoted 2 lines
Before this feature, a good device implementation should also support coalescing (of course we don't necessarily assume it has coalescing).
Again, I don't doubt the value of supporting coalescing.
Quoted 4 lines
In addition, virtual
machines that tend to favor latency and throughput exist. If the device supported by the manufacturer needs to provide a low-latency virtual machine, please continue to keep the default value of 0.No, that's not what I was asking. There's no such requirement for any vendor to provide a low-latency or high throughput VM. The more general use case is - the setup for real world workload might just be too complex that end users would prefer low-latency on some virtual NIC or even some specific queues, while the other queues of a virtual NIC, or reset virtual NIC might have very different dispositions. Due to the needs and dynamics of workload scaling up & down, they might have more or less queues or virtual NICs to configure, so these disposition would need to be readjusted at any point of time, for which there's no easy way for device to adapt to easily. The guest user should have best knowledge for the specific guest workload and setup than what device could/should offer.
Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
What will break as a result? Hard to predict.
Given the patch is big, I am inclined to say it just should use a feature bit.
This doesn't seem to break anything in my opinion, we just told the device that now you can set more values.
Quoted 7 lines
Another thing this could break is live migration between devices with varied default. How do you make sure the guest doesn't rely on some default from the source device, while on the destination it just doesn't get the same default coalescingjjj value? To get rid of this side effect the guest would still need to apply the initial config for its own, anyway... Which eventually would render this proposal with arbitrary default rather pointless.
Quoted 2 lines
I don't quite understand why this would affect hot migration, the values would be migrated over.
Then I don't see this has been discussed (wouldn't the initial value be part of virtio device state to migrate over?) in the thread or described in the proposed text. And your proposed per-queue coalescing parameters (default plus current value) would have to be described too as part of virtio-net device state, so that people don't misunderstand your proposal.
Thanks,
-SiweiQuoted 11 lines
Thanks.
Quoted 9 lines
Thanks,
-SiweiQuoted 7 lines
Using new feature bits does not seem necessary.
Thanks.
Quoted 3 lines
Signature · 1 line
MST
Quoted 48 lines · Si-Wei Liu
Quoted 43 lines · Heng Qi
Quoted 26 lines · Si-Wei Liu
Quoted 17 lines · Heng Qi
Quoted 8 lines · Si-Wei Liu
Quoted 7 lines · Heng Qi
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Parav Pandit
Quoted 5 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 11, 2024 10:00 PM
How we can we make progress with
the realease but sure we don't make backwards compat a pain?Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Ideas?
There is no functional break with this relaxation. Device set some non-zero defaults and driver didn't modify them.... Anything broken? Unlikely.
Generally it's inappropriate to leave this decision making to the device for what would be the best / most performant default config, as the device is generally considered agnostic to the guest load...
Quoted 7 lines
Instead, the performance of the virtual machine and the driver depends heavily on how the device is implemented, just as we have proposed various ways to offload the data queue in the device to the hardware. The reason why most devices use software to simulate ctrlq instead of using hardware offload is that the driver has no requirements for the performance of ctrlq before, that is, the device implementation is responsible for and meets the driver's performance requirements.
Quoted 16 lines
I am not sure I follow the arguments of ctrlq being s/w or h/w, I thought that it was for the debate why we need coalescing on hardware offload device in the first place, instead of reusing event index or similar s/w based notification suppression. I don't doubt the value of coalescing, but how's it relevant to the default value disposition? Generally the default config disposition is not considered to be part of device implementation, especially when it comes to the situation where device can't easily figure out the specific workload to occur in the guest, and there's no perfect single default value that could meet every single performance metric across the board. This is a typical tuning knob left up to the user to adjust, why the device or driver needs to set or load the initial value? The driver just needs to start with certain value, be it 0 or non-zero, which guest user can override at any point of time, depending on his/her need, and that's it! I guess I still don't understand your user case here, why device / driver default is of such importance.
Quoted 2 lines
I've explained that, and I understand your argument is why default value is needed, and users should be able to adjust them, right?
Quoted 9 lines
Sounds about right. You'll soon realize that there's no perfect default that could work with everyone - in that it's just a static value to begin with, no matter whatever initial value the device comes up with, one user or another will come over to you and complain that what is loaded from the device doesn't match the workload they have, so those users are still expected to adjust manually tweaking for their own. As long as it's a tunable that guest user can control and override anytime, I don't feel it too much different what initial config the device would start with.
When you have a large number of customers and they buy your machines, how many users do you think have experience adjusting this value? More below.
Quoted 5 lines
Quoted 3 lines
The default value is working when the user doesn't adjust them. It's not practical to rely entirely on user adjustment,
That's where the adaptive interrupt moderation framework (e.g. DIM for Linux) could come to the play, I think?
Last time, this work is only useful if DIM is not enabled. I don't want to explain it again, I've said it many times to different people in the history discussion, don't you want to check any history discussion?
Quoted 4 lines
Quoted 4 lines
and for devices
that serve hundreds of thousands of customers, the device implementation has to be comprehensive.
Quoted 8 lines
Would you mind elaborating the technical difficulty for why the device implementation has to be comprehensive to serve hundreds or thousands of customers (i.e. adaptive moderation in the device), and is there better way to improve the current interface (ctrlq v.s. doorbell/MMIO register) that is hardware offload implementation wise. I feel people reading the thread without understanding the full background would become even more confused as to why it's relevant to your proposal, to me it's completely distinct use case or problem area that we are talking about.
Regardless of whether DIM can be enabled by default (it cannot accelerate all scenarios, right?), therefore, a considerable number of machines on the cloud vendors' line still have to rely on static values to provide boot-up performance. The vendors will also make optimizations to optimize scenarios like ping-pong, so the boot-up performance is very good.
Thanks.
Quoted 157 lines
Thanks,
-SiweiQuoted 152 lines
Thanks.
Quoted 149 lines
Quoted 11 lines
Quoted 7 lines
Unless the
device is specially hard wired to some fixed guest setup that users couldn't change, it doesn't seem logical that the device could derive the best or most performant config on driver's behalf. What if the guest wants best latency for its load but the device just blindlessly guess the guest might prefer throughput friendly that it miserably uses latency impacting non-zero default?The device does not want to guess and cannot guess. This patch does not force the device to choose a non-zero value, but relaxes it to allow the device to choose 0 or non-zero, which is very friendly to virtual machines with different performance requirements, right?
I don't understand the friendly part - do you imply your VM users are on kinda fixed wired setup that they cannot change these coalescing parameters after driver is loaded? Can the owner of the VM in control apply certain initial config for the coalescing parameters to the VM image? Or is it the problem of the guest driver that doesn't yet expose coalescing parameters to the end user? Otherwise I would think that guest user should be able to set parameters accordingly that would best fit the specific performance requirement of their own. How the device could even help here? I don't feel there's a lot of value to grant the device or host admin the flexibility to policy the *best* config on guest user's behalf, to be honest. And you seem to admit the fact that the default doesn't really matter, be it 0 or non-zero.
Quoted 4 lines
Quoted 3 lines
Could this device side change for
the default config regress boot time performance (which may need best latency over throughput)?Don't make these assumptions, what if the driver needs better throughput?
There's a misconception here: what we think that driver may need in terms on performance does actually reflect what guest user would like to have. Driver cannot read guest user's mind to make the decision, either.
In history there was drastic change in the Linux virtio-net driver that ever changed the default disposition for XPS (and RPS as well) from throughput and long-lived connection oriented to concurrency and short-lived oriented, which regressed a lot of existing setups that expects sustained throughput and packet rate after kernel upgrade. Although the occurrence of such drastic change for default disposition is not so welcome (that is one of the reasons why I valued consistent initial value and back compatible behavior), I don't see people yelling at virtio spec for less flexibility of offering the default disposition, given that the guest user can override the config any time with their own tooling or script, there's no problem at all for them to just set the corresponding config back explicitly to what it was before.
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
And device/driver has better performance, is that a problem? Unlikely.
Even for rare case with a hard wired setup, the way to tackle the very problem using device's default is still quite questionable. Usually the mgmt software or network config utility should be equipped with some default value if need be. And we know the guest has the best position to impose the best / most performant config for its own load. What is the issue or use case that this initial config couldn't be applied by the guest mgmt software ahead but has to resort to the device to load some default (which is odd and irrelevant to any guest load), before the interface is brought up for operation i.e. performing I/O?
Quoted 2 lines
Use cases are everywhere, Alibaba Cloud, MLX and all other modern network cards have a default value that is not 0.
You seems to be referencing your own setup basically, and the question is still left answered - why the initial config can't be done through the mgmt software or network config utility within the guest?
Quoted 30 lines
(0 is actually a kind of default value)
Quoted 20 lines
Quoted 11 lines
Sorry, my vacation just ended.
Quoted 2 lines
Yes, it is possible. Driver can cache values it sets and never query device with get.
Don't we already have a lot of behaviors to drive queries from devices? RSS context, device stats.
Quoted 1 line
Before anything is set, driver will report incorrect values.
Devices that are widely supported and supported by good practices should have any initialization value. Just reporting 0 is incorrect value. Although the spec now says so.
I don't have an aligned view here, sorry. As I recall having 0 as the default is just to keep device started in a state where coalescing is disabled, so it's backward compatible and consistent with a non-coalescing supporting driver - such that it won't yield surprising effect (for e.g. regressed latency) inadvertently after user's getting driver software upgraded. Unlike the other virtio-net features that could 100% improve performance in all aspect, this coalescing feature is more of a performance tuning knob that may improve performance metrics (such as cpu usage or throughput) of one dimension while demoting the
Quoted 8 lines
others (such as latency, jitter or connection rate) from the equation. That said, there's not a single and fixed set of default config that device could supply which is able to satisfy all kind of guest load. Rather than rely on the device to offer a matching default for driver (which I think it's technically wrong), I'd lean to having guest software or network utility to apply the initial config for the guest, where they should have best knowledge for the specific guest workload than what device could do.
Quoted 2 lines
Before this feature, a good device implementation should also support coalescing (of course we don't necessarily assume it has coalescing).
Again, I don't doubt the value of supporting coalescing.
Quoted 4 lines
In addition, virtual
machines that tend to favor latency and throughput exist. If the device supported by the manufacturer needs to provide a low-latency virtual machine, please continue to keep the default value of 0.No, that's not what I was asking. There's no such requirement for any vendor to provide a low-latency or high throughput VM. The more general use case is - the setup for real world workload might just be too complex that end users would prefer low-latency on some virtual NIC or even some specific queues, while the other queues of a virtual NIC, or reset virtual NIC might have very different dispositions. Due to the needs and dynamics of workload scaling up & down, they might have more or less queues or virtual NICs to configure, so these disposition would need to be readjusted at any point of time, for which there's no easy way for device to adapt to easily. The guest user should have best knowledge for the specific guest workload and setup than what device could/should offer.
Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
What will break as a result? Hard to predict.
Given the patch is big, I am inclined to say it just should use a feature bit.
This doesn't seem to break anything in my opinion, we just told the device that now you can set more values.
Quoted 7 lines
Another thing this could break is live migration between devices with varied default. How do you make sure the guest doesn't rely on some default from the source device, while on the destination it just doesn't get the same default coalescingjjj value? To get rid of this side effect the guest would still need to apply the initial config for its own, anyway... Which eventually would render this proposal with arbitrary default rather pointless.
Quoted 2 lines
I don't quite understand why this would affect hot migration, the values would be migrated over.
Then I don't see this has been discussed (wouldn't the initial value be part of virtio device state to migrate over?) in the thread or described in the proposed text. And your proposed per-queue coalescing parameters (default plus current value) would have to be described too as part of virtio-net device state, so that people don't misunderstand your proposal.
Thanks,
-SiweiQuoted 11 lines
Thanks.
Quoted 9 lines
Thanks,
-SiweiQuoted 7 lines
Using new feature bits does not seem necessary.
Thanks.
Quoted 3 lines
Signature · 1 line
MST
Quoted 55 lines · Heng Qi
Quoted 45 lines · Si-Wei Liu
Quoted 42 lines · Heng Qi
Quoted 25 lines · Si-Wei Liu
Quoted 17 lines · Heng Qi
Quoted 8 lines · Si-Wei Liu
Quoted 7 lines · Heng Qi
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Parav Pandit
Quoted 5 lines
From: Michael S. Tsirkin<[email protected]>
Sent: Tuesday, June 11, 2024 10:00 PM
How we can we make progress with
the realease but sure we don't make backwards compat a pain?Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Ideas?
There is no functional break with this relaxation. Device set some non-zero defaults and driver didn't modify them.... Anything broken? Unlikely.
Generally it's inappropriate to leave this decision making to the device for what would be the best / most performant default config, as the device is generally considered agnostic to the guest load...
Quoted 7 lines
Instead, the performance of the virtual machine and the driver depends heavily on how the device is implemented, just as we have proposed various ways to offload the data queue in the device to the hardware. The reason why most devices use software to simulate ctrlq instead of using hardware offload is that the driver has no requirements for the performance of ctrlq before, that is, the device implementation is responsible for and meets the driver's performance requirements.
Quoted 16 lines
I am not sure I follow the arguments of ctrlq being s/w or h/w, I thought that it was for the debate why we need coalescing on hardware offload device in the first place, instead of reusing event index or similar s/w based notification suppression. I don't doubt the value of coalescing, but how's it relevant to the default value disposition? Generally the default config disposition is not considered to be part of device implementation, especially when it comes to the situation where device can't easily figure out the specific workload to occur in the guest, and there's no perfect single default value that could meet every single performance metric across the board. This is a typical tuning knob left up to the user to adjust, why the device or driver needs to set or load the initial value? The driver just needs to start with certain value, be it 0 or non-zero, which guest user can override at any point of time, depending on his/her need, and that's it! I guess I still don't understand your user case here, why device / driver default is of such importance.
Quoted 2 lines
I've explained that, and I understand your argument is why default value is needed, and users should be able to adjust them, right?
Quoted 9 lines
Sounds about right. You'll soon realize that there's no perfect default that could work with everyone - in that it's just a static value to begin with, no matter whatever initial value the device comes up with, one user or another will come over to you and complain that what is loaded from the device doesn't match the workload they have, so those users are still expected to adjust manually tweaking for their own. As long as it's a tunable that guest user can control and override anytime, I don't feel it too much different what initial config the device would start with.
Quoted 2 lines
When you have a large number of customers and they buy your machines, how many users do you think have experience adjusting this value? More below.
This is not what this proposal alone could address, as it just shifts the sheer pain to the device and host admin rather than provide any flexibility. Technically I believe the right solution is to seek adaptive coalescing that doesn't require user to tune anything. Noted adaptive coalescing / suppression could be done from the guest or by the device. For e.g. the s/w based event index based notification suppression could also work with hardware if the event index is somehow propagated through a doorbell register rather than load from host memory over the PCI transport.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
The default value is working when the user doesn't adjust them. It's not practical to rely entirely on user adjustment,
That's where the adaptive interrupt moderation framework (e.g. DIM for Linux) could come to the play, I think?
Last time, this work is only useful if DIM is not enabled. I don't want to explain it again, I've said it many times to different people in the history discussion, don't you want to check any history discussion?
I did before I replied. I thought both Halil and Michael tried hard to convey is that what you claimed to work by just getting the initial config loaded from device will not be sufficient to address all the performance requirements in broad and general means other than satisfy the goal set by yourself. What I saw is that you kept proving yourself by trying to reference your own use case and design - this not just mixed different things up but also ended up with circular argument that wasted everyone's time unfortunately.
Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
and for devices
that serve hundreds of thousands of customers, the device implementation has to be comprehensive.Would you mind elaborating the technical difficulty for why the device implementation has to be comprehensive to serve hundreds or thousands of customers (i.e. adaptive moderation in the device), and is there better way to improve the current interface (ctrlq v.s. doorbell/MMIO register) that is hardware offload implementation wise. I feel people reading the thread without understanding the full background would become even more confused as to why it's relevant to your proposal, to me it's completely distinct use case or problem area that we are talking about.
Quoted 4 lines
Regardless of whether DIM can be enabled by default (it cannot accelerate all scenarios, right?), therefore, a considerable number of machines on the cloud vendors' line still have to rely on static values to provide boot-up performance.
Sorry, this is not a good excuse for people to accept a sub-optimal solution. Please talk to the other hardware vendor or read my above reply carefully.
-Siwei
Quoted 159 lines
The vendors will also make optimizations to optimize scenarios
like ping-pong, so the boot-up performance is very good.Thanks.
Quoted 154 lines
Thanks,
-SiweiQuoted 151 lines
Thanks.
Quoted 149 lines
Quoted 11 lines
Quoted 7 lines
Unless the
device is specially hard wired to some fixed guest setup that users couldn't change, it doesn't seem logical that the device could derive the best or most performant config on driver's behalf. What if the guest wants best latency for its load but the device just blindlessly guess the guest might prefer throughput friendly that it miserably uses latency impacting non-zero default?The device does not want to guess and cannot guess. This patch does not force the device to choose a non-zero value, but relaxes it to allow the device to choose 0 or non-zero, which is very friendly to virtual machines with different performance requirements, right?
I don't understand the friendly part - do you imply your VM users are on kinda fixed wired setup that they cannot change these coalescing parameters after driver is loaded? Can the owner of the VM in control apply certain initial config for the coalescing parameters to the VM image? Or is it the problem of the guest driver that doesn't yet expose coalescing parameters to the end user? Otherwise I would think that guest user should be able to set parameters accordingly that would best fit the specific performance requirement of their own. How the device could even help here? I don't feel there's a lot of value to grant the device or host admin the flexibility to policy the *best* config on guest user's behalf, to be honest. And you seem to admit the fact that the default doesn't really matter, be it 0 or non-zero.
Quoted 4 lines
Quoted 3 lines
Could this device side change for
the default config regress boot time performance (which may need best latency over throughput)?Don't make these assumptions, what if the driver needs better throughput?
There's a misconception here: what we think that driver may need in terms on performance does actually reflect what guest user would like to have. Driver cannot read guest user's mind to make the decision, either.
In history there was drastic change in the Linux virtio-net driver that ever changed the default disposition for XPS (and RPS as well) from throughput and long-lived connection oriented to concurrency and short-lived oriented, which regressed a lot of existing setups that expects sustained throughput and packet rate after kernel upgrade. Although the occurrence of such drastic change for default disposition is not so welcome (that is one of the reasons why I valued consistent initial value and back compatible behavior), I don't see people yelling at virtio spec for less flexibility of offering the default disposition, given that the guest user can override the config any time with their own tooling or script, there's no problem at all for them to just set the corresponding config back explicitly to what it was before.
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
And device/driver has better performance, is that a problem? Unlikely.
Even for rare case with a hard wired setup, the way to tackle the very problem using device's default is still quite questionable. Usually the mgmt software or network config utility should be equipped with some default value if need be. And we know the guest has the best position to impose the best / most performant config for its own load. What is the issue or use case that this initial config couldn't be applied by the guest mgmt software ahead but has to resort to the device to load some default (which is odd and irrelevant to any guest load), before the interface is brought up for operation i.e. performing I/O?
Quoted 2 lines
Use cases are everywhere, Alibaba Cloud, MLX and all other modern network cards have a default value that is not 0.
You seems to be referencing your own setup basically, and the question is still left answered - why the initial config can't be done through the mgmt software or network config utility within the guest?
Quoted 30 lines
(0 is actually a kind of default value)
Quoted 20 lines
Quoted 11 lines
Sorry, my vacation just ended.
Quoted 2 lines
Yes, it is possible. Driver can cache values it sets and never query device with get.
Don't we already have a lot of behaviors to drive queries from devices? RSS context, device stats.
Quoted 1 line
Before anything is set, driver will report incorrect values.
Devices that are widely supported and supported by good practices should have any initialization value. Just reporting 0 is incorrect value. Although the spec now says so.
I don't have an aligned view here, sorry. As I recall having 0 as the default is just to keep device started in a state where coalescing is disabled, so it's backward compatible and consistent with a non-coalescing supporting driver - such that it won't yield surprising effect (for e.g. regressed latency) inadvertently after user's getting driver software upgraded. Unlike the other virtio-net features that could 100% improve performance in all aspect, this coalescing feature is more of a performance tuning knob that may improve performance metrics (such as cpu usage or throughput) of one dimension while demoting the
Quoted 8 lines
others (such as latency, jitter or connection rate) from the equation. That said, there's not a single and fixed set of default config that device could supply which is able to satisfy all kind of guest load. Rather than rely on the device to offer a matching default for driver (which I think it's technically wrong), I'd lean to having guest software or network utility to apply the initial config for the guest, where they should have best knowledge for the specific guest workload than what device could do.
Quoted 2 lines
Before this feature, a good device implementation should also support coalescing (of course we don't necessarily assume it has coalescing).
Again, I don't doubt the value of supporting coalescing.
Quoted 4 lines
In addition, virtual
machines that tend to favor latency and throughput exist. If the device supported by the manufacturer needs to provide a low-latency virtual machine, please continue to keep the default value of 0.No, that's not what I was asking. There's no such requirement for any vendor to provide a low-latency or high throughput VM. The more general use case is - the setup for real world workload might just be too complex that end users would prefer low-latency on some virtual NIC or even some specific queues, while the other queues of a virtual NIC, or reset virtual NIC might have very different dispositions. Due to the needs and dynamics of workload scaling up & down, they might have more or less queues or virtual NICs to configure, so these disposition would need to be readjusted at any point of time, for which there's no easy way for device to adapt to easily. The guest user should have best knowledge for the specific guest workload and setup than what device could/should offer.
Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
What will break as a result? Hard to predict.
Given the patch is big, I am inclined to say it just should use a feature bit.
This doesn't seem to break anything in my opinion, we just told the device that now you can set more values.
Quoted 7 lines
Another thing this could break is live migration between devices with varied default. How do you make sure the guest doesn't rely on some default from the source device, while on the destination it just doesn't get the same default coalescingjjj value? To get rid of this side effect the guest would still need to apply the initial config for its own, anyway... Which eventually would render this proposal with arbitrary default rather pointless.
Quoted 2 lines
I don't quite understand why this would affect hot migration, the values would be migrated over.
Then I don't see this has been discussed (wouldn't the initial value be part of virtio device state to migrate over?) in the thread or described in the proposed text. And your proposed per-queue coalescing parameters (default plus current value) would have to be described too as part of virtio-net device state, so that people don't misunderstand your proposal.
Thanks,
-SiweiQuoted 11 lines
Thanks.
Quoted 9 lines
Thanks,
-SiweiQuoted 7 lines
Using new feature bits does not seem necessary.
Thanks.
Quoted 3 lines
Signature · 1 line
MST
Hi Si-Wei,
Quoted 266 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, June 25, 2024 10:22 AMQuoted 1 line · Heng Qi
On Fri, 21 Jun 2024 16:46:27 -0700, "Si-Wei Liu"<[email protected]>
Quoted 2 lines · wrote:
Quoted 2 lines · Heng Qi
Quoted 1 line
On Thu, 20 Jun 2024 18:21:03 -0700, "Si-Wei Liu"<[email protected]>
Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines · Heng Qi
Quoted 1 line
On Mon, 17 Jun 2024 16:31:25 -0700, "Si-Wei Liu"<si-
Quoted 2 lines · [email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines · Heng Qi
Quoted 1 line
On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S.
Quoted 10 lines · Tsirkin
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines · Parav Pandit
Quoted 9 lines
Quoted 7 lines
From: Michael S. Tsirkin<[email protected]>
Sent: Tuesday, June 11, 2024 10:00 PM How we can we make progress with the realease but sure we don't make backwards compat a pain?Ideas?
There is no functional break with this relaxation. Device set some non-zero defaults and driver didn't modify
them....
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Anything broken? Unlikely.
Generally it's inappropriate to leave this decision making to the device for what would be the best / most performant default config, as the device is generally considered agnostic to the guest
load...
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Instead, the performance of the virtual machine and the driver depends heavily on how the device is implemented, just as we have proposed various ways to offload the data queue in the device to the hardware. The reason why most devices use software to simulate ctrlq instead of using hardware offload is that the driver has no requirements for the performance of ctrlq before, that is, the device implementation is responsible for and meets the driver's
performance requirements.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
I am not sure I follow the arguments of ctrlq being s/w or h/w, I thought that it was for the debate why we need coalescing on hardware offload device in the first place, instead of reusing event index or similar s/w based notification suppression. I don't doubt the value of coalescing, but how's it relevant to the default value
disposition?
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Generally the default config disposition is not considered to be part of device implementation, especially when it comes to the situation where device can't easily figure out the specific workload to occur in the guest, and there's no perfect single default value that could meet every single performance metric across the board. This is a typical tuning knob left up to the user to adjust, why the device or driver needs to set or load the initial value? The driver just needs to start with certain value, be it 0 or non-zero, which guest user can override at any point of time, depending on his/her need, and that's it! I guess I still don't understand your user case here, why device / driver default is of
such importance.
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
I've explained that, and I understand your argument is why default value is needed, and users should be able to adjust them, right?
Sounds about right. You'll soon realize that there's no perfect default that could work with everyone - in that it's just a static value to begin with, no matter whatever initial value the device comes up with, one user or another will come over to you and complain that what is loaded from the device doesn't match the workload they have, so those users are still expected to adjust manually tweaking for their own. As long as it's a tunable that guest user can control and override anytime, I don't feel it too much different what initial config the device would start with.
Quoted 2 lines
When you have a large number of customers and they buy your machines, how many users do you think have experience adjusting this value? More
below.
This is not what this proposal alone could address, as it just shifts the sheer pain to the device and host admin rather than provide any flexibility. Technically I believe the right solution is to seek adaptive coalescing that doesn't require user to tune anything. Noted adaptive coalescing / suppression could be done from the guest or by the device. For e.g. the s/w based event index based notification suppression could also work with hardware if the event index is somehow propagated through a doorbell register rather than load from host memory over the PCI transport.Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
The default value is working when the user doesn't adjust them. It's not practical to rely entirely on user adjustment,
That's where the adaptive interrupt moderation framework (e.g. DIM for
Linux) could come to the play, I think?Last time, this work is only useful if DIM is not enabled. I don't want to explain it again, I've said it many times to different people in the history discussion, don't you want to check any history discussion?
I did before I replied. I thought both Halil and Michael tried hard to convey is that what you claimed to work by just getting the initial config loaded from device will not be sufficient to address all the performance requirements in broad and general means other than satisfy the goal set by yourself. What I saw is that you kept proving yourself by trying to reference your own use case and design - this not just mixed different things up but also ended up with circular argument that wasted everyone's time unfortunately.
Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
and for devices
that serve hundreds of thousands of customers, the device implementation has to be comprehensive.Would you mind elaborating the technical difficulty for why the device implementation has to be comprehensive to serve hundreds or thousands of customers (i.e. adaptive moderation in the device), and is there better way to improve the current interface (ctrlq v.s. doorbell/MMIO register) that is hardware offload implementation wise. I feel people reading the thread without understanding the full background would become even more confused as to why it's relevant to your proposal, to me it's completely distinct use case or problem area that
we are talking about.
Quoted 4 lines
Regardless of whether DIM can be enabled by default (it cannot accelerate all scenarios, right?), therefore, a considerable number of machines on the cloud vendors' line still have to rely on static values to provide boot-up performance.
Sorry, this is not a good excuse for people to accept a sub-optimal solution. Please talk to the other hardware vendor or read my above reply carefully.
-Siwei
Quoted 21 lines
The vendors will also make optimizations to optimize scenarios like
ping-pong, so the boot-up performance is very good.Thanks.
Quoted 16 lines
Thanks,
-SiweiQuoted 13 lines
Thanks.
Quoted 11 lines
Quoted 11 lines
Quoted 7 lines
Unless the
device is specially hard wired to some fixed guest setup that users couldn't change, it doesn't seem logical that the device could derive the best or most performant config on driver's behalf. What if the guest wants best latency for its load but the device just blindlessly guess the guest might prefer throughput friendly that it miserably uses latency impacting non-zero default?The device does not want to guess and cannot guess. This patch does not force the device to choose a non-zero value, but relaxes it to allow the device to choose 0 or non-zero, which is very friendly to virtual machines with different performance requirements,
right?
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
I don't understand the friendly part - do you imply your VM users are on kinda fixed wired setup that they cannot change these coalescing parameters after driver is loaded? Can the owner of the VM in control apply certain initial config for the coalescing
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
parameters to the VM image? Or is it the problem of the guest driver that doesn't yet expose coalescing parameters to the end user? Otherwise I would think that guest user should be able to set parameters accordingly that would best fit the specific performance requirement of their own. How the device could even help here? I don't feel there's a lot of value to grant the device or host admin the flexibility to policy the *best* config on guest user's behalf, to be honest. And you seem to admit the fact that the default doesn't
really matter, be it 0 or non-zero.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Could this device side change for the default config regress boot time performance (which may need best latency over throughput)?
Don't make these assumptions, what if the driver needs better
throughput?
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
There's a misconception here: what we think that driver may need in terms on performance does actually reflect what guest user would like to have. Driver cannot read guest user's mind to make the decision,
either.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
In history there was drastic change in the Linux virtio-net driver that ever changed the default disposition for XPS (and RPS as well) from throughput and long-lived connection oriented to concurrency and short-lived oriented, which regressed a lot of existing setups that expects sustained throughput and packet rate after kernel
upgrade.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Although the occurrence of such drastic change for default disposition is not so welcome (that is one of the reasons why I valued consistent initial value and back compatible behavior), I don't see people yelling at virtio spec for less flexibility of offering the default disposition, given that the guest user can override the config any time with their own tooling or script, there's no problem at all for them to just set the corresponding config
back explicitly to what it was before.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
And device/driver has better performance, is that a problem?
Unlikely.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Even for rare case with a hard wired setup, the way to tackle the very problem using device's default is still quite questionable. Usually the mgmt software or network config utility should be equipped with some default value if need be. And we know the guest has the best position to impose the best / most performant config for its own load. What is the issue or use case that this initial config couldn't be applied by the guest mgmt software ahead but has to resort to the device to load some default (which is odd and irrelevant to any guest load), before the interface is
brought up for operation i.e. performing I/O?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Use cases are everywhere, Alibaba Cloud, MLX and all other modern network cards have a default value that is not 0.
You seems to be referencing your own setup basically, and the question is still left answered - why the initial config can't be done through the mgmt software or network config utility within the
guest?
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
(0 is actually a kind of default value)
Quoted 5 lines
Quoted 5 lines
Sorry, my vacation just ended.
Quoted 2 lines
Yes, it is possible. Driver can cache values it sets and never query device with get.
Don't we already have a lot of behaviors to drive queries from
devices?
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
RSS context, device stats.
Quoted 1 line
Before anything is set, driver will report incorrect values.
Devices that are widely supported and supported by good practices should have any initialization value. Just reporting 0 is incorrect value. Although the spec now says so.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
I don't have an aligned view here, sorry. As I recall having 0 as the default is just to keep device started in a state where coalescing is disabled, so it's backward compatible and consistent with a non-coalescing supporting driver - such that it won't yield surprising effect (for e.g. regressed latency) inadvertently after user's getting driver software upgraded. Unlike the other virtio-net features that could 100% improve performance in all aspect, this coalescing feature is more of a performance tuning knob that may improve performance metrics (such as cpu usage or throughput) of one dimension while demoting
the others (such as latency, jitter or connection rate) from the equation.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
That said, there's not a single and fixed set of default config that device could supply which is able to satisfy all kind of guest load. Rather than rely on the device to offer a matching default for driver (which I think it's technically wrong), I'd lean to having guest software or network utility to apply the initial config for the guest, where they should have best knowledge for the specific guest workload than what device could do.
Before this feature, a good device implementation should also support coalescing (of course we don't necessarily assume it has
coalescing).
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Again, I don't doubt the value of supporting coalescing.
Quoted 4 lines
In addition, virtual
machines that tend to favor latency and throughput exist. If the device supported by the manufacturer needs to provide a low-latency virtual machine, please continue to keep the default valueof 0.
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
No, that's not what I was asking. There's no such requirement for any vendor to provide a low-latency or high throughput VM. The more general use case is - the setup for real world workload might just be too complex that end users would prefer low-latency on some virtual NIC or even some specific queues, while the other queues of a virtual NIC, or reset virtual NIC might have very different dispositions. Due to the needs and dynamics of workload scaling up & down, they might have more or less queues or virtual NICs to configure, so these disposition would need to be readjusted at any point of time, for which there's no easy way for device to adapt to easily. The guest user should have best knowledge for the specific guest workload and setup than what device could/should offer.
Quoted 14 lines
Quoted 14 lines
Quoted 7 lines
Quoted 5 lines
What will break as a result? Hard to predict.
Given the patch is big, I am inclined to say it just should use a feature bit.
This doesn't seem to break anything in my opinion, we just told the device that now you can set more values.
Another thing this could break is live migration between devices with varied default. How do you make sure the guest doesn't rely on some default from the source device, while on the destination it just doesn't get the same default coalescingjjj value? To get rid of this side effect the guest would still need to apply the initial config for its own, anyway... Which eventually would render this proposal with arbitrary default rather pointless.
Quoted 23 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
I don't quite understand why this would affect hot migration, the values would be migrated over.
Then I don't see this has been discussed (wouldn't the initial value be part of virtio device state to migrate over?) in the thread or described in the proposed text. And your proposed per-queue coalescing parameters (default plus current value) would have to be described too as part of virtio-net device state, so that
people don't misunderstand your proposal.
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Thanks,
-SiweiQuoted 11 lines
Thanks.
Quoted 9 lines
Thanks,
-SiweiQuoted 7 lines
Using new feature bits does not seem necessary.
Thanks.
Quoted 3 lines
--
MST
I saw the need of this proposal slightly differently in the discussion with Heng in v4.
The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offering VIRTIO_NET_F_VQ_NOTF_COAL.
Flow is:
1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL
2. The virtio-net driver negotiated VIRTNET_FEATURES that has VIRTIO_NET_F_VQ_NOTF_COAL
3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
4. the virtio-net driver even after negotiating VIRTIO_NET_F_VQ_NOTF_COAL, still kept the its dim disabled. vi->rq[queue].dim_enabled = false by default.
5. The virtio net driver enables dim only on user request via ethtool.
So here, just because one is using a new driver and new device, it may get lower performance.
Do we agree to above problem description?
If no, I likely missed something in the long discussion and need to read all of it. :(
If yes, few solutions are:
1. A user intervention is must to avoid this regression. The user in the guest VM must enable DIM if the device supports it. Very hard to do this plumbing.
2. A net driver enables the DIM by default as long as VIRTIO_NET_F_VQ_NOTF_COAL is negotiated. (without ethtool setting)
3. A device relaxes the limitation and continue to apply the coalescing, until driver overrides it.
In my humble opinion, Heng is solving it using option #3, that tends to work with existing and future drivers who may/may not enable the DIM.
Quoted 292 lines · Parav Pandit
Hi Si-Wei,
Quoted 261 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, June 25, 2024 10:22 AMQuoted 1 line · Heng Qi
On Fri, 21 Jun 2024 16:46:27 -0700, "Si-Wei Liu"<[email protected]>
Quoted 2 lines · wrote:
Quoted 2 lines · Heng Qi
Quoted 1 line
On Thu, 20 Jun 2024 18:21:03 -0700, "Si-Wei Liu"<[email protected]>
Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines · Heng Qi
Quoted 1 line
On Mon, 17 Jun 2024 16:31:25 -0700, "Si-Wei Liu"<si-
Quoted 2 lines · [email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines · Heng Qi
Quoted 1 line
On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S.
Quoted 10 lines · Tsirkin
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines · Parav Pandit
Quoted 9 lines
Quoted 7 lines
From: Michael S. Tsirkin<[email protected]>
Sent: Tuesday, June 11, 2024 10:00 PM How we can we make progress with the realease but sure we don't make backwards compat a pain?Ideas?
There is no functional break with this relaxation. Device set some non-zero defaults and driver didn't modify
them....
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Anything broken? Unlikely.
Generally it's inappropriate to leave this decision making to the device for what would be the best / most performant default config, as the device is generally considered agnostic to the guest
load...
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Instead, the performance of the virtual machine and the driver depends heavily on how the device is implemented, just as we have proposed various ways to offload the data queue in the device to the hardware. The reason why most devices use software to simulate ctrlq instead of using hardware offload is that the driver has no requirements for the performance of ctrlq before, that is, the device implementation is responsible for and meets the driver's
performance requirements.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
I am not sure I follow the arguments of ctrlq being s/w or h/w, I thought that it was for the debate why we need coalescing on hardware offload device in the first place, instead of reusing event index or similar s/w based notification suppression. I don't doubt the value of coalescing, but how's it relevant to the default value
disposition?
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Generally the default config disposition is not considered to be part of device implementation, especially when it comes to the situation where device can't easily figure out the specific workload to occur in the guest, and there's no perfect single default value that could meet every single performance metric across the board. This is a typical tuning knob left up to the user to adjust, why the device or driver needs to set or load the initial value? The driver just needs to start with certain value, be it 0 or non-zero, which guest user can override at any point of time, depending on his/her need, and that's it! I guess I still don't understand your user case here, why device / driver default is of
such importance.
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
I've explained that, and I understand your argument is why default value is needed, and users should be able to adjust them, right?
Sounds about right. You'll soon realize that there's no perfect default that could work with everyone - in that it's just a static value to begin with, no matter whatever initial value the device comes up with, one user or another will come over to you and complain that what is loaded from the device doesn't match the workload they have, so those users are still expected to adjust manually tweaking for their own. As long as it's a tunable that guest user can control and override anytime, I don't feel it too much different what initial config the device would start with.
Quoted 2 lines
When you have a large number of customers and they buy your machines, how many users do you think have experience adjusting this value? More
below.
This is not what this proposal alone could address, as it just shifts the sheer pain to the device and host admin rather than provide any flexibility. Technically I believe the right solution is to seek adaptive coalescing that doesn't require user to tune anything. Noted adaptive coalescing / suppression could be done from the guest or by the device. For e.g. the s/w based event index based notification suppression could also work with hardware if the event index is somehow propagated through a doorbell register rather than load from host memory over the PCI transport.Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
The default value is working when the user doesn't adjust them. It's not practical to rely entirely on user adjustment,
That's where the adaptive interrupt moderation framework (e.g. DIM for
Linux) could come to the play, I think?Last time, this work is only useful if DIM is not enabled. I don't want to explain it again, I've said it many times to different people in the history discussion, don't you want to check any history discussion?
I did before I replied. I thought both Halil and Michael tried hard to convey is that what you claimed to work by just getting the initial config loaded from device will not be sufficient to address all the performance requirements in broad and general means other than satisfy the goal set by yourself. What I saw is that you kept proving yourself by trying to reference your own use case and design - this not just mixed different things up but also ended up with circular argument that wasted everyone's time unfortunately.
Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
and for devices
that serve hundreds of thousands of customers, the device implementation has to be comprehensive.Would you mind elaborating the technical difficulty for why the device implementation has to be comprehensive to serve hundreds or thousands of customers (i.e. adaptive moderation in the device), and is there better way to improve the current interface (ctrlq v.s. doorbell/MMIO register) that is hardware offload implementation wise. I feel people reading the thread without understanding the full background would become even more confused as to why it's relevant to your proposal, to me it's completely distinct use case or problem area that
we are talking about.
Quoted 4 lines
Regardless of whether DIM can be enabled by default (it cannot accelerate all scenarios, right?), therefore, a considerable number of machines on the cloud vendors' line still have to rely on static values to provide boot-up performance.
Sorry, this is not a good excuse for people to accept a sub-optimal solution. Please talk to the other hardware vendor or read my above reply carefully.
-Siwei
Quoted 21 lines
The vendors will also make optimizations to optimize scenarios like
ping-pong, so the boot-up performance is very good.Thanks.
Quoted 16 lines
Thanks,
-SiweiQuoted 13 lines
Thanks.
Quoted 11 lines
Quoted 11 lines
Quoted 7 lines
Unless the
device is specially hard wired to some fixed guest setup that users couldn't change, it doesn't seem logical that the device could derive the best or most performant config on driver's behalf. What if the guest wants best latency for its load but the device just blindlessly guess the guest might prefer throughput friendly that it miserably uses latency impacting non-zero default?The device does not want to guess and cannot guess. This patch does not force the device to choose a non-zero value, but relaxes it to allow the device to choose 0 or non-zero, which is very friendly to virtual machines with different performance requirements,
right?
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
I don't understand the friendly part - do you imply your VM users are on kinda fixed wired setup that they cannot change these coalescing parameters after driver is loaded? Can the owner of the VM in control apply certain initial config for the coalescing
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
parameters to the VM image? Or is it the problem of the guest driver that doesn't yet expose coalescing parameters to the end user? Otherwise I would think that guest user should be able to set parameters accordingly that would best fit the specific performance requirement of their own. How the device could even help here? I don't feel there's a lot of value to grant the device or host admin the flexibility to policy the *best* config on guest user's behalf, to be honest. And you seem to admit the fact that the default doesn't
really matter, be it 0 or non-zero.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Could this device side change for the default config regress boot time performance (which may need best latency over throughput)?
Don't make these assumptions, what if the driver needs better
throughput?
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
There's a misconception here: what we think that driver may need in terms on performance does actually reflect what guest user would like to have. Driver cannot read guest user's mind to make the decision,
either.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
In history there was drastic change in the Linux virtio-net driver that ever changed the default disposition for XPS (and RPS as well) from throughput and long-lived connection oriented to concurrency and short-lived oriented, which regressed a lot of existing setups that expects sustained throughput and packet rate after kernel
upgrade.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Although the occurrence of such drastic change for default disposition is not so welcome (that is one of the reasons why I valued consistent initial value and back compatible behavior), I don't see people yelling at virtio spec for less flexibility of offering the default disposition, given that the guest user can override the config any time with their own tooling or script, there's no problem at all for them to just set the corresponding config
back explicitly to what it was before.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
And device/driver has better performance, is that a problem?
Unlikely.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Even for rare case with a hard wired setup, the way to tackle the very problem using device's default is still quite questionable. Usually the mgmt software or network config utility should be equipped with some default value if need be. And we know the guest has the best position to impose the best / most performant config for its own load. What is the issue or use case that this initial config couldn't be applied by the guest mgmt software ahead but has to resort to the device to load some default (which is odd and irrelevant to any guest load), before the interface is
brought up for operation i.e. performing I/O?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Use cases are everywhere, Alibaba Cloud, MLX and all other modern network cards have a default value that is not 0.
You seems to be referencing your own setup basically, and the question is still left answered - why the initial config can't be done through the mgmt software or network config utility within the
guest?
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
(0 is actually a kind of default value)
Quoted 5 lines
Quoted 5 lines
Sorry, my vacation just ended.
Quoted 2 lines
Yes, it is possible. Driver can cache values it sets and never query device with get.
Don't we already have a lot of behaviors to drive queries from
devices?
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
RSS context, device stats.
Quoted 1 line
Before anything is set, driver will report incorrect values.
Devices that are widely supported and supported by good practices should have any initialization value. Just reporting 0 is incorrect value. Although the spec now says so.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
I don't have an aligned view here, sorry. As I recall having 0 as the default is just to keep device started in a state where coalescing is disabled, so it's backward compatible and consistent with a non-coalescing supporting driver - such that it won't yield surprising effect (for e.g. regressed latency) inadvertently after user's getting driver software upgraded. Unlike the other virtio-net features that could 100% improve performance in all aspect, this coalescing feature is more of a performance tuning knob that may improve performance metrics (such as cpu usage or throughput) of one dimension while demoting
the others (such as latency, jitter or connection rate) from the equation.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
That said, there's not a single and fixed set of default config that device could supply which is able to satisfy all kind of guest load. Rather than rely on the device to offer a matching default for driver (which I think it's technically wrong), I'd lean to having guest software or network utility to apply the initial config for the guest, where they should have best knowledge for the specific guest workload than what device could do.
Before this feature, a good device implementation should also support coalescing (of course we don't necessarily assume it has
coalescing).
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Again, I don't doubt the value of supporting coalescing.
Quoted 4 lines
In addition, virtual
machines that tend to favor latency and throughput exist. If the device supported by the manufacturer needs to provide a low-latency virtual machine, please continue to keep the default valueof 0.
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
No, that's not what I was asking. There's no such requirement for any vendor to provide a low-latency or high throughput VM. The more general use case is - the setup for real world workload might just be too complex that end users would prefer low-latency on some virtual NIC or even some specific queues, while the other queues of a virtual NIC, or reset virtual NIC might have very different dispositions. Due to the needs and dynamics of workload scaling up & down, they might have more or less queues or virtual NICs to configure, so these disposition would need to be readjusted at any point of time, for which there's no easy way for device to adapt to easily. The guest user should have best knowledge for the specific guest workload and setup than what device could/should offer.
Quoted 14 lines
Quoted 14 lines
Quoted 7 lines
Quoted 5 lines
What will break as a result? Hard to predict.
Given the patch is big, I am inclined to say it just should use a feature bit.
This doesn't seem to break anything in my opinion, we just told the device that now you can set more values.
Another thing this could break is live migration between devices with varied default. How do you make sure the guest doesn't rely on some default from the source device, while on the destination it just doesn't get the same default coalescingjjj value? To get rid of this side effect the guest would still need to apply the initial config for its own, anyway... Which eventually would render this proposal with arbitrary default rather pointless.
Quoted 22 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
I don't quite understand why this would affect hot migration, the values would be migrated over.
Then I don't see this has been discussed (wouldn't the initial value be part of virtio device state to migrate over?) in the thread or described in the proposed text. And your proposed per-queue coalescing parameters (default plus current value) would have to be described too as part of virtio-net device state, so that
people don't misunderstand your proposal.
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Thanks,
-SiweiQuoted 11 lines
Thanks.
Quoted 9 lines
Thanks,
-SiweiQuoted 7 lines
Using new feature bits does not seem necessary.
Thanks.
Quoted 3 lines
--
MSTI saw the need of this proposal slightly differently in the discussion with Heng in v4.
The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offeringVIRTIO_NET_F_VQ_NOTF_COAL.Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL
2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL
Quoted 8 lines
3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.4. the virtio-net driver even after negotiating
VIRTIO_NET_F_VQ_NOTF_COAL, still kept the its dim disabled. vi->rq[queue].dim_enabled = false by default.5. The virtio net driver enables dim only on user request via ethtool.
So here, just because one is using a new driver and new device, it may get lower performance.
Impossible, older driver / device doesn't have coalescing advertise / enabled, and new device & new driver with coalescing negotiated starts safe with 0 with coalescing disabled effectively, how can it get lower performance?
Quoted 2 lines
Do we agree to above problem description?
This narrative has two problems:
- The term "lower performance" is misleading and inaccurate, the fact or implication of coalescing is that it yields worse latency, causing additional delay and jitter that would hurt quite a lot of real world workload that is latency sensitive. So, rather than say coalescing helps performance, it should clearly tell the truth that the coalescing parameter is no more than a tuning knob that has performance impact/implication, being positive or negative it is very subject to the guest workload. So the default disposition or initial parameters loaded from device just reflects the preference of the host admin/ device owner/ cloud vendor of one's own, for e.g. it is optimized for throughput oriented load versus latency oriented. Be noted, generally end users don't care what coalescing is about, they do not connect performance to "throughput" or "bandwidth" as that in your mind.
- The specific problem or use case is narrow scoped and specifically tailored (e.g. for a hard wired guest setup that is known to the device owner / host admin already), while for some reason the advantage of the proposed solution is overly exaggerated. For one hand, the ethtool interface is very Linux specific that exposed quite a lit lower level device implementation details to normal end users. In case of static and fixed configuration, it definitely requires dedicated knowledge of the user to fine tune for their load. For the other hand, if we are talking about normal users who don't have such knowledge AND don't care so much about performance, then whatever coalescing parameter will just work with them, and they don't really care about the default. However, For those normal users who care about performance but still care about getting the best or well balanced performance (N.B. I say performance in general, not limited to only throughput or latency in one dimension) for their load, what options do they have? IMHO what they really like is to lean on guest/driver/device to auto tune by itself rather than for them to take the deep learning curve to fiddle with the low level device parameters manually by themselves, right?
If you agree with the above, I guess we could proceed with unlocking the other good options, as listed below. To me, the question I got from the above problem statement but still left unanswered is that, what makes the cloud vendor think that they could have better posture to decide certain default values for guest users' behalf? Why it is a MUST to do so for every cloud vendor in general?
Thanks,
-Siwei
Quoted 16 lines
If no, I likely missed something in the long discussion and need to read all of it. :(
If yes, few solutions are:
1. A user intervention is must to avoid this regression. The user in the guest VM must enable DIM if the device supports it. Very hard to do this plumbing.2. A net driver enables the DIM by default as long as
VIRTIO_NET_F_VQ_NOTF_COALis negotiated. (without ethtool setting)3. A device relaxes the limitation and continue to apply the coalescing, until driver overrides it.
In my humble opinion, Heng is solving it using option #3, that tends to work with existing and future drivers who may/may not enable the DIM.
Quoted 1 line · On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu <[email protected]> wrote:
On 6/24/2024 10:56 PM, Parav Pandit wrote:
[..]
Quoted 8 lines
Quoted 8 lines
I saw the need of this proposal slightly differently in the discussion with Heng in v4.
The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offeringVIRTIO_NET_F_VQ_NOTF_COAL.Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL
2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
I have certainly understood this differently. WhenVIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).
If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed to/MUST do the coalescing according to the parameters as described by the virtio spec.
Michael, Jason: Can you guys weigh in on this?
Quoted 11 lines
Quoted 7 lines
4. the virtio-net driver even after negotiating
VIRTIO_NET_F_VQ_NOTF_COAL, still kept the its dim disabled. vi->rq[queue].dim_enabled = false by default.5. The virtio net driver enables dim only on user request via ethtool.
So here, just because one is using a new driver and new device, it may get lower performance.
Impossible, older driver / device doesn't have coalescing advertise / enabled, and new device & new driver with coalescing negotiated starts safe with 0 with coalescing disabled effectively, how can it get lower performance?
Si-Wei and Parav, I think it is the other way around. Without the change proposed here, with no user/management software intervention assumed, we are guaranteed to not see any performance regression but also no performance benefit because of the coalescing mechanism, because with zeros as default it is essentially dormant.
With the change proposed here however, we open up the possibility to see either a performance improvement or a performance degradation depending on the defaults chosen by the device and the workload (sill assuming new device & new driver & no intervention by management software).
Quoted 5 lines
Quoted 3 lines
Do we agree to above problem description?
This narrative has two problems:
Quoted 14 lines
- The term "lower performance" is misleading and inaccurate, the fact or implication of coalescing is that it yields worse latency, causing additional delay and jitter that would hurt quite a lot of real world workload that is latency sensitive. So, rather than say coalescing helps performance, it should clearly tell the truth that the coalescing parameter is no more than a tuning knob that has performance impact/implication, being positive or negative it is very subject to the guest workload. So the default disposition or initial parameters loaded from device just reflects the preference of the host admin/ device owner/ cloud vendor of one's own, for e.g. it is optimized for throughput oriented load versus latency oriented. Be noted, generally end users don't care what coalescing is about, they do not connect performance to "throughput" or "bandwidth" as that in your mind.
Si-Wei, I sympathies with your argument but I'm willing to tolerate this change nevertheless.
Yes the device and via the device whoever controls these parameters can shoot the guest in the foot, if the guest admin is not going out of his way to do something about controlling coalescing.
But via the same mechanism we could end up in a better place as well.
Because I assume that the device and its "admin" are at least trying to do good, I'm not straight out against this proposal, as long nothing wonky happens with the code which is already out there, in a sense of a split brain scenario where Linux thinks those values are 0 but they aren't. Unfortunately can't tell you how we stand on this.
[..]
Quoted 4 lines
Quoted 4 lines
If no, I likely missed something in the long discussion and need to read all of it. :(
If yes, few solutions are:
1. A user intervention is must to avoid this regression. The user in the guest VM must enable DIM if the device supports it.
I don't think burdening up the users with avoiding regressions is a good idea.
Regards,
Halil
Quoted 13 lines
Quoted 12 lines
Very hard to do this plumbing.
2. A net driver enables the DIM by default as long as
VIRTIO_NET_F_VQ_NOTF_COALis negotiated. (without ethtool setting)3. A device relaxes the limitation and continue to apply the coalescing, until driver overrides it.
In my humble opinion, Heng is solving it using option #3, that tends to work with existing and future drivers who may/may not enable the DIM.
Hi Halil,
Quoted 19 lines
From: Halil Pasic <[email protected]>
Sent: Thursday, June 27, 2024 4:08 PMQuoted 1 line · On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu <[email protected]> wrote:
On 6/24/2024 10:56 PM, Parav Pandit wrote:
[..]
Quoted 1 line
Quoted 1 line
I saw the need of this proposal slightly differently in the discussion with
Heng in v4.
Quoted 1 line
Quoted 1 line
The way I understood is, proposed relaxation enables below Linux driver
flow to work as equally as without device offering
VIRTIO_NET_F_VQ_NOTF_COAL.Quoted 6 lines
Quoted 6 lines
Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL
Quoted 9 lines
Quoted 1 line
Quoted 1 line
3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is notapplying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
I have certainly understood this differently. When
VIRTIO_NET_F_VQ_NOTF_COALis not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).
If device does not do interrupt coalescing, it will be worse because of interrupt per packet.
Quoted 4 lines
If
VIRTIO_NET_F_VQ_NOTF_COALis negotiated the device is supposed to/MUST do the coalescing according to the parameters as described by the virtio spec.
Yes.
Quoted 16 lines
Michael, Jason: Can you guys weigh in on this?
Quoted 2 lines
Quoted 2 lines
4. the virtio-net driver even after negotiating
VIRTIO_NET_F_VQ_NOTF_COAL, still kept the its dim disabled.Quoted 5 lines
Quoted 5 lines
vi->rq[queue].dim_enabled = false by default.
5. The virtio net driver enables dim only on user request via ethtool.
So here, just because one is using a new driver and new device, it may get
lower performance.
Quoted 4 lines
Impossible, older driver / device doesn't have coalescing advertise / enabled, and new device & new driver with coalescing negotiated starts safe with 0 with coalescing disabled effectively, how can it get lower performance?
Quoted 6 lines
Si-Wei and Parav, I think it is the other way around. Without the change proposed here, with no user/management software intervention assumed, we are guaranteed to not see any performance regression but also no performance benefit because of the coalescing mechanism, because with zeros as default it is essentially dormant.
I will avoid quoting the term "performance regression".
I believe there will be behavior regression.
For example, devices do apply notification suppression today when VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated. Not doing so will not allow virtio-net to come this far.
By setting them to zero, device stops applying what it was doing until now.
And it results in "behavior regression".
One can argue that now the latency is best because very first packet will trigger a notification.
The proposed change in my humble view is allowing the device to restore its behavior when driver has NOT configured notification params and yet negotiated VIRTIO_NET_F_VQ_NOTF_COAL.
Quoted 5 lines
With the change proposed here however, we open up the possibility to see either a performance improvement or a performance degradation depending on the defaults chosen by the device and the workload (sill assuming new device & new driver & no intervention by management software).
Exactly. I agree with you.
It seems fine because it restores the behavior. Whether that behavior is good/bad/average - it is upto the device implementation.
Quoted 22 lines
Quoted 6 lines
Quoted 4 lines
Do we agree to above problem description?
This narrative has two problems:
Quoted 13 lines
- The term "lower performance" is misleading and inaccurate, the fact or implication of coalescing is that it yields worse latency, causing additional delay and jitter that would hurt quite a lot of real world workload that is latency sensitive. So, rather than say coalescing helps performance, it should clearly tell the truth that the coalescing parameter is no more than a tuning knob that has performance impact/implication, being positive or negative it is very subject to the guest workload. So the default disposition or initial parameters loaded from device just reflects the preference of the host admin/ device owner/ cloud vendor of one's own, for e.g. it is optimized for throughput oriented load versus latency oriented. Be noted, generally end users don't care what coalescing is about, they do not connect performance to "throughput" or "bandwidth" as that in your
mind.
Quoted 1 line
Quoted 7 lines
Si-Wei, I sympathies with your argument but I'm willing to tolerate this change nevertheless.
Yes the device and via the device whoever controls these parameters can shoot the guest in the foot, if the guest admin is not going out of his way to do something about controlling coalescing.
Right.
Quoted 2 lines
But via the same mechanism we could end up in a better place as well.
Right if not better, at least it brings to the same place as where "this feature is not offered".
In the second example for txq, current Linux dim does not have netdim for TX. And negotiating this feature (and not doing actual TXQ level moderation) results in zero tx moderation.
All of these would have been fine, if the Linux net driver would have implemented and enabled netdim whenever the feature is negotiated.
Driver negotiating and driver not controlling is the main issue. The device is eagerly hoping that driver will control, but it doesn't.
The proposed change overcomes this limitation, that until driver controls, it let it control the moderation. Once driver takes over (by issuing first command), device stops applying its own logic.
Quoted 7 lines
Because I assume that the device and its "admin" are at least trying to do good, I'm not straight out against this proposal, as long nothing wonky happens with the code which is already out there, in a sense of a split brain scenario where Linux thinks those values are 0 but they aren't. Unfortunately can't tell you how we stand on this.
[..]
Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
If no, I likely missed something in the long discussion and need to read all of it. :(
If yes, few solutions are:
1. A user intervention is must to avoid this regression. The user in the guestVM must enable DIM if the device supports it.
I don't think burdening up the users with avoiding regressions is a good idea.
The user is the vnet driver who doesnt even has netdim implemented for the TX.
So device is in dark, should it offer this feature or not?
If it offers (without the proposed change), there is behavior regression until the driver fix is done.
Hence, I agree to Halil's point.
We should relax the requirement to restore the behavior. New feature bit won't help the behavior restoration..
Quoted 19 lines
Regards,
HalilQuoted 3 lines
Quoted 3 lines
Very hard to do this plumbing.
2. A net driver enables the DIM by default as long as
VIRTIO_NET_F_VQ_NOTF_COALis negotiated.Quoted 3 lines
Quoted 3 lines
(without ethtool setting)
3. A device relaxes the limitation and continue to apply the coalescing, until
driver overrides it.
Quoted 2 lines
Quoted 2 lines
In my humble opinion, Heng is solving it using option #3, that tends to
work with existing and future drivers who may/may not enable the DIM.
Quoted 5 lines
Quoted 4 lines
Quoted 14 lines · Halil Pasic
Quoted 1 line · On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu <[email protected]> wrote:
On 6/24/2024 10:56 PM, Parav Pandit wrote:
[..]
Quoted 8 lines
Quoted 8 lines
I saw the need of this proposal slightly differently in the discussion with Heng in v4.
The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offeringVIRTIO_NET_F_VQ_NOTF_COAL.Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL
2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
Quoted 8 lines
I have certainly understood this differently. When
VIRTIO_NET_F_VQ_NOTF_COALis not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).If
VIRTIO_NET_F_VQ_NOTF_COALis negotiated the device is supposed to/MUST do the coalescing according to the parameters as described by the virtio spec.Michael, Jason: Can you guys weigh in on this?
I still don't understand why this change is needed. We have this text:
The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
and
The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
the device MAY generate notifications more or less frequently than specified.
So with no spec changes, devices already can do what this patch says they can do - send notifications less frequently.
Re-reading this spec text, maybe the confusion is that it mentions set commands specifically? And it's also stuck in the middle where it's easy to miss.
So it would seem that the following should be sufficient, and it looks like a small clarification we could just apply and include in the vote for the csd. What do you guys think?
----
device-types/net/description.tex +4 -3
@@ -1711,8 +1711,6 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi for an enabled transmit/receive virtqueue whose index is \field{vq_index}. \end{enumerate} -The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class. - If coalescing parameters are being set, the device applies the last coalescing parameters set for a virtqueue, regardless of the command used to set the parameters. Use the following command sequence with two pairs of virtqueues as an example:
@@ -1726,6 +1724,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5. \end{itemize} +The device can generate notifications more or less frequently +than specified by the coalescing parameters. + \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
@@ -1798,7 +1799,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0. -The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort: +The behavior of the device in response to specific coalescing parameters is best-effort: the device MAY generate notifications more or less frequently than specified. A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
Signature · 1 line
MST
Hi Michael,
Quoted 107 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, June 27, 2024 6:05 PMQuoted 6 lines · Halil Pasic
Quoted 1 line · On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu <[email protected]> wrote:
On 6/24/2024 10:56 PM, Parav Pandit wrote:
[..]
Quoted 1 line
Quoted 1 line
I saw the need of this proposal slightly differently in the discussion with
Heng in v4.
Quoted 1 line
Quoted 1 line
Quoted 1 line
The way I understood is, proposed relaxation enables below Linux driver
flow to work as equally as without device offering
VIRTIO_NET_F_VQ_NOTF_COAL.Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is notapplying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
Quoted 3 lines
I have certainly understood this differently. When
VIRTIO_NET_F_VQ_NOTF_COALis not negotiated then the device is notsupposed/allowed to do any interrupt coalescing (notification suppression may still apply).
Quoted 6 lines
If
VIRTIO_NET_F_VQ_NOTF_COALis negotiated the device is supposed to/MUST do the coalescing according to the parameters as described by the virtio spec.Michael, Jason: Can you guys weigh in on this?
I still don't understand why this change is needed. We have this text:
The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COAL
class.and
The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:
the device MAY generate notifications more or less frequently than
specified.So with no spec changes, devices already can do what this patch says they can do - send notifications less frequently.
Re-reading this spec text, maybe the confusion is that it mentions set commands specifically? And it's also stuck in the middle where it's easy to miss.
So it would seem that the following should be sufficient, and it looks like a small clarification we could just apply and include in the vote for the csd. What do you guys think?
Signed-off-by Michael S. Tsirkin <[email protected]>----
device- renamed from device-types/net/description.tex +0
-0types/net/description.tex
index 76585b0..d6788df 100644device-types/net/description.tex +0
-0@@ -1711,8 +1711,6 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi
for an enabled transmit/receive virtqueue whose index is
\field{vq_index}.
\end{enumerate}-The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass.
-
If coalescing parameters are being set, the device applies the last coalescing
parameters set for a virtqueue, regardless of the command used to set the parameters. Use the following command sequence with two pairs of virtqueues as an example:device-types/net/description.tex +0
-0@@ -1726,6 +1724,9 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi \item Command6:
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETwith \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5.
\end{itemize}device-types/net/description.tex +3
-0+The device can generate notifications more or less frequently than +specified by the coalescing parameters. + \subparagraph{Operation}\label{sec:Device Types / Network Device / DeviceOperation / Control Virtqueue / Notifications Coalescing / Operation}
The device sends a used buffer notification once the notification conditions
are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.device-types/net/description.tex +0
-0@@ -1798,7 +1799,7 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue to those configured through the
VIRTIO_NET_CTRL_NOTF_COAL_RX_SETcommand, or, if the driver did not set any RX coalescing parameters, to 0.
Quoted 9 lines
-The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:
+The behavior of the device in response to specific coalescing parameters is best-effort:
the device MAY generate notifications more or less frequently than specified.A device SHOULD NOT send used buffer notifications to the driver if the
notifications are suppressed, even if the notification conditions are met.
Your changes above are good and they are applicable when the specific coalescing parameters are set by the driver.
The issue that is being resolved is, when these parameters are NOT set by the driver, when feature is negotiated. What should device do?
The parameters are not specified...
The spec says " Upon reset, a device MUST initialize all coalescing parameters to 0.".
And Heng attempt is to clarify that by adding a changing above line something like,
"when driver does not set coalescing params, but have negotiated the feature, device may generate notifications more or less frequently than specified".
Quoted 115 lines · Parav Pandit
Hi Michael,
Quoted 113 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, June 27, 2024 6:05 PMQuoted 6 lines · Halil Pasic
Quoted 1 line · On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu <[email protected]> wrote:
On 6/24/2024 10:56 PM, Parav Pandit wrote:
[..]
Quoted 1 line
Quoted 1 line
I saw the need of this proposal slightly differently in the discussion with
Heng in v4.
Quoted 1 line
Quoted 1 line
Quoted 1 line
The way I understood is, proposed relaxation enables below Linux driver
flow to work as equally as without device offering
VIRTIO_NET_F_VQ_NOTF_COAL.Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is notapplying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
Quoted 3 lines
I have certainly understood this differently. When
VIRTIO_NET_F_VQ_NOTF_COALis not negotiated then the device is notsupposed/allowed to do any interrupt coalescing (notification suppression may still apply).
Quoted 6 lines
If
VIRTIO_NET_F_VQ_NOTF_COALis negotiated the device is supposed to/MUST do the coalescing according to the parameters as described by the virtio spec.Michael, Jason: Can you guys weigh in on this?
I still don't understand why this change is needed. We have this text:
The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COAL
class.and
The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:
the device MAY generate notifications more or less frequently than
specified.So with no spec changes, devices already can do what this patch says they can do - send notifications less frequently.
Re-reading this spec text, maybe the confusion is that it mentions set commands specifically? And it's also stuck in the middle where it's easy to miss.
So it would seem that the following should be sufficient, and it looks like a small clarification we could just apply and include in the vote for the csd. What do you guys think?
Signed-off-by Michael S. Tsirkin <[email protected]>----
device- renamed from device-types/net/description.tex +0
-0types/net/description.tex
index 76585b0..d6788df 100644device-types/net/description.tex +0
-0@@ -1711,8 +1711,6 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi
for an enabled transmit/receive virtqueue whose index is
\field{vq_index}.
\end{enumerate}-The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass.
-
If coalescing parameters are being set, the device applies the last coalescing
parameters set for a virtqueue, regardless of the command used to set the parameters. Use the following command sequence with two pairs of virtqueues as an example:device-types/net/description.tex +0
-0@@ -1726,6 +1724,9 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi \item Command6:
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETwith \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5.
\end{itemize}device-types/net/description.tex +3
-0+The device can generate notifications more or less frequently than +specified by the coalescing parameters. + \subparagraph{Operation}\label{sec:Device Types / Network Device / DeviceOperation / Control Virtqueue / Notifications Coalescing / Operation}
The device sends a used buffer notification once the notification conditions
are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.device-types/net/description.tex +0
-0@@ -1798,7 +1799,7 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue to those configured through the
VIRTIO_NET_CTRL_NOTF_COAL_RX_SETcommand, or, if the driver did not set any RX coalescing parameters, to 0.-The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:
+The behavior of the device in response to specific coalescing parameters is best-effort:
the device MAY generate notifications more or less frequently than specified.
Quoted 5 lines
Quoted 3 lines
A device SHOULD NOT send used buffer notifications to the driver if the
notifications are suppressed, even if the notification conditions are met.Your changes above are good and they are applicable when the specific coalescing parameters are set by the driver.
What makes you say this?
No they apply universally whenever parameters are specified by the spec:
whether parameters are set by driver or by device.
Is that unclear somehow?
I suspect you are just reading the wrong part of the patch ...
Quoted 3 lines
The issue that is being resolved is, when these parameters are NOT set by the driver, when feature is negotiated. What should device do?
The parameters are not specified...
They are specified - set to 0.
Quoted 1 line
The spec says " Upon reset, a device MUST initialize all coalescing parameters to 0.".
yes. and device can then coalesce if it wants to.
Quoted 3 lines
And Heng attempt is to clarify that by adding a changing above line something like,
"when driver does not set coalescing params, but have negotiated the feature, device may generate notifications more or less frequently than specified".
Maybe that is the intention, but the proposed text is instead saying the parameters themselves are !0. Which is what triggered the whole mess with a misvote: commit log says it's a clarification when it is actually a behaviour change.
Signature · 1 line
MST
Quoted 128 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, June 27, 2024 6:23 PMQuoted 45 lines · Parav Pandit
Hi Michael,
Quoted 43 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, June 27, 2024 6:05 PMQuoted 7 lines · Halil Pasic
Quoted 1 line · On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu <[email protected]> wrote:
On 6/24/2024 10:56 PM, Parav Pandit wrote:
[..]
Quoted 2 lines
Quoted 2 lines
I saw the need of this proposal slightly differently in the discussion with
Heng in v4.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The way I understood is, proposed relaxation enables below Linux driver
flow to work as equally as without device offering
VIRTIO_NET_F_VQ_NOTF_COAL.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is notapplying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
Quoted 3 lines
I have certainly understood this differently. When
VIRTIO_NET_F_VQ_NOTF_COALis not negotiated then the device is notsupposed/allowed to do any interrupt coalescing (notification suppression may still apply).
Quoted 6 lines
If
VIRTIO_NET_F_VQ_NOTF_COALis negotiated the device is supposed to/MUST do the coalescing according to the parameters as described by the virtio spec.Michael, Jason: Can you guys weigh in on this?
I still don't understand why this change is needed. We have this text:
The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass.
Quoted 35 lines
Quoted 35 lines
and
The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:
the device MAY generate notifications more or less frequently than
specified.So with no spec changes, devices already can do what this patch says they can do - send notifications less frequently.
Re-reading this spec text, maybe the confusion is that it mentions set commands specifically? And it's also stuck in the middle where it's easy to miss.
So it would seem that the following should be sufficient, and it looks like a small clarification we could just apply and include in the vote for the csd. What do you guys think?
Signed-off-by Michael S. Tsirkin <[email protected]>----
device- renamed from device-types/net/description.tex +0
-0types/net/description.tex index 76585b0..d6788df 100644
device-types/net/description.tex +0
-0@@ -1711,8 +1711,6 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi
for an enabled
transmit/receive virtqueue whose index is \field{vq_index}.
\end{enumerate}Quoted 9 lines
Quoted 9 lines
-The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass. -
If coalescing parameters are being set, the device applies the last
coalescing parameters set for a virtqueue, regardless of the command used to set the parameters. Use the following command sequence with two pairs of virtqueues as an example:device-types/net/description.tex +0
-0@@ -1726,6 +1724,9 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi \item
Command6:
Quoted 22 lines
Quoted 22 lines
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETwith \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5.
\end{itemize}device-types/net/description.tex +3
-0+The device can generate notifications more or less frequently than +specified by the coalescing parameters. + \subparagraph{Operation}\label{sec:Device Types / Network Device /Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
The device sends a used buffer notification once the notification
conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.device-types/net/description.tex +0
-0@@ -1798,7 +1799,7 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue to those configured through the
VIRTIO_NET_CTRL_NOTF_COAL_RX_SETcommand, or, if the driver did not set any RX coalescing parameters, to 0.Quoted 6 lines
Quoted 6 lines
-The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:device-types/net/description.tex +2
-0+The behavior of the device in response to specific coalescing +parameters isbest-effort:
the device MAY generate notifications more or less frequently thanspecified.
Quoted 3 lines
Quoted 3 lines
A device SHOULD NOT send used buffer notifications to the driver if
the notifications are suppressed, even if the notification conditions are
Quoted 7 lines
met.
Quoted 3 lines
Quoted 1 line
Your changes above are good and they are applicable when the specific
coalescing parameters are set by the driver.
What makes you say this?
Because I was reading this in the CVQ section of the spec where CVQ commands are setting it.
Quoted 2 lines
No they apply universally whenever parameters are specified by the spec:
whether parameters are set by driver or by device.
I see. Upon reset they are set to 0 in the below text, means no coalescing, means quick interrupts, right?
Quoted 9 lines
Is that unclear somehow?
I suspect you are just reading the wrong part of the patch ...Quoted 1 line
The issue that is being resolved is, when these parameters are NOT set by the
driver, when feature is negotiated.
Quoted 2 lines
What should device do?
The parameters are not specified...
Quoted 7 lines
They are specified - set to 0.
Quoted 1 line
The spec says " Upon reset, a device MUST initialize all coalescing parameters
to 0.".
yes. and device can then coalesce if it wants to.
You mean 0 means coalescing is disabled (and not quick notifications) ? And hence, it is device chosen value?
If so, that interpretation would work too.
Quoted 10 lines
Quoted 4 lines
And Heng attempt is to clarify that by adding a changing above line something like,
"when driver does not set coalescing params, but have negotiated the
feature, device may generate notifications more or less frequently than specified".
Maybe that is the intention, but the proposed text is instead saying the parameters themselves are !0.
A clarification like above would be better or 0 means device chosen parameters would also work fine.
Quoted 4 lines
Which is what triggered the whole mess with a
misvote: commit log says it's a clarification when it is actually a behaviour change.
It is a behavior change to fix/restore.
But if you suggest: 0 means device chosen value, that will work too to restore current regression.
Quoted 128 lines · Parav Pandit
Quoted 128 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, June 27, 2024 6:23 PMQuoted 45 lines · Parav Pandit
Hi Michael,
Quoted 43 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, June 27, 2024 6:05 PMQuoted 7 lines · Halil Pasic
Quoted 1 line · On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu <[email protected]> wrote:
On 6/24/2024 10:56 PM, Parav Pandit wrote:
[..]
Quoted 2 lines
Quoted 2 lines
I saw the need of this proposal slightly differently in the discussion with
Heng in v4.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The way I understood is, proposed relaxation enables below Linux driver
flow to work as equally as without device offering
VIRTIO_NET_F_VQ_NOTF_COAL.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is notapplying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
Quoted 3 lines
I have certainly understood this differently. When
VIRTIO_NET_F_VQ_NOTF_COALis not negotiated then the device is notsupposed/allowed to do any interrupt coalescing (notification suppression may still apply).
Quoted 6 lines
If
VIRTIO_NET_F_VQ_NOTF_COALis negotiated the device is supposed to/MUST do the coalescing according to the parameters as described by the virtio spec.Michael, Jason: Can you guys weigh in on this?
I still don't understand why this change is needed. We have this text:
The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass.
Quoted 35 lines
Quoted 35 lines
and
The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:
the device MAY generate notifications more or less frequently than
specified.So with no spec changes, devices already can do what this patch says they can do - send notifications less frequently.
Re-reading this spec text, maybe the confusion is that it mentions set commands specifically? And it's also stuck in the middle where it's easy to miss.
So it would seem that the following should be sufficient, and it looks like a small clarification we could just apply and include in the vote for the csd. What do you guys think?
Signed-off-by Michael S. Tsirkin <[email protected]>----
device- renamed from device-types/net/description.tex +0
-0types/net/description.tex index 76585b0..d6788df 100644
device-types/net/description.tex +0
-0@@ -1711,8 +1711,6 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi
for an enabled
transmit/receive virtqueue whose index is \field{vq_index}.
\end{enumerate}Quoted 9 lines
Quoted 9 lines
-The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass. -
If coalescing parameters are being set, the device applies the last
coalescing parameters set for a virtqueue, regardless of the command used to set the parameters. Use the following command sequence with two pairs of virtqueues as an example:device-types/net/description.tex +0
-0@@ -1726,6 +1724,9 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi \item
Command6:
Quoted 22 lines
Quoted 22 lines
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETwith \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5.
\end{itemize}device-types/net/description.tex +3
-0+The device can generate notifications more or less frequently than +specified by the coalescing parameters. + \subparagraph{Operation}\label{sec:Device Types / Network Device /Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
The device sends a used buffer notification once the notification
conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.device-types/net/description.tex +0
-0@@ -1798,7 +1799,7 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue to those configured through the
VIRTIO_NET_CTRL_NOTF_COAL_RX_SETcommand, or, if the driver did not set any RX coalescing parameters, to 0.Quoted 6 lines
Quoted 6 lines
-The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:device-types/net/description.tex +2
-0+The behavior of the device in response to specific coalescing +parameters isbest-effort:
the device MAY generate notifications more or less frequently thanspecified.
Quoted 3 lines
Quoted 3 lines
A device SHOULD NOT send used buffer notifications to the driver if
the notifications are suppressed, even if the notification conditions are
Quoted 8 lines
Quoted 7 lines
met.
Quoted 3 lines
Quoted 1 line
Your changes above are good and they are applicable when the specific
coalescing parameters are set by the driver.
What makes you say this?
Because I was reading this in the CVQ section of the spec where CVQ commands are setting it.
So? Reset also sets it.
Quoted 3 lines
Quoted 2 lines
No they apply universally whenever parameters are specified by the spec:
whether parameters are set by driver or by device.I see. Upon reset they are set to 0 in the below text, means no coalescing, means quick interrupts, right?
But device can make them as slow as it likes, spec says so.
Quoted 11 lines
Quoted 11 lines
Is that unclear somehow?
I suspect you are just reading the wrong part of the patch ...Quoted 1 line
The issue that is being resolved is, when these parameters are NOT set by the
driver, when feature is negotiated.
Quoted 2 lines
What should device do?
The parameters are not specified...They are specified - set to 0.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
The spec says " Upon reset, a device MUST initialize all coalescing parameters
to 0.".
yes. and device can then coalesce if it wants to.
You mean 0 means coalescing is disabled (and not quick notifications) ? And hence, it is device chosen value?
If so, that interpretation would work too.
No 0 means send notification after any # of packets.
Quoted 19 lines
Quoted 10 lines
Quoted 4 lines
And Heng attempt is to clarify that by adding a changing above line something like,
"when driver does not set coalescing params, but have negotiated the
feature, device may generate notifications more or less frequently than specified".
Maybe that is the intention, but the proposed text is instead saying the parameters themselves are !0.
A clarification like above would be better or 0 means device chosen parameters would also work fine.
Quoted 4 lines
Which is what triggered the whole mess with a
misvote: commit log says it's a clarification when it is actually a behaviour change.It is a behavior change to fix/restore.
But if you suggest: 0 means device chosen value, that will work too to restore current regression.
Quoted 123 lines · Michael S. Tsirkin
Quoted 111 lines · Parav Pandit
Hi Michael,
Quoted 109 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, June 27, 2024 6:05 PMQuoted 6 lines · Halil Pasic
Quoted 1 line · On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu <[email protected]> wrote:
On 6/24/2024 10:56 PM, Parav Pandit wrote:
[..]
Quoted 1 line
Quoted 1 line
I saw the need of this proposal slightly differently in the discussion with
Heng in v4.
Quoted 1 line
Quoted 1 line
Quoted 1 line
The way I understood is, proposed relaxation enables below Linux driver
flow to work as equally as without device offering
VIRTIO_NET_F_VQ_NOTF_COAL.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is notapplying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
Quoted 2 lines
I have certainly understood this differently. When
VIRTIO_NET_F_VQ_NOTF_COALis not negotiated then the device is notsupposed/allowed to do any interrupt coalescing (notification suppression may still apply).
Quoted 5 lines
If
VIRTIO_NET_F_VQ_NOTF_COALis negotiated the device is supposed to/MUST do the coalescing according to the parameters as described by the virtio spec.Michael, Jason: Can you guys weigh in on this?
I still don't understand why this change is needed. We have this text:
The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COAL
class.and
The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:
the device MAY generate notifications more or less frequently than
specified.So with no spec changes, devices already can do what this patch says they can do - send notifications less frequently.
Re-reading this spec text, maybe the confusion is that it mentions set commands specifically? And it's also stuck in the middle where it's easy to miss.
So it would seem that the following should be sufficient, and it looks like a small clarification we could just apply and include in the vote for the csd. What do you guys think?
Signed-off-by Michael S. Tsirkin <[email protected]>----
device- renamed from device-types/net/description.tex +0
-0types/net/description.tex
index 76585b0..d6788df 100644device-types/net/description.tex +0
-0@@ -1711,8 +1711,6 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi
for an enabled transmit/receive virtqueue whose index is
\field{vq_index}.
\end{enumerate}-The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass.
-
If coalescing parameters are being set, the device applies the last coalescing
parameters set for a virtqueue, regardless of the command used to set the parameters. Use the following command sequence with two pairs of virtqueues as an example:device-types/net/description.tex +0
-0@@ -1726,6 +1724,9 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi \item Command6:
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETwith \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5.
\end{itemize}device-types/net/description.tex +3
-0+The device can generate notifications more or less frequently than +specified by the coalescing parameters. + \subparagraph{Operation}\label{sec:Device Types / Network Device / DeviceOperation / Control Virtqueue / Notifications Coalescing / Operation}
The device sends a used buffer notification once the notification conditions
are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.device-types/net/description.tex +0
-0@@ -1798,7 +1799,7 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue to those configured through the
VIRTIO_NET_CTRL_NOTF_COAL_RX_SETcommand, or, if the driver did not set any RX coalescing parameters, to 0.-The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:
+The behavior of the device in response to specific coalescing parameters is best-effort:
the device MAY generate notifications more or less frequently than specified.Quoted 4 lines
Quoted 3 lines
A device SHOULD NOT send used buffer notifications to the driver if the
notifications are suppressed, even if the notification conditions are met.Your changes above are good and they are applicable when the specific coalescing parameters are set by the driver.
What makes you say this?
No they apply universally whenever parameters are specified by the spec:
whether parameters are set by driver or by device.
Is that unclear somehow?
I suspect you are just reading the wrong part of the patch ...
Quoted 7 lines
Quoted 3 lines
The issue that is being resolved is, when these parameters are NOT set by the driver, when feature is negotiated. What should device do?
The parameters are not specified...They are specified - set to 0.
Quoted 1 line
The spec says " Upon reset, a device MUST initialize all coalescing parameters to 0.".
yes. and device can then coalesce if it wants to.
Well, maybe coalesce here in this context is a little confusing but I guess I got what you're saying: setting 0 effectively disables the driver's control of coalescing, right. And device may have its own way to suppress notification...
Quoted 10 lines
Quoted 4 lines
And Heng attempt is to clarify that by adding a changing above line something like,
"when driver does not set coalescing params, but have negotiated the feature, device may generate notifications more or less frequently than specified".
Maybe that is the intention, but the proposed text is instead saying the parameters themselves are !0. Which is what triggered the whole mess with a misvote: commit log says it's a clarification when it is actually a behaviour change.
+1.
Whatever beautified wording there was, the proposed spec fundamentally leads to a behavior change.
-Siwei
Quoted 28 lines · Michael S. Tsirkin
Quoted 13 lines · Halil Pasic
Quoted 1 line · On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu <[email protected]> wrote:
On 6/24/2024 10:56 PM, Parav Pandit wrote:
[..]
Quoted 8 lines
Quoted 8 lines
I saw the need of this proposal slightly differently in the discussion with Heng in v4.
The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offeringVIRTIO_NET_F_VQ_NOTF_COAL.Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL
2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.Quoted 8 lines
I have certainly understood this differently. When
VIRTIO_NET_F_VQ_NOTF_COALis not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).If
VIRTIO_NET_F_VQ_NOTF_COALis negotiated the device is supposed to/MUST do the coalescing according to the parameters as described by the virtio spec.Michael, Jason: Can you guys weigh in on this?
I still don't understand why this change is needed. We have this text:
The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass.
Quoted 7 lines
and
The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:
the device MAY generate notifications more or less frequently than specified.So with no spec changes, devices already can do what this patch says they can do - send notifications less frequently.
Right. If I still recall it correctly, the above text was to accommodate the interaction with existing used buffer notification mechanism, F_EVENT_IDX or whatnot. When coalescing is in place and effective, interrupt delivery is a collaborative consideration based on the coalescing parameters specified *AND * event index / NO_INTERRUPT flag as well.
Quoted 8 lines
Re-reading this spec text, maybe the confusion is that it mentions set commands specifically? And it's also stuck in the middle where it's easy to miss.
So it would seem that the following should be sufficient, and it looks like a small clarification we could just apply and include in the vote for the csd. What do you guys think?
The revised text below looks good to me.
Thanks,
-Siwei
Quoted 40 lines
Signed-off-by Michael S. Tsirkin <[email protected]>----
device-types/net/description.tex +4
-3@@ -1711,8 +1711,6 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi for an enabled transmit/receive virtqueue whose index is \field{vq_index}. \end{enumerate} -The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class. - If coalescing parameters are being set, the device applies the last coalescing parameters set for a virtqueue, regardless of the command used to set the parameters. Use the following command sequence with two pairs of virtqueues as an example:@@ -1726,6 +1724,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5. \end{itemize} +The device can generate notifications more or less frequently +than specified by the coalescing parameters. + \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.@@ -1798,7 +1799,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0. -The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort: +The behavior of the device in response to specific coalescing parameters is best-effort: the device MAY generate notifications more or less frequently than specified. A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
Quoted 36 lines · Si-Wei Liu
Quoted 33 lines · Michael S. Tsirkin
Quoted 13 lines · Halil Pasic
Quoted 1 line · On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu <[email protected]> wrote:
On 6/24/2024 10:56 PM, Parav Pandit wrote:
[..]
Quoted 8 lines
Quoted 8 lines
I saw the need of this proposal slightly differently in the discussion with Heng in v4.
The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offeringVIRTIO_NET_F_VQ_NOTF_COAL.Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL
2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.Quoted 8 lines
I have certainly understood this differently. When
VIRTIO_NET_F_VQ_NOTF_COALis not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).If
VIRTIO_NET_F_VQ_NOTF_COALis negotiated the device is supposed to/MUST do the coalescing according to the parameters as described by the virtio spec.Michael, Jason: Can you guys weigh in on this?
I still don't understand why this change is needed. We have this text:
The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass.and
The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:
the device MAY generate notifications more or less frequently than specified.
Quoted 8 lines
Quoted 2 lines
So with no spec changes, devices already can do what this patch says they can do - send notifications less frequently.
Right. If I still recall it correctly, the above text was to accommodate the interaction with existing used buffer notification mechanism,
F_EVENT_IDXor whatnot.
When coalescing is in place and effective, interrupt delivery is a collaborative consideration based on the coalescing parameters specified *AND * event index /NO_INTERRUPTflag as well.
No, not really.
For that we have the next sentence:
A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
And of course event index never makes you send interrupts more frequently.
Quoted 48 lines
Quoted 8 lines
Re-reading this spec text, maybe the confusion is that it mentions set commands specifically? And it's also stuck in the middle where it's easy to miss.
So it would seem that the following should be sufficient, and it looks like a small clarification we could just apply and include in the vote for the csd. What do you guys think?
The revised text below looks good to me.
Thanks,
-SiweiQuoted 35 lines
Signed-off-by Michael S. Tsirkin <[email protected]>----
device-types/net/description.tex +4
-3@@ -1711,8 +1711,6 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi for an enabled transmit/receive virtqueue whose index is \field{vq_index}. \end{enumerate} -The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class. - If coalescing parameters are being set, the device applies the last coalescing parameters set for a virtqueue, regardless of the command used to set the parameters. Use the following command sequence with two pairs of virtqueues as an example:@@ -1726,6 +1724,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5. \end{itemize} +The device can generate notifications more or less frequently +than specified by the coalescing parameters. + \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.@@ -1798,7 +1799,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0. -The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort: +The behavior of the device in response to specific coalescing parameters is best-effort: the device MAY generate notifications more or less frequently than specified. A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
Quoted 46 lines · Michael S. Tsirkin
Quoted 35 lines · Si-Wei Liu
Quoted 33 lines · Michael S. Tsirkin
Quoted 13 lines · Halil Pasic
Quoted 1 line · On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu <[email protected]> wrote:
On 6/24/2024 10:56 PM, Parav Pandit wrote:
[..]
Quoted 8 lines
Quoted 8 lines
I saw the need of this proposal slightly differently in the discussion with Heng in v4.
The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offeringVIRTIO_NET_F_VQ_NOTF_COAL.Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL
2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.Quoted 8 lines
I have certainly understood this differently. When
VIRTIO_NET_F_VQ_NOTF_COALis not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).If
VIRTIO_NET_F_VQ_NOTF_COALis negotiated the device is supposed to/MUST do the coalescing according to the parameters as described by the virtio spec.Michael, Jason: Can you guys weigh in on this?
I still don't understand why this change is needed. We have this text:
The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass.and
The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:
the device MAY generate notifications more or less frequently than specified.Quoted 8 lines
Quoted 2 lines
So with no spec changes, devices already can do what this patch says they can do - send notifications less frequently.
Right. If I still recall it correctly, the above text was to accommodate the interaction with existing used buffer notification mechanism,
F_EVENT_IDXor whatnot.
When coalescing is in place and effective, interrupt delivery is a collaborative consideration based on the coalescing parameters specified *AND * event index /NO_INTERRUPTflag as well.No, not really.
Quoted 7 lines
For that we have the next sentence:
A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
And of course event index never makes you send interrupts more frequently.
Hmmm, this next sentence you reference above was indeed for the interaction between coalescing and used buffer suppression. Then what's the best-effort part was about, really? Round-up or round down the set value to the power of 2 to save space? How is it relevant to our discussion? I think even with rounding it shouldn't be too off? (as said, by best-effort v.s. give up)
-Siwei
Quoted 46 lines
Quoted 46 lines
Quoted 8 lines
Re-reading this spec text, maybe the confusion is that it mentions set commands specifically? And it's also stuck in the middle where it's easy to miss.
So it would seem that the following should be sufficient, and it looks like a small clarification we could just apply and include in the vote for the csd. What do you guys think?
The revised text below looks good to me.
Thanks,
-SiweiQuoted 34 lines
Signed-off-by Michael S. Tsirkin <[email protected]>----
device-types/net/description.tex +4
-3@@ -1711,8 +1711,6 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi for an enabled transmit/receive virtqueue whose index is \field{vq_index}. \end{enumerate} -The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class. - If coalescing parameters are being set, the device applies the last coalescing parameters set for a virtqueue, regardless of the command used to set the parameters. Use the following command sequence with two pairs of virtqueues as an example:@@ -1726,6 +1724,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5. \end{itemize} +The device can generate notifications more or less frequently +than specified by the coalescing parameters. + \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.@@ -1798,7 +1799,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0. -The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort: +The behavior of the device in response to specific coalescing parameters is best-effort: the device MAY generate notifications more or less frequently than specified. A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
Quoted 55 lines · Si-Wei Liu
Quoted 51 lines · Michael S. Tsirkin
Quoted 35 lines · Si-Wei Liu
Quoted 33 lines · Michael S. Tsirkin
Quoted 13 lines · Halil Pasic
Quoted 1 line · On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu <[email protected]> wrote:
On 6/24/2024 10:56 PM, Parav Pandit wrote:
[..]
Quoted 8 lines
Quoted 8 lines
I saw the need of this proposal slightly differently in the discussion with Heng in v4.
The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offeringVIRTIO_NET_F_VQ_NOTF_COAL.Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL
2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.Quoted 8 lines
I have certainly understood this differently. When
VIRTIO_NET_F_VQ_NOTF_COALis not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).If
VIRTIO_NET_F_VQ_NOTF_COALis negotiated the device is supposed to/MUST do the coalescing according to the parameters as described by the virtio spec.Michael, Jason: Can you guys weigh in on this?
I still don't understand why this change is needed. We have this text:
The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass.and
The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:
the device MAY generate notifications more or less frequently than specified.Quoted 8 lines
Quoted 2 lines
So with no spec changes, devices already can do what this patch says they can do - send notifications less frequently.
Right. If I still recall it correctly, the above text was to accommodate the interaction with existing used buffer notification mechanism,
F_EVENT_IDXor whatnot.
When coalescing is in place and effective, interrupt delivery is a collaborative consideration based on the coalescing parameters specified *AND * event index /NO_INTERRUPTflag as well.No, not really.
For that we have the next sentence:
A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
Quoted 8 lines
Quoted 2 lines
And of course event index never makes you send interrupts more frequently.
Hmmm, this next sentence you reference above was indeed for the interaction between coalescing and used buffer suppression. Then what's the best-effort part was about, really? Round-up or round down the set value to the power of 2 to save space? How is it relevant to our discussion? I think even with rounding it shouldn't be too off? (as said, by best-effort v.s. give up)
Would it help if we add something like below in the guidance?
When the device wants to send a notification it needs:
1) check notification suppression, don't send interrupt if it is suppressed otherwise goto 2)
2) check event index, don't send interrupt if event index is not crossed otherwise goto 3)
3) check coalescing, don't send interrupt if the coalescing limit is not exceed, otherwise send the interrupt
(Just to demonstrate the idea, needs tweaking for sure)
THanks
Quoted 50 lines
-Siwei
Quoted 46 lines
Quoted 46 lines
Quoted 8 lines
Re-reading this spec text, maybe the confusion is that it mentions set commands specifically? And it's also stuck in the middle where it's easy to miss.
So it would seem that the following should be sufficient, and it looks like a small clarification we could just apply and include in the vote for the csd. What do you guys think?
The revised text below looks good to me.
Thanks,
-SiweiQuoted 34 lines
Signed-off-by Michael S. Tsirkin <[email protected]>----
device-types/net/description.tex +4
-3@@ -1711,8 +1711,6 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi for an enabled transmit/receive virtqueue whose index is \field{vq_index}. \end{enumerate} -The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class. - If coalescing parameters are being set, the device applies the last coalescing parameters set for a virtqueue, regardless of the command used to set the parameters. Use the following command sequence with two pairs of virtqueues as an example:@@ -1726,6 +1724,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5. \end{itemize} +The device can generate notifications more or less frequently +than specified by the coalescing parameters. + \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.@@ -1798,7 +1799,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0. -The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort: +The behavior of the device in response to specific coalescing parameters is best-effort: the device MAY generate notifications more or less frequently than specified. A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
Quoted 66 lines · Jason Wang
Quoted 53 lines · Si-Wei Liu
Quoted 50 lines · Michael S. Tsirkin
Quoted 34 lines · Si-Wei Liu
Quoted 33 lines · Michael S. Tsirkin
Quoted 13 lines · Halil Pasic
Quoted 1 line · On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu <[email protected]> wrote:
On 6/24/2024 10:56 PM, Parav Pandit wrote:
[..]
Quoted 8 lines
Quoted 8 lines
I saw the need of this proposal slightly differently in the discussion with Heng in v4.
The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offeringVIRTIO_NET_F_VQ_NOTF_COAL.Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL
2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.Quoted 8 lines
I have certainly understood this differently. When
VIRTIO_NET_F_VQ_NOTF_COALis not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).If
VIRTIO_NET_F_VQ_NOTF_COALis negotiated the device is supposed to/MUST do the coalescing according to the parameters as described by the virtio spec.Michael, Jason: Can you guys weigh in on this?
I still don't understand why this change is needed. We have this text:
The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass.and
The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:
the device MAY generate notifications more or less frequently than specified.Quoted 8 lines
Quoted 2 lines
So with no spec changes, devices already can do what this patch says they can do - send notifications less frequently.
Right. If I still recall it correctly, the above text was to accommodate the interaction with existing used buffer notification mechanism,
F_EVENT_IDXor whatnot.
When coalescing is in place and effective, interrupt delivery is a collaborative consideration based on the coalescing parameters specified *AND * event index /NO_INTERRUPTflag as well.No, not really.
For that we have the next sentence:
A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
Quoted 8 lines
Quoted 2 lines
And of course event index never makes you send interrupts more frequently.
Hmmm, this next sentence you reference above was indeed for the interaction between coalescing and used buffer suppression. Then what's the best-effort part was about, really? Round-up or round down the set value to the power of 2 to save space? How is it relevant to our discussion? I think even with rounding it shouldn't be too off? (as said, by best-effort v.s. give up)
Would it help if we add something like below in the guidance?
When the device wants to send a notification it needs:
Quoted 8 lines
1) check notification suppression, don't send interrupt if it is suppressed otherwise goto 2)
2) check event index, don't send interrupt if event index is not crossed otherwise goto 3)
3) check coalescing, don't send interrupt if the coalescing limit is not exceed, otherwise send the interrupt(Just to demonstrate the idea, needs tweaking for sure)
Yeah, this looks good to me, in principle. I recall there was detailed description like this in the early revisions from the first set of coalescing proposal, but those useful descriptions were gone for some reason. We should really come up with clear clarifications to help readers set expectation rather than being vague or worrying too much about verbosity. I was having a hard time in understanding what the best-effort part is about...
-Siwei
Quoted 50 lines
THanks
Quoted 46 lines
-Siwei
Quoted 44 lines
Quoted 44 lines
Quoted 7 lines
Re-reading this spec text, maybe the confusion is that it mentions set commands specifically? And it's also stuck in the middle where it's easy to miss.
So it would seem that the following should be sufficient, and it looks like a small clarification we could just apply and include in the vote for the csd. What do you guys think?
The revised text below looks good to me.
Thanks,
-SiweiQuoted 33 lines
Signed-off-by Michael S. Tsirkin <[email protected]>----
device-types/net/description.tex +4
-3@@ -1711,8 +1711,6 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi for an enabled transmit/receive virtqueue whose index is \field{vq_index}. \end{enumerate} -The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class. - If coalescing parameters are being set, the device applies the last coalescing parameters set for a virtqueue, regardless of the command used to set the parameters. Use the following command sequence with two pairs of virtqueues as an example:@@ -1726,6 +1724,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5. \end{itemize} +The device can generate notifications more or less frequently +than specified by the coalescing parameters. + \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.@@ -1798,7 +1799,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0. -The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort: +The behavior of the device in response to specific coalescing parameters is best-effort: the device MAY generate notifications more or less frequently than specified. A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
Quoted 85 lines · Si-Wei Liu
Quoted 66 lines · Jason Wang
Quoted 53 lines · Si-Wei Liu
Quoted 50 lines · Michael S. Tsirkin
Quoted 34 lines · Si-Wei Liu
Quoted 33 lines · Michael S. Tsirkin
Quoted 13 lines · Halil Pasic
Quoted 1 line · On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu <[email protected]> wrote:
On 6/24/2024 10:56 PM, Parav Pandit wrote:
[..]
Quoted 8 lines
Quoted 8 lines
I saw the need of this proposal slightly differently in the discussion with Heng in v4.
The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offeringVIRTIO_NET_F_VQ_NOTF_COAL.Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL
2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.Quoted 8 lines
I have certainly understood this differently. When
VIRTIO_NET_F_VQ_NOTF_COALis not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).If
VIRTIO_NET_F_VQ_NOTF_COALis negotiated the device is supposed to/MUST do the coalescing according to the parameters as described by the virtio spec.Michael, Jason: Can you guys weigh in on this?
I still don't understand why this change is needed. We have this text:
The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass.and
The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:
the device MAY generate notifications more or less frequently than specified.Quoted 8 lines
Quoted 2 lines
So with no spec changes, devices already can do what this patch says they can do - send notifications less frequently.
Right. If I still recall it correctly, the above text was to accommodate the interaction with existing used buffer notification mechanism,
F_EVENT_IDXor whatnot.
When coalescing is in place and effective, interrupt delivery is a collaborative consideration based on the coalescing parameters specified *AND * event index /NO_INTERRUPTflag as well.No, not really.
For that we have the next sentence:
A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
Quoted 8 lines
Quoted 2 lines
And of course event index never makes you send interrupts more frequently.
Hmmm, this next sentence you reference above was indeed for the interaction between coalescing and used buffer suppression. Then what's the best-effort part was about, really? Round-up or round down the set value to the power of 2 to save space? How is it relevant to our discussion? I think even with rounding it shouldn't be too off? (as said, by best-effort v.s. give up)
Would it help if we add something like below in the guidance?
When the device wants to send a notification it needs:
Quoted 8 lines
1) check notification suppression, don't send interrupt if it is suppressed otherwise goto 2)
2) check event index, don't send interrupt if event index is not crossed otherwise goto 3)
3) check coalescing, don't send interrupt if the coalescing limit is not exceed, otherwise send the interrupt(Just to demonstrate the idea, needs tweaking for sure)
Yeah, this looks good to me, in principle. I recall there was detailed description like this in the early revisions from the first set of coalescing proposal, but those useful descriptions were gone for some reason. We should really come up with clear clarifications to help readers set expectation rather than being vague or worrying too much about verbosity. I was having a hard time in understanding what the best-effort part is about...
Quoted 1 line
-Siwei
It means it's a hint from the driver.
As another example, devices might have a coarse grained clock, or a limited packet counter.
Or whatever. No need to guess.
There's a balance with verbosity - more text needs to add value not just repeat same thing many times. It also needs to be written in an exceptionally clear way otherwise we are confusing the reader instead of helping.
And in this case, I think I agree - instead of saying it is best effort and then clarifying, we should straight away say this is a hint.
The text in the conformance section should be thus changed:
-The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
+The device MAY treat the coalescing parameters as a hint:
the device MAY generate notifications more or less frequently than specified.
The text in non conformance section is fine, but let's make it
consistent:
-The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
+ The device can treat coalescing parameters as a hint,
+ generating notifications more or less frequently
+ than specified by the coalescing parameters.
Quoted 50 lines
Quoted 50 lines
THanks
Quoted 46 lines
-Siwei
Quoted 44 lines
Quoted 44 lines
Quoted 7 lines
Re-reading this spec text, maybe the confusion is that it mentions set commands specifically? And it's also stuck in the middle where it's easy to miss.
So it would seem that the following should be sufficient, and it looks like a small clarification we could just apply and include in the vote for the csd. What do you guys think?
The revised text below looks good to me.
Thanks,
-SiweiQuoted 33 lines
Signed-off-by Michael S. Tsirkin <[email protected]>----
device-types/net/description.tex +4
-3@@ -1711,8 +1711,6 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi for an enabled transmit/receive virtqueue whose index is \field{vq_index}. \end{enumerate} -The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class. - If coalescing parameters are being set, the device applies the last coalescing parameters set for a virtqueue, regardless of the command used to set the parameters. Use the following command sequence with two pairs of virtqueues as an example:@@ -1726,6 +1724,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5. \end{itemize} +The device can generate notifications more or less frequently +than specified by the coalescing parameters. + \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.@@ -1798,7 +1799,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0. -The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort: +The behavior of the device in response to specific coalescing parameters is best-effort: the device MAY generate notifications more or less frequently than specified. A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
Quoted 77 lines · Si-Wei Liu
Quoted 73 lines · Jason Wang
Quoted 53 lines · Si-Wei Liu
Quoted 50 lines · Michael S. Tsirkin
Quoted 34 lines · Si-Wei Liu
Quoted 33 lines · Michael S. Tsirkin
Quoted 13 lines · Halil Pasic
Quoted 1 line · On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu <[email protected]> wrote:
On 6/24/2024 10:56 PM, Parav Pandit wrote:
[..]
Quoted 8 lines
Quoted 8 lines
I saw the need of this proposal slightly differently in the discussion with Heng in v4.
The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offeringVIRTIO_NET_F_VQ_NOTF_COAL.Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL
2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.Quoted 8 lines
I have certainly understood this differently. When
VIRTIO_NET_F_VQ_NOTF_COALis not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).If
VIRTIO_NET_F_VQ_NOTF_COALis negotiated the device is supposed to/MUST do the coalescing according to the parameters as described by the virtio spec.Michael, Jason: Can you guys weigh in on this?
I still don't understand why this change is needed. We have this text:
The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass.and
The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:
the device MAY generate notifications more or less frequently than specified.Quoted 8 lines
Quoted 2 lines
So with no spec changes, devices already can do what this patch says they can do - send notifications less frequently.
Right. If I still recall it correctly, the above text was to accommodate the interaction with existing used buffer notification mechanism,
F_EVENT_IDXor whatnot.
When coalescing is in place and effective, interrupt delivery is a collaborative consideration based on the coalescing parameters specified *AND * event index /NO_INTERRUPTflag as well.No, not really.
For that we have the next sentence:
A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
Quoted 8 lines
Quoted 2 lines
And of course event index never makes you send interrupts more frequently.
Hmmm, this next sentence you reference above was indeed for the interaction between coalescing and used buffer suppression. Then what's the best-effort part was about, really? Round-up or round down the set value to the power of 2 to save space? How is it relevant to our discussion? I think even with rounding it shouldn't be too off? (as said, by best-effort v.s. give up)
Would it help if we add something like below in the guidance?
When the device wants to send a notification it needs:
1) check notification suppression, don't send interrupt if it is suppressed otherwise goto 2)
2) check event index, don't send interrupt if event index is not crossed otherwise goto 3)
3) check coalescing, don't send interrupt if the coalescing limit is not exceed, otherwise send the interrupt
Quoted 7 lines
Quoted 1 line
(Just to demonstrate the idea, needs tweaking for sure)
Yeah, this looks good to me, in principle. I recall there was detailed description like this in the early revisions from the first set of coalescing proposal, but those useful descriptions were gone for some reason. We should really come up with clear clarifications to help readers set expectation rather than being vague or worrying too much about verbosity.
Yes.
Quoted 2 lines
I was having a hard time in understanding what the best-effort part is about...
Being verbose might be helpful for vendors. Otherwise the may guess (since we don't have a reference software implementation so far).
Thanks
Quoted 53 lines
-Siwei
Quoted 50 lines
THanks
Quoted 46 lines
-Siwei
Quoted 44 lines
Quoted 44 lines
Quoted 7 lines
Re-reading this spec text, maybe the confusion is that it mentions set commands specifically? And it's also stuck in the middle where it's easy to miss.
So it would seem that the following should be sufficient, and it looks like a small clarification we could just apply and include in the vote for the csd. What do you guys think?
The revised text below looks good to me.
Thanks,
-SiweiQuoted 33 lines
Signed-off-by Michael S. Tsirkin <[email protected]>----
device-types/net/description.tex +4
-3@@ -1711,8 +1711,6 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi for an enabled transmit/receive virtqueue whose index is \field{vq_index}. \end{enumerate} -The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class. - If coalescing parameters are being set, the device applies the last coalescing parameters set for a virtqueue, regardless of the command used to set the parameters. Use the following command sequence with two pairs of virtqueues as an example:@@ -1726,6 +1724,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5. \end{itemize} +The device can generate notifications more or less frequently +than specified by the coalescing parameters. + \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.@@ -1798,7 +1799,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0. -The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort: +The behavior of the device in response to specific coalescing parameters is best-effort: the device MAY generate notifications more or less frequently than specified. A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
Quoted 9 lines · On Fri, 28 Jun 2024 16:23:17 +0800 Jason Wang <[email protected]> wrote:
Quoted 6 lines
Hmmm, this next sentence you reference above was indeed for the interaction between coalescing and used buffer suppression. Then what's the best-effort part was about, really? Round-up or round down the set value to the power of 2 to save space? How is it relevant to our discussion? I think even with rounding it shouldn't be too off? (as said, by best-effort v.s. give up)
Would it help if we add something like below in the guidance?
It would IMHO greatly benefit clarity, provided we do get it right.
Quoted 10 lines
When the device wants to send a notification it needs:
1) check notification suppression, don't send interrupt if it is suppressed otherwise goto 2)
2) check event index, don't send interrupt if event index is not crossed otherwise goto 3)
3) check coalescing, don't send interrupt if the coalescing limit is not exceed, otherwise send the interrupt(Just to demonstrate the idea, needs tweaking for sure)
Yes I agree this indeed needs tweaking. Let me make some points.
1) for virtio-net we have 3 distinct mechanisms that police used buffer notification avoidance, but 2 of the three are actually mutually exclusive if I understand it correctly. Namely we have coalescing guarded by VIRTIO_NET_F_NOTF_COAL and then depending on whether _F_EVENT_IDX was negotiated or not, we have "event_idx based" or "crude flag based" notification suppression.
2) Let me just quote the entire section on the good old used buffer notification suppression
"""
2.7.7.2 Device Requirements: Used Buffer Notification Suppression
If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
* The device MUST ignore the used_event value.
* After the device writes a descriptor index into the used ring:
- If flags is 1, the device SHOULD NOT send a notification.
- If flags is 0, the device MUST send a notification.
Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
* The device MUST ignore the lower bit of flags.
* After the device writes a descriptor index into the used ring:
- If the idx field in the used ring (which determined where that
descriptor index was placed) was equal to used_event, the device MUST send a notification.
- Otherwise the device SHOULD NOT send a notification.
Note: For example, if used_event is 0, then a device using
VIRTIO_F_EVENT_IDX would send a used buffer notification to the driver after the first buffer is used (and again after the 65536th buffer, etc).
"""
Frankly, a MUST is a must, and I don't see very little leeway for the coalescing to avoid any notifications. But on the other hand, that is the very purpose of the coalescing so IMHO we have a problem here.
The only loophole here is "after", which could be stretched to "eventually, before the end of the world". Which would basically reduce the MUST ad-absurdum, and I would not like that.
3) AFAIU if multiple suppression mechanisms are active concurrently we are aiming for some sort of an logical AND semantic i.e. a further reduction over just a single one being employed (and not OR).
But IMHO the both the wording for the event_idx based and curde suppression, as well as for coalescing.
Let me quote the full description for the coalescing:
"""
5.1.6.5.9.1 Operation
The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in 2.7.7.
When the device has non-zero max_usecs and non-zero max_packets, it starts counting microseconds and packets upon receiving/sending a packet. The device counts packets and microseconds for each receive virtqueue and transmit virtqueue separately. In this case, the notification conditions are met when max_usecs microseconds elapse, or upon sending/receiving max_packets packets, whichever happens first. Afterwards, the device waits for the next packet and starts counting packets and microseconds again.
When the device has max_usecs = 0 or max_packets = 0, the notification conditions are met after every packet received/sent. """
Here "Afterwards" is ambiguous. What we want here is "afterwards" == "after the notification has been sent", but one can also read it as "after the condition has been met".
4) I see two ways to make sense out of it.
My preferred way would be to replace "send a notification" with "generate a notification" in the main text including "2.7.7.2 Device
Requirements: Used Buffer Notification Suppression", along with adding an explanation which states that generated notifications are sent immediately unless a device has an active facility that under a certain set of conditions retains and possibly coalesces notifications.
Please notice that here coalescence is used with its dictionary meaning (see https://dictionary.cambridge.org/dictionary/english/coalesce),
and that the current description of NET_F_NOTIF_COAL is more akin to a suppression mechanism where notifications are suppressed or discarded and not coalesced.
The notification coalescing facility would then retain the first used buffer notification request for each queue (those are still generated by the same rules), until either the packet count or the timeout type max frequency (or delay we need to clarify that) condition is met, and coalesce any further notification requests with the retained one until it is sent.
The other way, which I'm not sure is actually viable, is basically to try to clarify the hell out of the current approach as taken by "5.1.6.5.9.1 Operation" by making it crystal clear when exactly are the "notification conditions met" and what is the complement state of that, and add a sentence to 2.7.7.2 that references device specific mechanisms to do more suppression. And of course we would have to find a nice definition for "if the notifications are not suppressed as explained in 2.7.7." such that loss of initiative is not possible.
IMHO any solution where things can stall because we would have sent an used buffer notification without "coalescing" but we did not because of coalescing is not acceptable. This could happen if we cross event_idx when the coalescing notification conditions are not met and thus do not notify, and then when the coalescing notification condition are finally met (e.g. via timeout) the event_id type suppression is active again because we crossed event_idx when coalescing notification condition was not met, and we end up not notifying again.
Sorry for the wall of text.
Regards,
Halil
Quoted 24 lines · Halil Pasic
Quoted 9 lines · On Fri, 28 Jun 2024 16:23:17 +0800 Jason Wang <[email protected]> wrote:
Quoted 6 lines
Hmmm, this next sentence you reference above was indeed for the interaction between coalescing and used buffer suppression. Then what's the best-effort part was about, really? Round-up or round down the set value to the power of 2 to save space? How is it relevant to our discussion? I think even with rounding it shouldn't be too off? (as said, by best-effort v.s. give up)
Would it help if we add something like below in the guidance?
It would IMHO greatly benefit clarity, provided we do get it right.
Quoted 9 lines
When the device wants to send a notification it needs:
1) check notification suppression, don't send interrupt if it is suppressed otherwise goto 2)
2) check event index, don't send interrupt if event index is not crossed otherwise goto 3)
3) check coalescing, don't send interrupt if the coalescing limit is not exceed, otherwise send the interrupt
Quoted 3 lines
Quoted 1 line
(Just to demonstrate the idea, needs tweaking for sure)
Yes I agree this indeed needs tweaking. Let me make some points.
Very good points below.
Quoted 35 lines
1) for virtio-net we have 3 distinct mechanisms that police used buffer notification avoidance, but 2 of the three are actually mutually exclusive if I understand it correctly. Namely we have coalescing guarded by
VIRTIO_NET_F_NOTF_COALand then depending on whether_F_EVENT_IDXwas negotiated or not, we have "event_idxbased" or "crude flag based" notification suppression.2) Let me just quote the entire section on the good old used buffer notification suppression
"""
2.7.7.2 Device Requirements: Used Buffer Notification SuppressionIf the
VIRTIO_F_EVENT_IDXfeature bit is not negotiated:* The device MUST ignore the
used_eventvalue.
* After the device writes a descriptor index into the used ring:
- If flags is 1, the device SHOULD NOT send a notification.
- If flags is 0, the device MUST send a notification.Otherwise, if the
VIRTIO_F_EVENT_IDXfeature bit is negotiated:* The device MUST ignore the lower bit of flags.
* After the device writes a descriptor index into the used ring:
- If the idx field in the used ring (which determined where that
descriptor index was placed) was equal toused_event, the device MUST send a notification.
- Otherwise the device SHOULD NOT send a notification.Note: For example, if
used_eventis 0, then a device using
VIRTIO_F_EVENT_IDXwould send a used buffer notification to the driver after the first buffer is used (and again after the 65536th buffer, etc).
"""
Quoted 8 lines
Frankly, a MUST is a must, and I don't see very little leeway for the coalescing to avoid any notifications. But on the other hand, that is the very purpose of the coalescing so IMHO we have a problem here.
The only loophole here is "after", which could be stretched to "eventually, before the end of the world". Which would basically reduce the MUST ad-absurdum, and I would not like that.
I agree as such but I also think we can just go ahead and make this explicit. That is s/after/at some point in time after/.
And then let's add text that actually explains how
- devices can defer notifications for performance
- some devices can have device specific mechanisms for driver to control how much notifications are deferred.
WDYT?
Quoted 69 lines
3) AFAIU if multiple suppression mechanisms are active concurrently we are aiming for some sort of an logical AND semantic i.e. a further reduction over just a single one being employed (and not OR).
But IMHO the both the wording for the
event_idxbased and curde suppression, as well as for coalescing.Let me quote the full description for the coalescing:
"""
5.1.6.5.9.1 Operation
The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in 2.7.7.When the device has non-zero
max_usecsand non-zeromax_packets, it starts counting microseconds and packets upon receiving/sending a packet. The device counts packets and microseconds for each receive virtqueue and transmit virtqueue separately. In this case, the notification conditions are met whenmax_usecsmicroseconds elapse, or upon sending/receivingmax_packetspackets, whichever happens first. Afterwards, the device waits for the next packet and starts counting packets and microseconds again.When the device has
max_usecs= 0 ormax_packets= 0, the notification conditions are met after every packet received/sent. """Here "Afterwards" is ambiguous. What we want here is "afterwards" == "after the notification has been sent", but one can also read it as "after the condition has been met".
4) I see two ways to make sense out of it.
My preferred way would be to replace "send a notification" with "generate a notification" in the main text including "2.7.7.2 Device
Requirements: Used Buffer Notification Suppression", along with adding an explanation which states that generated notifications are sent immediately unless a device has an active facility that under a certain set of conditions retains and possibly coalesces notifications.Please notice that here coalescence is used with its dictionary meaning (see https://dictionary.cambridge.org/dictionary/english/coalesce),
and that the current description ofNET_F_NOTIF_COALis more akin to a suppression mechanism where notifications are suppressed or discarded and not coalesced.The notification coalescing facility would then retain the first used buffer notification request for each queue (those are still generated by the same rules), until either the packet count or the timeout type max frequency (or delay we need to clarify that) condition is met, and coalesce any further notification requests with the retained one until it is sent.
The other way, which I'm not sure is actually viable, is basically to try to clarify the hell out of the current approach as taken by "5.1.6.5.9.1 Operation" by making it crystal clear when exactly are the "notification conditions met" and what is the complement state of that, and add a sentence to 2.7.7.2 that references device specific mechanisms to do more suppression. And of course we would have to find a nice definition for "if the notifications are not suppressed as explained in 2.7.7." such that loss of initiative is not possible.
IMHO any solution where things can stall because we would have sent an used buffer notification without "coalescing" but we did not because of coalescing is not acceptable. This could happen if we cross
event_idxwhen the coalescing notification conditions are not met and thus do not notify, and then when the coalescing notification condition are finally met (e.g. via timeout) theevent_idtype suppression is active again because we crossedevent_idxwhen coalescing notification condition was not met, and we end up not notifying again.Sorry for the wall of text.
Regards,
Halil
Quoted 11 lines · Halil Pasic
Quoted 7 lines · On Fri, 28 Jun 2024 16:23:17 +0800 Jason Wang <[email protected]> wrote:
Quoted 6 lines
Hmmm, this next sentence you reference above was indeed for the interaction between coalescing and used buffer suppression. Then what's the best-effort part was about, really? Round-up or round down the set value to the power of 2 to save space? How is it relevant to our discussion? I think even with rounding it shouldn't be too off? (as said, by best-effort v.s. give up)
Quoted 4 lines
Quoted 2 lines
Would it help if we add something like below in the guidance?
It would IMHO greatly benefit clarity, provided we do get it right.
Exactly.
Quoted 71 lines
Quoted 11 lines
When the device wants to send a notification it needs:
1) check notification suppression, don't send interrupt if it is suppressed otherwise goto 2)
2) check event index, don't send interrupt if event index is not crossed otherwise goto 3)
3) check coalescing, don't send interrupt if the coalescing limit is not exceed, otherwise send the interrupt(Just to demonstrate the idea, needs tweaking for sure)
Yes I agree this indeed needs tweaking. Let me make some points.
1) for virtio-net we have 3 distinct mechanisms that police used buffer notification avoidance, but 2 of the three are actually mutually exclusive if I understand it correctly. Namely we have coalescing guarded by
VIRTIO_NET_F_NOTF_COALand then depending on whether_F_EVENT_IDXwas negotiated or not, we have "event_idxbased" or "crude flag based" notification suppression.2) Let me just quote the entire section on the good old used buffer notification suppression
"""
2.7.7.2 Device Requirements: Used Buffer Notification SuppressionIf the
VIRTIO_F_EVENT_IDXfeature bit is not negotiated:* The device MUST ignore the
used_eventvalue.
* After the device writes a descriptor index into the used ring:
- If flags is 1, the device SHOULD NOT send a notification.
- If flags is 0, the device MUST send a notification.Otherwise, if the
VIRTIO_F_EVENT_IDXfeature bit is negotiated:* The device MUST ignore the lower bit of flags.
* After the device writes a descriptor index into the used ring:
- If the idx field in the used ring (which determined where that
descriptor index was placed) was equal toused_event, the device MUST send a notification.
- Otherwise the device SHOULD NOT send a notification.Note: For example, if
used_eventis 0, then a device using
VIRTIO_F_EVENT_IDXwould send a used buffer notification to the driver after the first buffer is used (and again after the 65536th buffer, etc).
"""Frankly, a MUST is a must, and I don't see very little leeway for the coalescing to avoid any notifications. But on the other hand, that is the very purpose of the coalescing so IMHO we have a problem here.
The only loophole here is "after", which could be stretched to "eventually, before the end of the world". Which would basically reduce the MUST ad-absurdum, and I would not like that.
3) AFAIU if multiple suppression mechanisms are active concurrently we are aiming for some sort of an logical AND semantic i.e. a further reduction over just a single one being employed (and not OR).
But IMHO the both the wording for the
event_idxbased and curde suppression, as well as for coalescing.Let me quote the full description for the coalescing:
"""
5.1.6.5.9.1 Operation
The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in 2.7.7.
Quoted 8 lines
When the device has non-zero
max_usecsand non-zeromax_packets, it starts counting microseconds and packets upon receiving/sending a packet. The device counts packets and microseconds for each receive virtqueue and transmit virtqueue separately. In this case, the notification conditions are met whenmax_usecsmicroseconds elapse, or upon sending/receivingmax_packetspackets, whichever happens first. Afterwards, the device waits for the next packet and starts counting packets and microseconds again.When the device has
max_usecs= 0 ormax_packets= 0, the notification conditions are met after every packet received/sent. """Here "Afterwards" is ambiguous. What we want here is "afterwards" == "after the notification has been sent", but one can also read it as "after the condition has been met".
I agree and unfortunately it could be understood differently by vendors (and we don't even have an open source software implementation so far to refer to).
Btw, here is the one that is used by Intel E810. It seems better than the "afterwards" we used here.
"""
Interrupt throttling is a mechanism that guarantees a minimum gap between two consecutive interrupts (other than possible jitter caused by handling the interrupts). The E810 counts the time since the last interrupt is scheduled and compares it against the ITR setting. If an event associated with this ITR happens before the ITR expires, the interrupt assertion is delayed until the ITR expires. If the ITR expires before any event associated with this interrupt, the interrupt logic is “armed” and the interrupt can be asserted the moment the event happens.
"""
And if we follow this it should be "after the notification has been sent".
Quoted 10 lines
4) I see two ways to make sense out of it.
My preferred way would be to replace "send a notification" with "generate a notification" in the main text including "2.7.7.2 Device
Requirements: Used Buffer Notification Suppression", along with adding an explanation which states that generated notifications are sent immediately unless a device has an active facility that under a certain set of conditions retains and possibly coalesces notifications.
Quoted 2 lines
Please notice that here coalescence is used with its dictionary meaning (see https://dictionary.cambridge.org/dictionary/english/coalesce),
Probably, because we don't tell the driver how many interrupts were coalesced. I guess that's why it is usually named as "interrupt moderation" for most of the nic vendors.
Quoted 11 lines
and that the current description of
NET_F_NOTIF_COALis more akin to a suppression mechanism where notifications are suppressed or discarded and not coalesced.The notification coalescing facility would then retain the first used buffer notification request for each queue (those are still generated by the same rules), until either the packet count or the timeout type max frequency (or delay we need to clarify that) condition is met, and coalesce any further notification requests with the retained one until it is sent.
Quoted 8 lines
The other way, which I'm not sure is actually viable, is basically to try to clarify the hell out of the current approach as taken by "5.1.6.5.9.1 Operation" by making it crystal clear when exactly are the "notification conditions met" and what is the complement state of that, and add a sentence to 2.7.7.2 that references device specific mechanisms to do more suppression. And of course we would have to find a nice definition for "if the notifications are not suppressed as explained in 2.7.7." such that loss of initiative is not possible.
This is what I try to achieve. Not sure if it's the best. Or we can have a pseudo code to explain if it helps.
Quoted 10 lines
IMHO any solution where things can stall because we would have sent an used buffer notification without "coalescing" but we did not because of coalescing is not acceptable. This could happen if we cross
event_idxwhen the coalescing notification conditions are not met and thus do not notify, and then when the coalescing notification condition are finally met (e.g. via timeout) theevent_idtype suppression is active again because we crossedevent_idxwhen coalescing notification condition was not met, and we end up not notifying again.
Quoted 4 lines
Sorry for the wall of text.
Regards,
Halil
Thanks
Quoted 4 lines
Quoted 22 lines · On Thu, 27 Jun 2024 08:35:16 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 21 lines · Halil Pasic
Quoted 1 line · On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu <[email protected]> wrote:
On 6/24/2024 10:56 PM, Parav Pandit wrote:
[..]
Quoted 8 lines
Quoted 8 lines
I saw the need of this proposal slightly differently in the discussion with Heng in v4.
The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offeringVIRTIO_NET_F_VQ_NOTF_COAL.Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL
2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.I have certainly understood this differently. When
VIRTIO_NET_F_VQ_NOTF_COALis not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).If
VIRTIO_NET_F_VQ_NOTF_COALis negotiated the device is supposed to/MUST do the coalescing according to the parameters as described by the virtio spec.
Quoted 7 lines
Quoted 1 line
Michael, Jason: Can you guys weigh in on this?
I still don't understand why this change is needed. We have this text:
The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass.
The less frequently could be understood like due to notification suppression.
Quoted 5 lines
and
The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:
the device MAY generate notifications more or less frequently than specified.
To me best-effort implies making an effort. Here we talk about intentionally ignoring the parameter values. For me stuff like not being precise about the counting or timekeeping, or running into the danger of having to drop packets because the queue is about to fill up, would qualify as best-effort deviations from the specified behavior.
I believe we want the drivers written under the assumption that the workings of a notification coalescing in any device that implements it are close enough to what is described in the spec. Just my opinion.
Regards,
Halil
Quoted 27 lines · Halil Pasic
Quoted 24 lines · On Thu, 27 Jun 2024 08:35:16 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 22 lines · Halil Pasic
Quoted 1 line · On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu <[email protected]> wrote:
On 6/24/2024 10:56 PM, Parav Pandit wrote:
[..]
Quoted 8 lines
Quoted 8 lines
I saw the need of this proposal slightly differently in the discussion with Heng in v4.
The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offeringVIRTIO_NET_F_VQ_NOTF_COAL.Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL
2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.I have certainly understood this differently. When
VIRTIO_NET_F_VQ_NOTF_COALis not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).If
VIRTIO_NET_F_VQ_NOTF_COALis negotiated the device is supposed to/MUST do the coalescing according to the parameters as described by the virtio spec.Michael, Jason: Can you guys weigh in on this?
Quoted 8 lines
Quoted 5 lines
I still don't understand why this change is needed. We have this text:
The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass.The less frequently could be understood like due to notification suppression.
I think you are missing that there is a separate text about notification suppression immediately after it:
A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
I think that makes it clear this text is separate from suppression.
Quoted 12 lines
Quoted 5 lines
and
The behavior of the device in response to set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass is best-effort:
the device MAY generate notifications more or less frequently than specified.To me best-effort implies making an effort. Here we talk about intentionally ignoring the parameter values. For me stuff like not being precise about the counting or timekeeping, or running into the danger of having to drop packets because the queue is about to fill up, would qualify as best-effort deviations from the specified behavior.
Quoted 6 lines
I believe we want the drivers written under the assumption that the workings of a notification coalescing in any device that implements it are close enough to what is described in the spec. Just my opinion.
Regards,
Halil
OK, I guess we can clarify:
in particular, the device MAY coalesce notifications when coalescing parameters are set to 0.
will that make it better?
Signature · 1 line
MST
Quoted 1 line · On Sun, 30 Jun 2024 12:55:35 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
On Sat, Jun 29, 2024 at 08:47:47AM +0200, Halil Pasic wrote:
[..]
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
I have certainly understood this differently. When
VIRTIO_NET_F_VQ_NOTF_COALis not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).If
VIRTIO_NET_F_VQ_NOTF_COALis negotiated the device is supposed to/MUST do the coalescing according to the parameters as described by the virtio spec.Michael, Jason: Can you guys weigh in on this?
Quoted 8 lines
Quoted 5 lines
I still don't understand why this change is needed. We have this text:
The device may generate notifications more or less frequently than specified by set commands of the
VIRTIO_NET_CTRL_NOTF_COALclass.The less frequently could be understood like due to notification suppression.
Quoted 7 lines
I think you are missing that there is a separate text about notification suppression immediately after it:
A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
I think that makes it clear this text is separate from suppression.
No I did not miss it, and I don't think that the two sentences being in adjacent paragraphs makes it clear that this is separate from suppression.
For me the "notification conditions" part and the "even" is making it sound like normally meeting these conditions would mean that a notification should be sent but then notification suppression overrules that, so there will be no notification even though if it wasn't for the suppression there would be one.
[..]
Quoted 7 lines
OK, I guess we can clarify:
in particular, the device MAY coalesce notifications when coalescing parameters are set to 0.will that make it better?
I don't think so. Sorry, there are things I would like to understand before committing. Back then when I voted on the coalescing I convinced myself that the interaction between used buffer notification suppression and used buffer notification coalescing on the TX and RX queues of virtio-net is specified with enough clarity. Now I'm asking myself if I came to that conclusion along the lines "nothing else would make any sense, not spelling it out properly is not tragic". Please see my comment on Jason's proposal.
Regards,
Halil
Quoted 336 lines
From: Si-Wei Liu <[email protected]>
Sent: Wednesday, June 26, 2024 6:44 AMQuoted 82 lines · Parav Pandit
Hi Si-Wei,
Quoted 80 lines
From: Si-Wei Liu <[email protected]>
Sent: Tuesday, June 25, 2024 10:22 AMQuoted 2 lines · Heng Qi
On Fri, 21 Jun 2024 16:46:27 -0700, "Si-Wei
Liu"<[email protected]>Quoted 3 lines · wrote:
Quoted 3 lines · Heng Qi
Quoted 2 lines
On Thu, 20 Jun 2024 18:21:03 -0700, "Si-Wei
Liu"<[email protected]>Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines · Heng Qi
Quoted 1 line
On Mon, 17 Jun 2024 16:31:25 -0700, "Si-Wei Liu"<si-
Quoted 2 lines · [email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines · Heng Qi
Quoted 1 line
On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S.
Quoted 10 lines · Tsirkin
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines · Parav Pandit
Quoted 9 lines
Quoted 7 lines
From: Michael S. Tsirkin<[email protected]>
Sent: Tuesday, June 11, 2024 10:00 PM How we can we make
progress with the realease but sure we don't make backwards
compat a pain?Ideas?
There is no functional break with this relaxation. Device set some non-zero defaults and driver didn't modify
them....
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Anything broken? Unlikely.
Generally it's inappropriate to leave this decision making to the device for what would be the best / most performant default config, as the device is generally considered agnostic to the guest
load...
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Instead, the performance of the virtual machine and the driver depends heavily on how the device is implemented, just as we have proposed various ways to offload the data queue in the device to the hardware. The reason why most devices use software to simulate ctrlq instead of using hardware offload is that the driver has no requirements for the performance of ctrlq before, that is, the device implementation is responsible for and meets the driver's
performance requirements.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
I am not sure I follow the arguments of ctrlq being s/w or h/w, I thought that it was for the debate why we need coalescing on hardware offload device in the first place, instead of reusing event index or similar s/w based notification suppression. I don't doubt the value of coalescing, but how's it relevant to the default value
disposition?
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Generally the default config disposition is not considered to be part of device implementation, especially when it comes to the situation where device can't easily figure out the specific workload to occur in the guest, and there's no perfect single
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
default value that could meet every single performance metric across the board. This is a typical tuning knob left up to the user to adjust, why the device or driver needs to set or load the initial value? The driver just needs to start with certain value, be it 0 or non-zero, which guest user can override at any point of time, depending on his/her need, and that's it! I guess I still don't understand your user case here, why device / driver default is of
such importance.
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
I've explained that, and I understand your argument is why default value is needed, and users should be able to adjust them, right?
Sounds about right. You'll soon realize that there's no perfect default that could work with everyone - in that it's just a static value to begin with, no matter whatever initial value the device comes up with, one user or another will come over to you and complain that what is loaded from the device doesn't match the workload they have, so those users are still expected to adjust manually tweaking for their own. As long as it's a tunable that guest user can control and override anytime, I don't feel it too much different what initial config the device would start with.
Quoted 3 lines
When you have a large number of customers and they buy your machines, how many users do you think have experience adjusting this value? More
below.
This is not what this proposal alone could address, as it just shifts the sheer pain to the device and host admin rather than provide anyflexibility.
Quoted 6 lines
Quoted 6 lines
Technically I believe the right solution is to seek adaptive coalescing that doesn't require user to tune anything. Noted adaptive coalescing / suppression could be done from the guest or by the device. For e.g. the s/w based event index based notification suppression could also work with hardware if the event index is somehow propagated through a doorbell register rather than load from
host memory over the PCI transport.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
The default value is working when the user doesn't adjust them. It's not practical to rely entirely on user adjustment,
That's where the adaptive interrupt moderation framework (e.g. DIM for
Linux) could come to the play, I think?Last time, this work is only useful if DIM is not enabled. I don't want to explain it again, I've said it many times to different people in the history discussion, don't you want to check any history
discussion?
Quoted 7 lines
Quoted 7 lines
I did before I replied. I thought both Halil and Michael tried hard to convey is that what you claimed to work by just getting the initial config loaded from device will not be sufficient to address all the performance requirements in broad and general means other than satisfy the goal set by yourself. What I saw is that you kept proving yourself by trying to reference your own use case and design
- this not just mixed different things up but also ended up with circularargument that wasted everyone's time unfortunately.
Quoted 33 lines
Quoted 33 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
and for devices
that serve hundreds of thousands of customers, the device implementation has to be comprehensive.Quoted 9 lines
Quoted 9 lines
Would you mind elaborating the technical difficulty for why the device implementation has to be comprehensive to serve hundreds or thousands of customers (i.e. adaptive moderation in the device), and is there better way to improve the current interface (ctrlq v.s. doorbell/MMIO register) that is hardware offload implementation wise. I feel people reading the thread without understanding the full background would become even more confused as to why it's relevant to your proposal, to me it's completely distinct use case or problem area that
we are talking about.
Quoted 4 lines
Regardless of whether DIM can be enabled by default (it cannot accelerate all scenarios, right?), therefore, a considerable number of machines on the cloud vendors' line still have to rely on static values to provide boot-up performance.
Sorry, this is not a good excuse for people to accept a sub-optimal solution. Please talk to the other hardware vendor or read my above reply carefully.
-Siwei
Quoted 10 lines
The vendors will also make optimizations to optimize scenarios
like ping-pong, so the boot-up performance is very good.Thanks.
Quoted 5 lines
Thanks,
-SiweiQuoted 2 lines
Thanks.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Unless the
device is specially hard wired to some fixed guest setup that users couldn't change, it doesn't seem logical that the device could derive the best or most performant config on driver's behalf. What if the guest wants best latency for its load but the device just blindlessly guess the guest might prefer throughput friendly that it miserably uses latency impacting non-zero default?
Quoted 19 lines
Quoted 19 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
The device does not want to guess and cannot guess. This patch does not force the device to choose a non-zero value, but relaxes it to allow the device to choose 0 or non-zero, which is very friendly to virtual machines with different performance requirements,
right?
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
I don't understand the friendly part - do you imply your VM users are on kinda fixed wired setup that they cannot change these coalescing parameters after driver is loaded? Can the owner of the VM in control apply certain initial config for the coalescing parameters to the VM image? Or is it the problem of the guest driver that doesn't yet expose coalescing parameters to the end user? Otherwise I would think that guest user should be able to set parameters accordingly that would best fit the specific performance requirement of their own. How the device could even help here? I don't feel there's a lot of value to grant the device or host admin the flexibility to policy the *best* config on guest user's behalf, to be honest. And you seem to admit the fact that the default doesn't
Quoted 3 lines
Quoted 3 lines
really matter, be it 0 or non-zero.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Could this device side change for the default config regress boot time performance (which may need best latency over
throughput)?
Quoted 68 lines
Quoted 68 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Don't make these assumptions, what if the driver needs better
throughput?
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
There's a misconception here: what we think that driver may need in terms on performance does actually reflect what guest user would like to have. Driver cannot read guest user's mind to make the decision,
either.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
In history there was drastic change in the Linux virtio-net driver that ever changed the default disposition for XPS (and RPS as well) from throughput and long-lived connection oriented to concurrency and short-lived oriented, which regressed a lot of existing setups that expects sustained throughput and packet rate after kernel
upgrade.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Although the occurrence of such drastic change for default disposition is not so welcome (that is one of the reasons why I valued consistent initial value and back compatible behavior), I don't see people yelling at virtio spec for less flexibility of offering the default disposition, given that the guest user can override the config any time with their own tooling or script, there's no problem at all for them to just set the corresponding config
back explicitly to what it was before.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
And device/driver has better performance, is that a problem?
Unlikely.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Even for rare case with a hard wired setup, the way to tackle the very problem using device's default is still quite questionable. Usually the mgmt software or network config utility should be equipped with some default value if need be. And we know the guest has the best position to impose the best / most performant config for its own load. What is the issue or use case that this initial config couldn't be applied by the guest mgmt software ahead but has to resort to the device to load some default (which is odd and irrelevant to any guest load), before the interface is
brought up for operation i.e. performing I/O?
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Use cases are everywhere, Alibaba Cloud, MLX and all other modern network cards have a default value that is not 0.
You seems to be referencing your own setup basically, and the question is still left answered - why the initial config can't be done through the mgmt software or network config utility within the
guest?
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
(0 is actually a kind of default value)
Quoted 5 lines
Quoted 5 lines
Sorry, my vacation just ended.
Quoted 2 lines
Yes, it is possible. Driver can cache values it sets and never query device with get.
Don't we already have a lot of behaviors to drive queries from
devices?
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 6 lines
RSS context, device stats.
Quoted 1 line
Before anything is set, driver will report incorrect values.
Devices that are widely supported and supported by good practices should have any initialization value. Just reporting 0 is incorrect value. Although the spec now says so.
I don't have an aligned view here, sorry. As I recall having 0 as the default is just to keep device started in a state where coalescing is disabled, so it's backward compatible and consistent with a non-coalescing supporting driver - such that it won't yield surprising effect (for e.g. regressed latency) inadvertently after user's getting driver software upgraded. Unlike the other virtio-net features that could 100% improve performance in all aspect, this coalescing feature is more of a performance tuning knob that may improve performance metrics (such as cpu usage or throughput) of one dimension while demoting
Quoted 3 lines
Quoted 3 lines
the others (such as latency, jitter or connection rate) from the equation.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
That said, there's not a single and fixed set of default config that device could supply which is able to satisfy all kind of guest
load.
Quoted 16 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Rather than rely on the device to offer a matching default for driver (which I think it's technically wrong), I'd lean to having guest software or network utility to apply the initial config for the guest, where they should have best knowledge for the specific guest workload than what device could do.
Before this feature, a good device implementation should also support coalescing (of course we don't necessarily assume it has
coalescing).
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Again, I don't doubt the value of supporting coalescing.
Quoted 5 lines
In addition, virtual
machines that tend to favor latency and throughput exist. If the device supported by the manufacturer needs to provide a low-latency virtual machine, please continue to keep the default valueof 0.
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
No, that's not what I was asking. There's no such requirement for any vendor to provide a low-latency or high throughput VM. The more general use case is - the setup for real world workload might just be too complex that end users would prefer low-latency
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
on some virtual NIC or even some specific queues, while the other queues of a virtual NIC, or reset virtual NIC might have very different dispositions. Due to the needs and dynamics of workload scaling up & down, they might have more or less queues or virtual NICs to configure, so these disposition would need to be readjusted at any point of time, for which there's no easy way for device to adapt to easily. The guest user should have best knowledge for the specific guest workload and setup than what device
could/should offer.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
What will break as a result? Hard to predict.
Given the patch is big, I am inclined to say it just should use a feature bit.
This doesn't seem to break anything in my opinion, we just told the device that now you can set more values.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Another thing this could break is live migration between devices with varied default. How do you make sure the guest doesn't rely on some default from the source device, while on the destination it just doesn't get the same default coalescingjjj value? To get rid of this side effect the guest would still need to apply the initial config for its own, anyway... Which eventually would render this proposal with
arbitrary default rather pointless.
Quoted 16 lines
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
I don't quite understand why this would affect hot migration, the values would be migrated over.
Then I don't see this has been discussed (wouldn't the initial value be part of virtio device state to migrate over?) in the thread or described in the proposed text. And your proposed per-queue coalescing parameters (default plus current value) would have to be described too as part of virtio-net device state, so that
people don't misunderstand your proposal.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Thanks,
-SiweiQuoted 4 lines
Thanks.
Quoted 2 lines
Thanks,
-SiweiQuoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Using new feature bits does not seem necessary.
Thanks.
Quoted 3 lines
--
MSTI saw the need of this proposal slightly differently in the discussion with Heng
in v4.
Quoted 1 line
The way I understood is, proposed relaxation enables below Linux driver
flow to work as equally as without device offering
VIRTIO_NET_F_VQ_NOTF_COAL.Quoted 7 lines
Flow is:
1. The device offered featureVIRTIO_NET_F_VQ_NOTF_COAL2. The virtio-net driver negotiatedVIRTNET_FEATURESthat hasVIRTIO_NET_F_VQ_NOTF_COAL3. Because
VIRTIO_NET_F_VQ_NOTF_COALis negotiated, device is notapplying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
Quoted 2 lines
4. the virtio-net driver even after negotiating
VIRTIO_NET_F_VQ_NOTF_COAL, still kept the its dim disabled.Quoted 2 lines
vi->rq[queue].dim_enabled = false by default.
Quoted 7 lines
Quoted 3 lines
5. The virtio net driver enables dim only on user request via ethtool.
So here, just because one is using a new driver and new device, it may get
lower performance.
Impossible, older driver / device doesn't have coalescing advertise / enabled, and new device & new driver with coalescing negotiated starts safe with 0 with coalescing disabled effectively, how can it get lower performance?
The current Linux driver does not even have the code for netdim for tx, not enabling netdim by default. And enabling netdim for rx requires user space.
So in it current form, the device implementation wont apply any notification suppression (which otherwise it did in absence of this feature).
Quoted 36 lines
Quoted 3 lines
Do we agree to above problem description?
This narrative has two problems:
- The term "lower performance" is misleading and inaccurate, the fact or implication of coalescing is that it yields worse latency, causing additional delay and jitter that would hurt quite a lot of real world workload that is latency sensitive. So, rather than say coalescing helps performance, it should clearly tell the truth that the coalescing parameter is no more than a tuning knob that has performance impact/implication, being positive or negative it is very subject to the guest workload. So the default disposition or initial parameters loaded from device just reflects the preference of the host admin/ device owner/ cloud vendor of one's own, for e.g. it is optimized for throughput oriented load versus latency oriented. Be noted, generally end users don't care what coalescing is about, they do not connect performance to "throughput" or "bandwidth" as that in your mind.
- The specific problem or use case is narrow scoped and specifically tailored (e.g. for a hard wired guest setup that is known to the device owner / host admin already), while for some reason the advantage of the proposed solution is overly exaggerated. For one hand, the ethtool interface is very Linux specific that exposed quite a lit lower level device implementation details to normal end users. In case of static and fixed configuration, it definitely requires dedicated knowledge of the user to fine tune for their load. For the other hand, if we are talking about normal users who don't have such knowledge AND don't care so much about performance, then whatever coalescing parameter will just work with them, and they don't really care about the default. However, For those normal users who care about performance but still care about getting the best or well balanced performance (N.B. I say performance in general, not limited to only throughput or latency in one dimension) for their load, what options do they have? IMHO what they really like is to lean on guest/driver/device to auto tune by itself rather than for them to take the deep learning curve to fiddle with the low level device parameters manually by themselves, right?
Quoted 7 lines
If you agree with the above, I guess we could proceed with unlocking the other good options, as listed below. To me, the question I got from the above problem statement but still left unanswered is that, what makes the cloud vendor think that they could have better posture to decide certain default values for guest users' behalf? Why it is a MUST to do so for every cloud vendor in general?
It likely does not.
The intent is to restore the behavior to same as what it used to be when VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated.
Quoted 24 lines
Thanks,
-SiweiQuoted 5 lines
If no, I likely missed something in the long discussion and need to read all of it. :(
If yes, few solutions are:
1. A user intervention is must to avoid this regression. The user in the guestVM must enable DIM if the device supports it.
Quoted 3 lines
Very hard to do this plumbing.
2. A net driver enables the DIM by default as long as
VIRTIO_NET_F_VQ_NOTF_COALis negotiated.Quoted 3 lines
(without ethtool setting)
3. A device relaxes the limitation and continue to apply the coalescing, until
driver overrides it.
Quoted 2 lines
In my humble opinion, Heng is solving it using option #3, that tends to work
with existing and future drivers who may/may not enable the DIM.
Quoted 4 lines
Quoted 1 line · Parav Pandit
The intent is to restore the behavior to same as what it used to be when
VIRTIO_NET_F_VQ_NOTF_COALis not negotiated.
This is not what the commit log or the github issue say though. The more I look at the change the less I understand it.
Can the issue be clarified please?
Current spec text ABC is bad for performance because DEF, change it to GHI - as a result device can get better performance by JKL.
Signature · 1 line
MST
Quoted 49 lines · Si-Wei Liu
Quoted 43 lines · Heng Qi
Quoted 26 lines · Si-Wei Liu
Quoted 17 lines · Heng Qi
Quoted 8 lines · Si-Wei Liu
Quoted 7 lines · Heng Qi
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Parav Pandit
Quoted 5 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 11, 2024 10:00 PM
How we can we make progress with
the realease but sure we don't make backwards compat a pain?Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Ideas?
There is no functional break with this relaxation. Device set some non-zero defaults and driver didn't modify them.... Anything broken? Unlikely.
Generally it's inappropriate to leave this decision making to the device for what would be the best / most performant default config, as the device is generally considered agnostic to the guest load...
Quoted 7 lines
Instead, the performance of the virtual machine and the driver depends heavily on how the device is implemented, just as we have proposed various ways to offload the data queue in the device to the hardware. The reason why most devices use software to simulate ctrlq instead of using hardware offload is that the driver has no requirements for the performance of ctrlq before, that is, the device implementation is responsible for and meets the driver's performance requirements.
Quoted 16 lines
I am not sure I follow the arguments of ctrlq being s/w or h/w, I thought that it was for the debate why we need coalescing on hardware offload device in the first place, instead of reusing event index or similar s/w based notification suppression. I don't doubt the value of coalescing, but how's it relevant to the default value disposition? Generally the default config disposition is not considered to be part of device implementation, especially when it comes to the situation where device can't easily figure out the specific workload to occur in the guest, and there's no perfect single default value that could meet every single performance metric across the board. This is a typical tuning knob left up to the user to adjust, why the device or driver needs to set or load the initial value? The driver just needs to start with certain value, be it 0 or non-zero, which guest user can override at any point of time, depending on his/her need, and that's it! I guess I still don't understand your user case here, why device / driver default is of such importance.
Quoted 2 lines
I've explained that, and I understand your argument is why default value is needed, and users should be able to adjust them, right?
Quoted 9 lines
Sounds about right. You'll soon realize that there's no perfect default that could work with everyone - in that it's just a static value to begin with, no matter whatever initial value the device comes up with, one user or another will come over to you and complain that what is loaded from the device doesn't match the workload they have, so those users are still expected to adjust manually tweaking for their own. As long as it's a tunable that guest user can control and override anytime, I don't feel it too much different what initial config the device would start with.
Think hard of these, I think there's probably no conflict between the points of each.
Generally, it's very hard to have a default value for virtio-net moderation as
1) various implementations of virtio-net
2) various setups of virtio-net (from virtual to physical, from 1G to 100G)
3) various kind of workloads
But it doesn't mean the device can not have something by default which the vendor thinks it works well for in most cases. And anyhow the value could be overridden by the driver if it wants (e.g several Linux networking drivers will try to set a good default moderation value during probe).
Thanks
Quoted 69 lines · Jason Wang
Quoted 49 lines · Si-Wei Liu
Quoted 43 lines · Heng Qi
Quoted 26 lines · Si-Wei Liu
Quoted 17 lines · Heng Qi
Quoted 8 lines · Si-Wei Liu
Quoted 7 lines · Heng Qi
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Parav Pandit
Quoted 5 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 11, 2024 10:00 PM
How we can we make progress with
the realease but sure we don't make backwards compat a pain?Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Ideas?
There is no functional break with this relaxation. Device set some non-zero defaults and driver didn't modify them.... Anything broken? Unlikely.
Generally it's inappropriate to leave this decision making to the device for what would be the best / most performant default config, as the device is generally considered agnostic to the guest load...
Quoted 7 lines
Instead, the performance of the virtual machine and the driver depends heavily on how the device is implemented, just as we have proposed various ways to offload the data queue in the device to the hardware. The reason why most devices use software to simulate ctrlq instead of using hardware offload is that the driver has no requirements for the performance of ctrlq before, that is, the device implementation is responsible for and meets the driver's performance requirements.
Quoted 16 lines
I am not sure I follow the arguments of ctrlq being s/w or h/w, I thought that it was for the debate why we need coalescing on hardware offload device in the first place, instead of reusing event index or similar s/w based notification suppression. I don't doubt the value of coalescing, but how's it relevant to the default value disposition? Generally the default config disposition is not considered to be part of device implementation, especially when it comes to the situation where device can't easily figure out the specific workload to occur in the guest, and there's no perfect single default value that could meet every single performance metric across the board. This is a typical tuning knob left up to the user to adjust, why the device or driver needs to set or load the initial value? The driver just needs to start with certain value, be it 0 or non-zero, which guest user can override at any point of time, depending on his/her need, and that's it! I guess I still don't understand your user case here, why device / driver default is of such importance.
Quoted 2 lines
I've explained that, and I understand your argument is why default value is needed, and users should be able to adjust them, right?
Quoted 9 lines
Sounds about right. You'll soon realize that there's no perfect default that could work with everyone - in that it's just a static value to begin with, no matter whatever initial value the device comes up with, one user or another will come over to you and complain that what is loaded from the device doesn't match the workload they have, so those users are still expected to adjust manually tweaking for their own. As long as it's a tunable that guest user can control and override anytime, I don't feel it too much different what initial config the device would start with.
Think hard of these, I think there's probably no conflict between the points of each.
Generally, it's very hard to have a default value for virtio-net moderation as
1) various implementations of virtio-net
2) various setups of virtio-net (from virtual to physical, from 1G to 100G)
3) various kind of workloads
Quoted 7 lines
But it doesn't mean the device can not have something by default which the vendor thinks it works well for in most cases. And anyhow the value could be overridden by the driver if it wants (e.g several Linux networking drivers will try to set a good default moderation value during probe).
Thanks
Do we want to just burn a feature bit on this then? I also think we need to have a way to reset to that default value. Maybe setting #packets to 0 should do that?
Signature · 1 line
MST
Quoted 77 lines · Michael S. Tsirkin
Quoted 75 lines · Jason Wang
Quoted 49 lines · Si-Wei Liu
Quoted 43 lines · Heng Qi
Quoted 26 lines · Si-Wei Liu
Quoted 17 lines · Heng Qi
Quoted 8 lines · Si-Wei Liu
Quoted 7 lines · Heng Qi
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Parav Pandit
Quoted 5 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 11, 2024 10:00 PM
How we can we make progress with
the realease but sure we don't make backwards compat a pain?Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Ideas?
There is no functional break with this relaxation. Device set some non-zero defaults and driver didn't modify them.... Anything broken? Unlikely.
Generally it's inappropriate to leave this decision making to the device for what would be the best / most performant default config, as the device is generally considered agnostic to the guest load...
Quoted 7 lines
Instead, the performance of the virtual machine and the driver depends heavily on how the device is implemented, just as we have proposed various ways to offload the data queue in the device to the hardware. The reason why most devices use software to simulate ctrlq instead of using hardware offload is that the driver has no requirements for the performance of ctrlq before, that is, the device implementation is responsible for and meets the driver's performance requirements.
Quoted 16 lines
I am not sure I follow the arguments of ctrlq being s/w or h/w, I thought that it was for the debate why we need coalescing on hardware offload device in the first place, instead of reusing event index or similar s/w based notification suppression. I don't doubt the value of coalescing, but how's it relevant to the default value disposition? Generally the default config disposition is not considered to be part of device implementation, especially when it comes to the situation where device can't easily figure out the specific workload to occur in the guest, and there's no perfect single default value that could meet every single performance metric across the board. This is a typical tuning knob left up to the user to adjust, why the device or driver needs to set or load the initial value? The driver just needs to start with certain value, be it 0 or non-zero, which guest user can override at any point of time, depending on his/her need, and that's it! I guess I still don't understand your user case here, why device / driver default is of such importance.
Quoted 2 lines
I've explained that, and I understand your argument is why default value is needed, and users should be able to adjust them, right?
Quoted 9 lines
Sounds about right. You'll soon realize that there's no perfect default that could work with everyone - in that it's just a static value to begin with, no matter whatever initial value the device comes up with, one user or another will come over to you and complain that what is loaded from the device doesn't match the workload they have, so those users are still expected to adjust manually tweaking for their own. As long as it's a tunable that guest user can control and override anytime, I don't feel it too much different what initial config the device would start with.
Think hard of these, I think there's probably no conflict between the points of each.
Generally, it's very hard to have a default value for virtio-net moderation as
1) various implementations of virtio-net
2) various setups of virtio-net (from virtual to physical, from 1G to 100G)
3) various kind of workloadsBut it doesn't mean the device can not have something by default which the vendor thinks it works well for in most cases. And anyhow the value could be overridden by the driver if it wants (e.g several Linux networking drivers will try to set a good default moderation value during probe).
Quoted 3 lines
Quoted 1 line
Thanks
Do we want to just burn a feature bit on this then?
It might be more safe to do that.
Quoted 2 lines
I also think we need to have a way to reset to that default value. Maybe setting #packets to 0 should do that?
Or a dedicated command.
Thanks
Quoted 5 lines
--
MST
Quoted 4 lines · Jason Wang
Quoted 2 lines
I also think we need to have a way to reset to that default value. Maybe setting #packets to 0 should do that?
Or a dedicated command.
Kind of seems like an overkill, no?
Signature · 1 line
MST
Quoted 32 lines · Heng Qi
The device can set any initial coalescing parameters (0 or non-zero) for the receive/send queue before the setting command is executed, not just 0, enhancing device performance even without DIM enabled.
So we need to clarify descriptions that don't fit the behavior.
Suggested-by Jason Wang <[email protected]>Signed-off-by Heng Qi <[email protected]>Reviewed-by Parav Pandit <[email protected]>Reviewed-by Jason Wang <[email protected]>---
v4->v5:
- Add RB tags from Parav and Jason. Thanks!
v3->v4:
- Doesn't force the device to remember more stuff. @Parav
v2->v3:
- Clarify description to be more generic. @Parav
v1->v2:
- Update description. @Jasondevice-types/net/description.tex | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)device-types/net/description.tex +0
-0
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1808
Quoted 6 lines
device-types/net/description.tex +3
-0@@ -1803,6 +1803,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi for an enabled transmit/receive virtqueue whose index is \field{vq_index}. \end{enumerate} +If the VIRTIO_NET_F_NOTF_COAL or VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, +the device may apply any coalescing parameters to each transmit/receive virtqueue +before the driver successfully performs one of the VIRTIO_NET_CTRL_NOTF_COAL set commands.
may outside of conformance section, not good.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1813
Quoted 5 lines
device-types/net/description.tex +5
-0+ +The driver can query the coalescing parameters of any enabled transmit/receive +virtqueue using the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command, before or after any +VIRTIO_NET_CTRL_NOTF_COAL set command is done. +
Vague. There are just 3 set commands ust list them.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1816
Quoted 3 lines
device-types/net/description.tex +0
-0The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class. If coalescing parameters are being set, the device applies the last coalescing parameters set for a
This needs more work but I guess we can skip the above part.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1897
Quoted 7 lines
device-types/net/description.tex +1
-1@@ -1885,17 +1893,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue. Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0. +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, the device MAY initialize them to any values.
If device does not initialize them then what? Actually something like
0 -> the default TX coalescing parameters chosen by the device?
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1906
Quoted 11 lines
device-types/net/description.tex +2
-2Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0. +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, the device MAY initialize them to any values. The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort: the device MAY generate notifications more or less frequently than specified. A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. -Upon reset, a device MUST initialize all coalescing parameters to 0. +Upon reset, a device MAY set any coalescing parameters for all transmit and receive virtqueues.
Confusing.
This seems to allow any vq to have a different set of parameters. Now if driver calls VIRTIO_NET_CTRL_NOTF_COAL_RX_SET then what?
Quoted 9 lines
device-types/net/description.tex +0
-0\paragraph{Device Statistics}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Statistics}Signature · 2 lines
-- 2.32.0.3.g01195cf9f
Quoted 36 lines · Michael S. Tsirkin
Quoted 35 lines · Heng Qi
The device can set any initial coalescing parameters (0 or non-zero) for the receive/send queue before the setting command is executed, not just 0, enhancing device performance even without DIM enabled.
So we need to clarify descriptions that don't fit the behavior.
Suggested-by Jason Wang <[email protected]>Signed-off-by Heng Qi <[email protected]>Reviewed-by Parav Pandit <[email protected]>Reviewed-by Jason Wang <[email protected]>---
v4->v5:
- Add RB tags from Parav and Jason. Thanks!
v3->v4:
- Doesn't force the device to remember more stuff. @Parav
v2->v3:
- Clarify description to be more generic. @Parav
v1->v2:
- Update description. @Jasondevice-types/net/description.tex | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)device-types/net/description.tex +0
-0@@ -1803,6 +1803,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi for an enabled transmit/receive virtqueue whose index is \field{vq_index}. \end{enumerate}
Quoted 5 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1808
Quoted 3 lines
device-types/net/description.tex +3
-0+If the VIRTIO_NET_F_NOTF_COAL or VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, +the device may apply any coalescing parameters to each transmit/receive virtqueue +before the driver successfully performs one of the VIRTIO_NET_CTRL_NOTF_COAL set commands.may outside of conformance section, not good.
This is outside of the conformance section. Do you want inside?
Quoted 18 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1813
Quoted 6 lines
device-types/net/description.tex +5
-0+ +The driver can query the coalescing parameters of any enabled transmit/receive +virtqueue using the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command, before or after any +VIRTIO_NET_CTRL_NOTF_COAL set command is done. +Vague. There are just 3 set commands ust list them.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1816
Quoted 3 lines
device-types/net/description.tex +0
-0The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class. If coalescing parameters are being set, the device applies the last coalescing parameters set for aThis needs more work but I guess we can skip the above part.
Quoted 3 lines
device-types/net/description.tex +0
-0@@ -1885,17 +1893,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
Quoted 6 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1897
Quoted 4 lines
device-types/net/description.tex +1
-1Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0. +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, the device MAY initialize them to any values.If device does not initialize them then what?
The device's customized behavior, at least, does not conform to the spec.
Quoted 2 lines
Actually something like
0 -> the default TX coalescing parameters chosen by the device?
Deefault coalescing parameters should also have a concrete "value".
Quoted 9 lines
Quoted 9 lines
device-types/net/description.tex +1
-1Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0. +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, the device MAY initialize them to any values. The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort: the device MAY generate notifications more or less frequently than specified.
Quoted 7 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1906
Quoted 4 lines
device-types/net/description.tex +1
-1A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. -Upon reset, a device MUST initialize all coalescing parameters to 0. +Upon reset, a device MAY set any coalescing parameters for all transmit and receive virtqueues.Confusing.
This seems to allow any vq to have a different set of parameters.
YES.
Quoted 2 lines
Now if driver calls
VIRTIO_NET_CTRL_NOTF_COAL_RX_SETthen what?
Then the coalescing parameters of all RX queues are updated to the values set by VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, and the coalescing parameters of all TX queues continue to retain the previous values.
Thanks.
Quoted 11 lines
Quoted 10 lines
device-types/net/description.tex +0
-0\paragraph{Device Statistics}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Statistics}Signature · 2 lines
-- 2.32.0.3.g01195cf9f
Quoted 37 lines · Heng Qi
Quoted 36 lines · Michael S. Tsirkin
Quoted 35 lines · Heng Qi
The device can set any initial coalescing parameters (0 or non-zero) for the receive/send queue before the setting command is executed, not just 0, enhancing device performance even without DIM enabled.
So we need to clarify descriptions that don't fit the behavior.
Suggested-by Jason Wang <[email protected]>Signed-off-by Heng Qi <[email protected]>Reviewed-by Parav Pandit <[email protected]>Reviewed-by Jason Wang <[email protected]>---
v4->v5:
- Add RB tags from Parav and Jason. Thanks!
v3->v4:
- Doesn't force the device to remember more stuff. @Parav
v2->v3:
- Clarify description to be more generic. @Parav
v1->v2:
- Update description. @Jasondevice-types/net/description.tex | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)device-types/net/description.tex +0
-0@@ -1803,6 +1803,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi for an enabled transmit/receive virtqueue whose index is \field{vq_index}. \end{enumerate}
Quoted 7 lines
Quoted 5 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1808
Quoted 3 lines
device-types/net/description.tex +3
-0+If the VIRTIO_NET_F_NOTF_COAL or VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, +the device may apply any coalescing parameters to each transmit/receive virtqueue +before the driver successfully performs one of the VIRTIO_NET_CTRL_NOTF_COAL set commands.may outside of conformance section, not good.
This is outside of the conformance section. Do you want inside?
needs to be rephrased to avoid using reserved keywords.
Quoted 18 lines
Quoted 18 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1813
Quoted 6 lines
device-types/net/description.tex +5
-0+ +The driver can query the coalescing parameters of any enabled transmit/receive +virtqueue using the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command, before or after any +VIRTIO_NET_CTRL_NOTF_COAL set command is done. +Vague. There are just 3 set commands ust list them.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1816
Quoted 3 lines
device-types/net/description.tex +0
-0The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class. If coalescing parameters are being set, the device applies the last coalescing parameters set for aThis needs more work but I guess we can skip the above part.
Quoted 3 lines
device-types/net/description.tex +0
-0@@ -1885,17 +1893,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
Quoted 8 lines
Quoted 6 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1897
Quoted 4 lines
device-types/net/description.tex +1
-1Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0. +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, the device MAY initialize them to any values.If device does not initialize them then what?
The device's customized behavior, at least, does not conform to the spec.
Then why do you say "may"?
Quoted 4 lines
Quoted 2 lines
Actually something like
0 -> the default TX coalescing parameters chosen by the device?Deefault coalescing parameters should also have a concrete "value".
and I just proposed how to say this in a clear way.
Quoted 19 lines
Quoted 9 lines
Quoted 9 lines
device-types/net/description.tex +1
-1Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0. +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, the device MAY initialize them to any values. The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort: the device MAY generate notifications more or less frequently than specified.Quoted 7 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1906
Quoted 4 lines
device-types/net/description.tex +1
-1A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. -Upon reset, a device MUST initialize all coalescing parameters to 0. +Upon reset, a device MAY set any coalescing parameters for all transmit and receive virtqueues.Confusing.
This seems to allow any vq to have a different set of parameters.YES.
Quoted 8 lines
Quoted 2 lines
Now if driver calls
VIRTIO_NET_CTRL_NOTF_COAL_RX_SETthen what?Then the coalescing parameters of all RX queues are updated to the values set by
VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, and the coalescing parameters of all TX queues continue to retain the previous values.Thanks.
It's not at all nice that there's apparently no way for driver to get back to the default values.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
device-types/net/description.tex +0
-0\paragraph{Device Statistics}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Statistics}Signature · 2 lines
-- 2.32.0.3.g01195cf9f