VIRTIO_F_IO_BARRIER: use I/O barriers in driver

PATCH In review Tiwei Bie 4 participants 12 messages May 10, 2018 – Jun 20, 2018

Share & tools

Tiwei Bie

author

2018-05-10 15:41 UTC #0 lore raw

PATCH v3 1/1 VIRTIO_F_IO_BARRIER: use I/O barriers in driver

There will be hardware virtio devices in the future, which require drivers to use the barriers suitable for I/O devices, compared with software virtio devices which just require drivers to use the barriers suitable for CPU cores.

To fix the ordering issue for hardware virtio devices, add a new feature: VIRTIO_F_IO_BARRIER. When negotiated, driver will use the barriers suitable for I/O devices.

Signed-off-by Tiwei Bie <[email protected]>

Reviewed-by Stefan Hajnoczi <[email protected]>

---
v2 -> v3:
- Update the feature bits allocation (Stefan);

v1 -> v2:
- Rebase to the latest spec (MST);
- Use a smaller textwidth (according to _vimrc);

RFC -> v1:
- Use plural (Stefan);
- Add more details (Stefan);

 content.tex | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

content.tex +18 -2

index 7a92cb1..95c243f 100644
@@ -95,10 +95,10 @@ Feature bits are allocated as follows:
 \begin{description}
 \item[0 to 23] Feature bits for the specific device type
 
-\item[24 to 33] Feature bits reserved for extensions to the queue and
+\item[24 to 36] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[34 and above] Feature bits reserved for future extensions.
+\item[37 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}

@@ -5348,6 +5348,15 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
   \item[VIRTIO_F_IN_ORDER(35)] This feature indicates
   that all buffers are used by the device in the same
   order in which they have been made available.
+  \item[VIRTIO_F_IO_BARRIER(36)] This feature indicates
+  that the device needs the driver to use the barriers
+  suitable for hardware devices.  Some transports require
+  barriers to ensure devices have a consistent view of
+  memory.  When devices are implemented in software a
+  weaker form of barrier may be sufficient and yield
+  better performance.  This feature indicates whether
+  a stronger form of barrier suitable for hardware
+  devices is necessary.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}

@@ -5363,6 +5372,10 @@ addresses to the device.
 
 A driver SHOULD accept VIRTIO_F_RING_PACKED if it is offered.
 
+A driver SHOULD accept VIRTIO_F_IO_BARRIER if it is offered.
+If VIRTIO_F_IO_BARRIER has been negotiated, a driver MUST use
+the barriers suitable for hardware devices.
+
 \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
 
 A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further

@@ -5376,6 +5389,9 @@ accepted.
 If VIRTIO_F_IN_ORDER has been negotiated, a device MUST use
 buffers in the same order in which they have been available.
 
+A device MAY fail to operate further if VIRTIO_F_IO_BARRIER
+is not accepted.
+
 \section{Legacy Interface: Reserved Feature Bits}\label{sec:Reserved Feature Bits / Legacy Interface: Reserved Feature Bits}
 
 Transitional devices MAY offer the following:

Signature · 6 lines

2.11.0


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Stefan Hajnoczi

Reviewed-by on v3 1/1

2018-05-15 09:40 UTC #1 lore raw

Quoted 19 lines · Tiwei Bie

There will be hardware virtio devices in the future, which require drivers to use the barriers suitable for I/O devices, compared with software virtio devices which just require drivers to use the barriers suitable for CPU cores.
To fix the ordering issue for hardware virtio devices, add a new feature: VIRTIO_F_IO_BARRIER. When negotiated, driver will use the barriers suitable for I/O devices.
Signed-off-by Tiwei Bie <[email protected]>
Reviewed-by Stefan Hajnoczi <[email protected]>
---
v2 -> v3:
- Update the feature bits allocation (Stefan);
v1 -> v2:
- Rebase to the latest spec (MST);
- Use a smaller textwidth (according to _vimrc);

Quoted 6 lines

RFC -> v1:
- Use plural (Stefan);
- Add more details (Stefan);
 content.tex | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

Thanks for your patience!

Reviewed-by Stefan Hajnoczi <[email protected]>

Tiwei Bie

author

2018-05-15 10:23 UTC #2 lore raw

Quoted 24 lines · Stefan Hajnoczi

Quoted 23 lines · Tiwei Bie
There will be hardware virtio devices in the future, which require drivers to use the barriers suitable for I/O devices, compared with software virtio devices which just require drivers to use the barriers suitable for CPU cores.
To fix the ordering issue for hardware virtio devices, add a new feature: VIRTIO_F_IO_BARRIER. When negotiated, driver will use the barriers suitable for I/O devices.
Signed-off-by Tiwei Bie <[email protected]>
Reviewed-by Stefan Hajnoczi <[email protected]>
---
v2 -> v3:
- Update the feature bits allocation (Stefan);
v1 -> v2:
- Rebase to the latest spec (MST);
- Use a smaller textwidth (according to _vimrc);
RFC -> v1:
- Use plural (Stefan);
- Add more details (Stefan);

Quoted 6 lines

Quoted 2 lines
 content.tex | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)
Thanks for your patience!
Reviewed-by Stefan Hajnoczi <[email protected]>

It's my pleasure. Thanks for your review! :)

Best regards,
Tiwei Bie

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

Michael S. Tsirkin

2018-05-15 12:39 UTC #3 lore raw

In reply to Tiwei Bie

Quoted 11 lines · Tiwei Bie

There will be hardware virtio devices in the future, which require drivers to use the barriers suitable for I/O devices, compared with software virtio devices which just require drivers to use the barriers suitable for CPU cores.
To fix the ordering issue for hardware virtio devices, add a new feature: VIRTIO_F_IO_BARRIER. When negotiated, driver will use the barriers suitable for I/O devices.
Signed-off-by Tiwei Bie <[email protected]>
Reviewed-by Stefan Hajnoczi <[email protected]>

When you are ready, please open a github issue, then send Fixes: tag in a response to this mail, asking to start voting.

Quoted 71 lines

---
v2 -> v3:
- Update the feature bits allocation (Stefan);

v1 -> v2:
- Rebase to the latest spec (MST);
- Use a smaller textwidth (according to _vimrc);

RFC -> v1:
- Use plural (Stefan);
- Add more details (Stefan);

 content.tex | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

content.tex +18 -2

index 7a92cb1..95c243f 100644
@@ -95,10 +95,10 @@ Feature bits are allocated as follows:
 \begin{description}
 \item[0 to 23] Feature bits for the specific device type
 
-\item[24 to 33] Feature bits reserved for extensions to the queue and
+\item[24 to 36] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[34 and above] Feature bits reserved for future extensions.
+\item[37 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}

@@ -5348,6 +5348,15 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
   \item[VIRTIO_F_IN_ORDER(35)] This feature indicates
   that all buffers are used by the device in the same
   order in which they have been made available.
+  \item[VIRTIO_F_IO_BARRIER(36)] This feature indicates
+  that the device needs the driver to use the barriers
+  suitable for hardware devices.  Some transports require
+  barriers to ensure devices have a consistent view of
+  memory.  When devices are implemented in software a
+  weaker form of barrier may be sufficient and yield
+  better performance.  This feature indicates whether
+  a stronger form of barrier suitable for hardware
+  devices is necessary.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}

@@ -5363,6 +5372,10 @@ addresses to the device.
 
 A driver SHOULD accept VIRTIO_F_RING_PACKED if it is offered.
 
+A driver SHOULD accept VIRTIO_F_IO_BARRIER if it is offered.
+If VIRTIO_F_IO_BARRIER has been negotiated, a driver MUST use
+the barriers suitable for hardware devices.
+
 \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
 
 A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further

@@ -5376,6 +5389,9 @@ accepted.
 If VIRTIO_F_IN_ORDER has been negotiated, a device MUST use
 buffers in the same order in which they have been available.
 
+A device MAY fail to operate further if VIRTIO_F_IO_BARRIER
+is not accepted.
+
 \section{Legacy Interface: Reserved Feature Bits}\label{sec:Reserved Feature Bits / Legacy Interface: Reserved Feature Bits}
 
 Transitional devices MAY offer the following:

Signature · 1 line

2.11.0

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

Tiwei Bie

author

2018-05-16 01:19 UTC #4 lore raw

Quoted 10 lines · Michael S. Tsirkin

Quoted 9 lines · Tiwei Bie
There will be hardware virtio devices in the future, which require drivers to use the barriers suitable for I/O devices, compared with software virtio devices which just require drivers to use the barriers suitable for CPU cores.
To fix the ordering issue for hardware virtio devices, add a new feature: VIRTIO_F_IO_BARRIER. When negotiated, driver will use the barriers suitable for I/O devices.

Quoted 6 lines

Quoted 2 lines
Signed-off-by Tiwei Bie <[email protected]>
Reviewed-by Stefan Hajnoczi <[email protected]>
When you are ready, please open a github issue, then send Fixes: tag in a response to this mail, asking to start voting.

Got it! Thank you very much!

Best regards,
Tiwei Bie

Quoted 64 lines

---
v2 -> v3:
- Update the feature bits allocation (Stefan);

v1 -> v2:
- Rebase to the latest spec (MST);
- Use a smaller textwidth (according to _vimrc);

RFC -> v1:
- Use plural (Stefan);
- Add more details (Stefan);

 content.tex | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

content.tex +15 -2

index 7a92cb1..95c243f 100644
@@ -95,10 +95,10 @@ Feature bits are allocated as follows:
 \begin{description}
 \item[0 to 23] Feature bits for the specific device type
 
-\item[24 to 33] Feature bits reserved for extensions to the queue and
+\item[24 to 36] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[34 and above] Feature bits reserved for future extensions.
+\item[37 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}

@@ -5348,6 +5348,15 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
   \item[VIRTIO_F_IN_ORDER(35)] This feature indicates
   that all buffers are used by the device in the same
   order in which they have been made available.
+  \item[VIRTIO_F_IO_BARRIER(36)] This feature indicates
+  that the device needs the driver to use the barriers
+  suitable for hardware devices.  Some transports require
+  barriers to ensure devices have a consistent view of
+  memory.  When devices are implemented in software a
+  weaker form of barrier may be sufficient and yield
+  better performance.  This feature indicates whether
+  a stronger form of barrier suitable for hardware
+  devices is necessary.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}

@@ -5363,6 +5372,10 @@ addresses to the device.
 
 A driver SHOULD accept VIRTIO_F_RING_PACKED if it is offered.
 
+A driver SHOULD accept VIRTIO_F_IO_BARRIER if it is offered.
+If VIRTIO_F_IO_BARRIER has been negotiated, a driver MUST use
+the barriers suitable for hardware devices.
+
 \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
 
 A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further

@@ -5376,6 +5389,9 @@ accepted.
 If VIRTIO_F_IN_ORDER has been negotiated, a device MUST use
 buffers in the same order in which they have been available.

Quoted 8 lines

content.tex +3 -0

+A device MAY fail to operate further if VIRTIO_F_IO_BARRIER
+is not accepted.
+
 \section{Legacy Interface: Reserved Feature Bits}\label{sec:Reserved Feature Bits / Legacy Interface: Reserved Feature Bits}
 
 Transitional devices MAY offer the following:

Signature · 1 line

2.11.0

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

Michael S. Tsirkin

2018-06-14 00:25 UTC #5 lore raw

Quoted 14 lines · Tiwei Bie

Quoted 13 lines · Michael S. Tsirkin
Quoted 11 lines · Tiwei Bie
There will be hardware virtio devices in the future, which require drivers to use the barriers suitable for I/O devices, compared with software virtio devices which just require drivers to use the barriers suitable for CPU cores.
To fix the ordering issue for hardware virtio devices, add a new feature: VIRTIO_F_IO_BARRIER. When negotiated, driver will use the barriers suitable for I/O devices.
Signed-off-by Tiwei Bie <[email protected]>
Reviewed-by Stefan Hajnoczi <[email protected]>

Quoted 8 lines

Quoted 3 lines
When you are ready, please open a github issue, then send Fixes: tag in a response to this mail, asking to start voting.
Got it! Thank you very much!
Best regards,
Tiwei Bie

I have a question: an upstream discussion recently touched on the subject of need to use bounce buffers in special memory, flush cache, or do other tricks for DMA.

Do we want to extend this flag to cover that behaviour as well? If yes should we rename it and tweak the definition? Or should do something else?

Quoted 69 lines

---
v2 -> v3:
- Update the feature bits allocation (Stefan);

v1 -> v2:
- Rebase to the latest spec (MST);
- Use a smaller textwidth (according to _vimrc);

RFC -> v1:
- Use plural (Stefan);
- Add more details (Stefan);

 content.tex | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

content.tex +18 -2

index 7a92cb1..95c243f 100644
@@ -95,10 +95,10 @@ Feature bits are allocated as follows:
 \begin{description}
 \item[0 to 23] Feature bits for the specific device type
 
-\item[24 to 33] Feature bits reserved for extensions to the queue and
+\item[24 to 36] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[34 and above] Feature bits reserved for future extensions.
+\item[37 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}

@@ -5348,6 +5348,15 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
   \item[VIRTIO_F_IN_ORDER(35)] This feature indicates
   that all buffers are used by the device in the same
   order in which they have been made available.
+  \item[VIRTIO_F_IO_BARRIER(36)] This feature indicates
+  that the device needs the driver to use the barriers
+  suitable for hardware devices.  Some transports require
+  barriers to ensure devices have a consistent view of
+  memory.  When devices are implemented in software a
+  weaker form of barrier may be sufficient and yield
+  better performance.  This feature indicates whether
+  a stronger form of barrier suitable for hardware
+  devices is necessary.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}

@@ -5363,6 +5372,10 @@ addresses to the device.
 
 A driver SHOULD accept VIRTIO_F_RING_PACKED if it is offered.
 
+A driver SHOULD accept VIRTIO_F_IO_BARRIER if it is offered.
+If VIRTIO_F_IO_BARRIER has been negotiated, a driver MUST use
+the barriers suitable for hardware devices.
+
 \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
 
 A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further

@@ -5376,6 +5389,9 @@ accepted.
 If VIRTIO_F_IN_ORDER has been negotiated, a device MUST use
 buffers in the same order in which they have been available.
 
+A device MAY fail to operate further if VIRTIO_F_IO_BARRIER
+is not accepted.
+
 \section{Legacy Interface: Reserved Feature Bits}\label{sec:Reserved Feature Bits / Legacy Interface: Reserved Feature Bits}

Quoted 7 lines

Quoted 3 lines
Quoted 3 lines
content.tex +0 -0
 Transitional devices MAY offer the following:
Signature · 1 line
2.11.0
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

Jason Wang

2018-06-14 03:30 UTC #6 lore raw

Quoted 21 lines · Michael S. Tsirkin

Quoted 13 lines · Tiwei Bie
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Tiwei Bie
There will be hardware virtio devices in the future, which require drivers to use the barriers suitable for I/O devices, compared with software virtio devices which just require drivers to use the barriers suitable for CPU cores.
To fix the ordering issue for hardware virtio devices, add a new feature: VIRTIO_F_IO_BARRIER. When negotiated, driver will use the barriers suitable for I/O devices.
Signed-off-by Tiwei Bie<[email protected]>
Reviewed-by Stefan Hajnoczi<[email protected]>
Quoted 7 lines
Quoted 3 lines
When you are ready, please open a github issue, then send Fixes: tag in a response to this mail, asking to start voting.
Got it! Thank you very much!
Best regards,
Tiwei Bie

Quoted 8 lines

I have a question: an upstream discussion recently touched on the subject of need to use bounce buffers in special memory, flush cache, or do other tricks for DMA.
Do we want to extend this flag to cover that behaviour as well? If yes should we rename it and tweak the definition? Or should do something else?

Looks not, will device behave differently if we have it?

So it looks to me it was more a platform or software stuffs that could be handled by driver itself.

Thanks

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

Tiwei Bie

author

2018-05-16 01:38 UTC #7 lore raw

In reply to Tiwei Bie

Quoted 11 lines · Tiwei Bie

There will be hardware virtio devices in the future, which require drivers to use the barriers suitable for I/O devices, compared with software virtio devices which just require drivers to use the barriers suitable for CPU cores.
To fix the ordering issue for hardware virtio devices, add a new feature: VIRTIO_F_IO_BARRIER. When negotiated, driver will use the barriers suitable for I/O devices.
Signed-off-by Tiwei Bie <[email protected]>
Reviewed-by Stefan Hajnoczi <[email protected]>

I have opened an issue for this:

Fixes https://github.com/oasis-tcs/virtio-spec/issues/10

Please start the voting. Thanks!

Best regards,
Tiwei Bie

Quoted 77 lines

---
v2 -> v3:
- Update the feature bits allocation (Stefan);

v1 -> v2:
- Rebase to the latest spec (MST);
- Use a smaller textwidth (according to _vimrc);

RFC -> v1:
- Use plural (Stefan);
- Add more details (Stefan);

 content.tex | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

content.tex +18 -2

index 7a92cb1..95c243f 100644
@@ -95,10 +95,10 @@ Feature bits are allocated as follows:
 \begin{description}
 \item[0 to 23] Feature bits for the specific device type
 
-\item[24 to 33] Feature bits reserved for extensions to the queue and
+\item[24 to 36] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[34 and above] Feature bits reserved for future extensions.
+\item[37 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}

@@ -5348,6 +5348,15 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
   \item[VIRTIO_F_IN_ORDER(35)] This feature indicates
   that all buffers are used by the device in the same
   order in which they have been made available.
+  \item[VIRTIO_F_IO_BARRIER(36)] This feature indicates
+  that the device needs the driver to use the barriers
+  suitable for hardware devices.  Some transports require
+  barriers to ensure devices have a consistent view of
+  memory.  When devices are implemented in software a
+  weaker form of barrier may be sufficient and yield
+  better performance.  This feature indicates whether
+  a stronger form of barrier suitable for hardware
+  devices is necessary.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}

@@ -5363,6 +5372,10 @@ addresses to the device.
 
 A driver SHOULD accept VIRTIO_F_RING_PACKED if it is offered.
 
+A driver SHOULD accept VIRTIO_F_IO_BARRIER if it is offered.
+If VIRTIO_F_IO_BARRIER has been negotiated, a driver MUST use
+the barriers suitable for hardware devices.
+
 \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
 
 A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further

@@ -5376,6 +5389,9 @@ accepted.
 If VIRTIO_F_IN_ORDER has been negotiated, a device MUST use
 buffers in the same order in which they have been available.
 
+A device MAY fail to operate further if VIRTIO_F_IO_BARRIER
+is not accepted.
+
 \section{Legacy Interface: Reserved Feature Bits}\label{sec:Reserved Feature Bits / Legacy Interface: Reserved Feature Bits}
 
 Transitional devices MAY offer the following:

Signature · 6 lines

2.11.0


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

Michael S. Tsirkin

2018-06-20 03:31 UTC #8 lore raw

In reply to Tiwei Bie

Quoted 11 lines · Tiwei Bie

There will be hardware virtio devices in the future, which require drivers to use the barriers suitable for I/O devices, compared with software virtio devices which just require drivers to use the barriers suitable for CPU cores.
To fix the ordering issue for hardware virtio devices, add a new feature: VIRTIO_F_IO_BARRIER. When negotiated, driver will use the barriers suitable for I/O devices.
Signed-off-by Tiwei Bie <[email protected]>
Reviewed-by Stefan Hajnoczi <[email protected]>

So some people noticed that some systems also
require a cache flush around DMA to real devices.

Shouldn't we add that as part of VIRTIO_F_IO_BARRIER?

Also rename it?

Others also noticed that there are systems where accesses include some kind of translation e.g. some high bits in the address are set to signal access properties. Should that be included in VIRTIO_F_IOMMU_PLATFORM?

Quoted 76 lines

---
v2 -> v3:
- Update the feature bits allocation (Stefan);

v1 -> v2:
- Rebase to the latest spec (MST);
- Use a smaller textwidth (according to _vimrc);

RFC -> v1:
- Use plural (Stefan);
- Add more details (Stefan);

 content.tex | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

content.tex +18 -2

index 7a92cb1..95c243f 100644
@@ -95,10 +95,10 @@ Feature bits are allocated as follows:
 \begin{description}
 \item[0 to 23] Feature bits for the specific device type
 
-\item[24 to 33] Feature bits reserved for extensions to the queue and
+\item[24 to 36] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[34 and above] Feature bits reserved for future extensions.
+\item[37 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}

@@ -5348,6 +5348,15 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
   \item[VIRTIO_F_IN_ORDER(35)] This feature indicates
   that all buffers are used by the device in the same
   order in which they have been made available.
+  \item[VIRTIO_F_IO_BARRIER(36)] This feature indicates
+  that the device needs the driver to use the barriers
+  suitable for hardware devices.  Some transports require
+  barriers to ensure devices have a consistent view of
+  memory.  When devices are implemented in software a
+  weaker form of barrier may be sufficient and yield
+  better performance.  This feature indicates whether
+  a stronger form of barrier suitable for hardware
+  devices is necessary.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}

@@ -5363,6 +5372,10 @@ addresses to the device.
 
 A driver SHOULD accept VIRTIO_F_RING_PACKED if it is offered.
 
+A driver SHOULD accept VIRTIO_F_IO_BARRIER if it is offered.
+If VIRTIO_F_IO_BARRIER has been negotiated, a driver MUST use
+the barriers suitable for hardware devices.
+
 \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
 
 A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further

@@ -5376,6 +5389,9 @@ accepted.
 If VIRTIO_F_IN_ORDER has been negotiated, a device MUST use
 buffers in the same order in which they have been available.
 
+A device MAY fail to operate further if VIRTIO_F_IO_BARRIER
+is not accepted.
+
 \section{Legacy Interface: Reserved Feature Bits}\label{sec:Reserved Feature Bits / Legacy Interface: Reserved Feature Bits}
 
 Transitional devices MAY offer the following:

Signature · 6 lines

2.11.0


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

Tiwei Bie

author

2018-06-20 13:40 UTC #9 lore raw

Quoted 13 lines · Michael S. Tsirkin

Quoted 11 lines · Tiwei Bie
There will be hardware virtio devices in the future, which require drivers to use the barriers suitable for I/O devices, compared with software virtio devices which just require drivers to use the barriers suitable for CPU cores.
To fix the ordering issue for hardware virtio devices, add a new feature: VIRTIO_F_IO_BARRIER. When negotiated, driver will use the barriers suitable for I/O devices.
Signed-off-by Tiwei Bie <[email protected]>
Reviewed-by Stefan Hajnoczi <[email protected]>

Quoted 7 lines

So some people noticed that some systems also require a cache flush around DMA to real devices.
Shouldn't we add that as part of VIRTIO_F_IO_BARRIER?
Also rename it?

Do we just need to add cache flush? One thing
worried me is that I'm not sure how to cover
all the possible cases..

Quoted 7 lines

Others also noticed that there are systems where accesses include some kind of translation e.g. some high bits in the address are set to signal access properties. Should that be included in VIRTIO_F_IOMMU_PLATFORM?

It's my first time to know such systems. I don't
know any details about it..

Best regards,
Tiwei Bie

Quoted 71 lines

---
v2 -> v3:
- Update the feature bits allocation (Stefan);

v1 -> v2:
- Rebase to the latest spec (MST);
- Use a smaller textwidth (according to _vimrc);

RFC -> v1:
- Use plural (Stefan);
- Add more details (Stefan);

 content.tex | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

content.tex +18 -2

index 7a92cb1..95c243f 100644
@@ -95,10 +95,10 @@ Feature bits are allocated as follows:
 \begin{description}
 \item[0 to 23] Feature bits for the specific device type
 
-\item[24 to 33] Feature bits reserved for extensions to the queue and
+\item[24 to 36] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[34 and above] Feature bits reserved for future extensions.
+\item[37 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}

@@ -5348,6 +5348,15 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
   \item[VIRTIO_F_IN_ORDER(35)] This feature indicates
   that all buffers are used by the device in the same
   order in which they have been made available.
+  \item[VIRTIO_F_IO_BARRIER(36)] This feature indicates
+  that the device needs the driver to use the barriers
+  suitable for hardware devices.  Some transports require
+  barriers to ensure devices have a consistent view of
+  memory.  When devices are implemented in software a
+  weaker form of barrier may be sufficient and yield
+  better performance.  This feature indicates whether
+  a stronger form of barrier suitable for hardware
+  devices is necessary.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}

@@ -5363,6 +5372,10 @@ addresses to the device.
 
 A driver SHOULD accept VIRTIO_F_RING_PACKED if it is offered.
 
+A driver SHOULD accept VIRTIO_F_IO_BARRIER if it is offered.
+If VIRTIO_F_IO_BARRIER has been negotiated, a driver MUST use
+the barriers suitable for hardware devices.
+
 \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
 
 A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further

@@ -5376,6 +5389,9 @@ accepted.
 If VIRTIO_F_IN_ORDER has been negotiated, a device MUST use
 buffers in the same order in which they have been available.
 
+A device MAY fail to operate further if VIRTIO_F_IO_BARRIER
+is not accepted.
+
 \section{Legacy Interface: Reserved Feature Bits}\label{sec:Reserved Feature Bits / Legacy Interface: Reserved Feature Bits}

Quoted 8 lines

content.tex +0 -0

 Transitional devices MAY offer the following:

Signature · 6 lines

2.11.0


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

Michael S. Tsirkin

2018-06-20 14:02 UTC #10 lore raw

Quoted 17 lines · Tiwei Bie

Quoted 16 lines · Michael S. Tsirkin
Quoted 11 lines · Tiwei Bie
There will be hardware virtio devices in the future, which require drivers to use the barriers suitable for I/O devices, compared with software virtio devices which just require drivers to use the barriers suitable for CPU cores.
To fix the ordering issue for hardware virtio devices, add a new feature: VIRTIO_F_IO_BARRIER. When negotiated, driver will use the barriers suitable for I/O devices.
Signed-off-by Tiwei Bie <[email protected]>
Reviewed-by Stefan Hajnoczi <[email protected]>
So some people noticed that some systems also require a cache flush around DMA to real devices.

Quoted 8 lines

Quoted 4 lines
Shouldn't we add that as part of VIRTIO_F_IO_BARRIER?
Also rename it?
Do we just need to add cache flush? One thing worried me is that I'm not sure how to cover all the possible cases..

It's a good point. Someone suggested "_I_AM_A_REAL_PCI_DEVICE". Joking aside, maybe say that it's not running on another CPU
so must not apply optimizations that assume another CPU?

IOMMU is somewhat special as it's used for protection within guests.

Quoted 8 lines

Quoted 7 lines
Others also noticed that there are systems where accesses include some kind of translation e.g. some high bits in the address are set to signal access properties. Should that be included in VIRTIO_F_IOMMU_PLATFORM?

Quoted 6 lines

It's my first time to know such systems. I don't know any details about it..
Best regards,
Tiwei Bie

See discussion around
virtio: Add platform specific DMA API translation for virito devices

you can try poking Christoph Hellwing and Benjamin Herren about details.

Quoted 72 lines

---
v2 -> v3:
- Update the feature bits allocation (Stefan);

v1 -> v2:
- Rebase to the latest spec (MST);
- Use a smaller textwidth (according to _vimrc);

RFC -> v1:
- Use plural (Stefan);
- Add more details (Stefan);

 content.tex | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

content.tex +18 -2

index 7a92cb1..95c243f 100644
@@ -95,10 +95,10 @@ Feature bits are allocated as follows:
 \begin{description}
 \item[0 to 23] Feature bits for the specific device type
 
-\item[24 to 33] Feature bits reserved for extensions to the queue and
+\item[24 to 36] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[34 and above] Feature bits reserved for future extensions.
+\item[37 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}

@@ -5348,6 +5348,15 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
   \item[VIRTIO_F_IN_ORDER(35)] This feature indicates
   that all buffers are used by the device in the same
   order in which they have been made available.
+  \item[VIRTIO_F_IO_BARRIER(36)] This feature indicates
+  that the device needs the driver to use the barriers
+  suitable for hardware devices.  Some transports require
+  barriers to ensure devices have a consistent view of
+  memory.  When devices are implemented in software a
+  weaker form of barrier may be sufficient and yield
+  better performance.  This feature indicates whether
+  a stronger form of barrier suitable for hardware
+  devices is necessary.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}

@@ -5363,6 +5372,10 @@ addresses to the device.
 
 A driver SHOULD accept VIRTIO_F_RING_PACKED if it is offered.
 
+A driver SHOULD accept VIRTIO_F_IO_BARRIER if it is offered.
+If VIRTIO_F_IO_BARRIER has been negotiated, a driver MUST use
+the barriers suitable for hardware devices.
+
 \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
 
 A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further

@@ -5376,6 +5389,9 @@ accepted.
 If VIRTIO_F_IN_ORDER has been negotiated, a device MUST use
 buffers in the same order in which they have been available.
 
+A device MAY fail to operate further if VIRTIO_F_IO_BARRIER
+is not accepted.
+
 \section{Legacy Interface: Reserved Feature Bits}\label{sec:Reserved Feature Bits / Legacy Interface: Reserved Feature Bits}

Quoted 8 lines

content.tex +0 -0

 Transitional devices MAY offer the following:

Signature · 6 lines

2.11.0


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

Tiwei Bie

author

2018-06-20 14:31 UTC #11 lore raw

Quoted 23 lines · Michael S. Tsirkin

Quoted 22 lines · Tiwei Bie
Quoted 20 lines · Michael S. Tsirkin
Quoted 11 lines · Tiwei Bie
There will be hardware virtio devices in the future, which require drivers to use the barriers suitable for I/O devices, compared with software virtio devices which just require drivers to use the barriers suitable for CPU cores.
To fix the ordering issue for hardware virtio devices, add a new feature: VIRTIO_F_IO_BARRIER. When negotiated, driver will use the barriers suitable for I/O devices.
Signed-off-by Tiwei Bie <[email protected]>
Reviewed-by Stefan Hajnoczi <[email protected]>
So some people noticed that some systems also require a cache flush around DMA to real devices.
Shouldn't we add that as part of VIRTIO_F_IO_BARRIER?
Also rename it?

Quoted 7 lines

Quoted 3 lines
Do we just need to add cache flush? One thing worried me is that I'm not sure how to cover all the possible cases..
It's a good point. Someone suggested "_I_AM_A_REAL_PCI_DEVICE". Joking aside, maybe say that it's not running on another CPU
so must not apply optimizations that assume another CPU?

Good idea! I'll try to compose a RFC ASAP.

Quoted 20 lines

IOMMU is somewhat special as it's used for protection within guests.
Quoted 8 lines
Quoted 7 lines
Others also noticed that there are systems where accesses include some kind of translation e.g. some high bits in the address are set to signal access properties. Should that be included in VIRTIO_F_IOMMU_PLATFORM?
Quoted 6 lines
It's my first time to know such systems. I don't know any details about it..
Best regards,
Tiwei Bie

Quoted 5 lines

See discussion around
virtio: Add platform specific DMA API translation for virito devices
you can try poking Christoph Hellwing and Benjamin Herren about details.

Got it. Thanks!

Best regards,
Tiwei Bie

Quoted 74 lines

---
v2 -> v3:
- Update the feature bits allocation (Stefan);

v1 -> v2:
- Rebase to the latest spec (MST);
- Use a smaller textwidth (according to _vimrc);

RFC -> v1:
- Use plural (Stefan);
- Add more details (Stefan);

 content.tex | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

content.tex +18 -2

index 7a92cb1..95c243f 100644
@@ -95,10 +95,10 @@ Feature bits are allocated as follows:
 \begin{description}
 \item[0 to 23] Feature bits for the specific device type
 
-\item[24 to 33] Feature bits reserved for extensions to the queue and
+\item[24 to 36] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[34 and above] Feature bits reserved for future extensions.
+\item[37 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}

@@ -5348,6 +5348,15 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
   \item[VIRTIO_F_IN_ORDER(35)] This feature indicates
   that all buffers are used by the device in the same
   order in which they have been made available.
+  \item[VIRTIO_F_IO_BARRIER(36)] This feature indicates
+  that the device needs the driver to use the barriers
+  suitable for hardware devices.  Some transports require
+  barriers to ensure devices have a consistent view of
+  memory.  When devices are implemented in software a
+  weaker form of barrier may be sufficient and yield
+  better performance.  This feature indicates whether
+  a stronger form of barrier suitable for hardware
+  devices is necessary.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}

@@ -5363,6 +5372,10 @@ addresses to the device.
 
 A driver SHOULD accept VIRTIO_F_RING_PACKED if it is offered.
 
+A driver SHOULD accept VIRTIO_F_IO_BARRIER if it is offered.
+If VIRTIO_F_IO_BARRIER has been negotiated, a driver MUST use
+the barriers suitable for hardware devices.
+
 \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
 
 A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further

@@ -5376,6 +5389,9 @@ accepted.
 If VIRTIO_F_IN_ORDER has been negotiated, a device MUST use
 buffers in the same order in which they have been available.
 
+A device MAY fail to operate further if VIRTIO_F_IO_BARRIER
+is not accepted.
+
 \section{Legacy Interface: Reserved Feature Bits}\label{sec:Reserved Feature Bits / Legacy Interface: Reserved Feature Bits}

Quoted 8 lines

content.tex +0 -0

 Transitional devices MAY offer the following:

Signature · 6 lines

2.11.0


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]