virtio-media: Add device specification
v9 cover virtio-media: Add device specification
New attempt of including virtio-media
device specification.
v8->v9:
- Rebased to virtio-1.4
v7->v8:
- Merged DQBUF and DQEVENT into the same
item for unsupported ioctls
- Rewrote/clarified a couple paragraphs
based on previous version reviews.
Virtio-media came from a discussion on virtio-dev mailing list, which lead to Alex Courbot presenting virtio-v4l2[1] specification as an alternative to virtio-video.
Later, virtio-v4l2 was renamed to virtio-media[2] and published at:
https://github.com/chromeos/virtio-media
The repository above includes a virtio-media driver able to pass v4l2-compliance when proxying the vivid/vicodec virtual devices or an actual UVC camera using the V4L2 vhost device (available in the repository). It also includes a FFmpeg-based video encoder device. Steps to reproduce are also detailed[3].
Recently, virtio-media has landed in AOSP[4]. Also the driver patch has been sent to the kernel, currently in its v2 [5].
Furthermore, virtio-media got a proposal to reserve device ID 48, which was finally approved for inclusion in v1.4.
There is some overlap with virtio-video in regards to which devices it can handle. However, they take different approaches, potentially making them the preferable choice for different scenarios. Furthermore, virtio-media has matured since it was first presented, and there are public efforts to have upstream driver and devices.
But mainly, given that virtio-media will be the virtualization solution for media devices for ChromeOS, Android, and possibly others, I think is important that it gets standardized and included in the specification, despite the aforementioned overlap.
Full PDF: https://drive.google.com/file/d/1sDZXWVlvPbHKKguhekCXMzkpNZZ0g3hV/view?usp=sharing
PDF with the media section only: https://drive.google.com/file/d/1JjOIzecnkFFlpFBRjPQEUyAOD3BDHAdw/view?usp=sharing
[1] https://mail.google.com/mail/u/0?ui=2&ik=73ebd65ebd&attid=0.1&permmsgid=msg-f:1767388565327924962&th=1887068940754ee2&view=att&disp=inline&realattid=f_libalimc0
[2] https://www.mail-archive.com/[email protected]/msg12665.html
[3] https://github.com/chromeos/virtio-media/blob/main/TRY_IT_OUT.md
[4] https://cs.android.com/android/platform/superproject/main/+/main:external/virtio-media/
[5] https://lore.kernel.org/lkml/[email protected]/T/
Albert Esteve (1):
virtio-media: Add virtio media device specification
conformance.tex | 6 + content.tex | 1 + device-types/media/description.tex | 639 ++++++++++++++++++++++ device-types/media/device-conformance.tex | 15 + device-types/media/driver-conformance.tex | 11 + 5 files changed, 672 insertions(+) create mode 100644 device-types/media/description.tex create mode 100644 device-types/media/device-conformance.tex create mode 100644 device-types/media/driver-conformance.tex
Signature · 1 line
2.49.0
PATCH v9 1/1 virtio-media: Add virtio media device specification
Virtio-media is an encapsulation of the V4L2 UAPI into virtio, able to virtualize any video device supported by V4L2.
Note that virtio-media does not require the use of a V4L2 device driver on the host or guest side - V4L2 is only used as a host-guest protocol, and both sides are free to convert it from/to any model that they wish to use.
--- conformance.tex | 6 + content.tex | 1 + device-types/media/description.tex | 639 ++++++++++++++++++++++ device-types/media/device-conformance.tex | 15 + device-types/media/driver-conformance.tex | 11 + 5 files changed, 672 insertions(+) create mode 100644 device-types/media/description.tex create mode 100644 device-types/media/device-conformance.tex create mode 100644 device-types/media/driver-conformance.tex
conformance.tex +6 -0
@@ -36,6 +36,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}, \ref{sec:Conformance / Driver Conformance / CAN Driver Conformance} or \ref{sec:Conformance / Driver Conformance / SPI Controller Driver Conformance}. +\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance} or +\ref{sec:Conformance / Driver Conformance / Media Driver Conformance}. \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}
@@ -65,6 +67,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Device Conformance / PMEM Device Conformance}, \ref{sec:Conformance / Device Conformance / CAN Device Conformance} or \ref{sec:Conformance / Device Conformance / SPI Controller Device Conformance}. +\ref{sec:Conformance / Device Conformance / PMEM Device Conformance} or +\ref{sec:Conformance / Device Conformance / Media Device Conformance}. \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}
@@ -162,6 +166,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/pmem/driver-conformance.tex} \input{device-types/can/driver-conformance.tex} \input{device-types/spi/driver-conformance.tex} +\input{device-types/media/driver-conformance.tex} \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}
@@ -254,6 +259,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/pmem/device-conformance.tex} \input{device-types/can/device-conformance.tex} \input{device-types/spi/device-conformance.tex} +\input{device-types/media/device-conformance.tex} \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance} A conformant implementation MUST be either transitional or
content.tex +1 -0
@@ -778,6 +778,7 @@ \chapter{Device Types}\label{sec:Device Types} \input{device-types/pmem/description.tex} \input{device-types/can/description.tex} \input{device-types/spi/description.tex} +\input{device-types/media/description.tex} \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
device-types/media/description.tex new file +639 -0
639 lines not on this page · raw message
device-types/media/device-conformance.tex new file +15 -0
@@ -0,0 +1,15 @@ +\conformance{\subsection}{Media Device Conformance}\label{sec:Conformance / Device Conformance / Media Device Conformance} + +A Media device MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Open device} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / V4L2 ioctls} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Pointers in V4l2 ioctls} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Unsupported ioctls} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Mapping a MMAP buffer} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Unmapping a MMAP buffer} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Shared Pages} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Virtio Object} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Device-side error} +\end{itemize}
device-types/media/driver-conformance.tex new file +11 -0
@@ -0,0 +1,11 @@ +\conformance{\subsection}{Media Driver Conformance}\label{sec:Conformance / Driver Conformance / Media Driver Conformance} + +A Media driver MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{drivernormative:Device Types / Media Device / Device Operation / Command Virtqueue} +\item \ref{drivernormative:Device Types / Media Device / Device Operation / Close device} +\item \ref{drivernormative:Device Types / Media Device / Device Operation / V4L2 ioctls} +\item \ref{drivernormative:Device Types / Media Device / Device Operation / Device-side error} +\item \ref{drivernormative:Device Types / Media Device / Device Operation / Dequeue buffer} +\end{itemize}
Signature · 1 line
2.49.0
Quoted 5 lines · Albert Esteve
New attempt of including virtio-media
device specification.
v8->v9:
- Rebased to virtio-1.4
I wonder if it wouldn't be better to just send a patch that fixes the series instead of the whole series again. Such a change would be a non editorial change thus not requiring to vote again. The v8 has been already voted and accepted.
Matias
Quoted 11 lines
From: Matias Ezequiel Vara Larsen <[email protected]>
Sent: 09 July 2025 03:14 PMQuoted 5 lines · Albert Esteve
New attempt of including virtio-media
device specification.
v8->v9:
- Rebased to virtio-1.4
Quoted 7 lines
I wonder if it wouldn't be better to just send a patch that fixes the series instead of the whole series again. Such a change would be a non editorial change thus not requiring to vote again. The v8 has been already voted and accepted.
Matias
Right. I was able to apply v9 without merge conflict.
Even though, I prefer what Matias suggested to send the fixup patch so that v9 does not need to go through the review and vote again.
Quoted 17 lines · Parav Pandit
Quoted 17 lines
From: Matias Ezequiel Vara Larsen <[email protected]>
Sent: 09 July 2025 03:14 PMQuoted 5 lines · Albert Esteve
New attempt of including virtio-media
device specification.
v8->v9:
- Rebased to virtio-1.4I wonder if it wouldn't be better to just send a patch that fixes the series instead of the whole series again. Such a change would be a non editorial change thus not requiring to vote again. The v8 has been already voted and accepted.
Quoted 5 lines
Quoted 2 lines
Matias
Right. I was able to apply v9 without merge conflict.
Even though, I prefer what Matias suggested to send the fixup patch so that v9 does not need to go through the review and vote again.
I'm ok with that. I assumed that as it had no changes from v8 whatsoever, it would not require a vote. If it does, let's disregard this new version.
What do you mean for a fixup patch? New patch series with just the changeset required to eliminate the conflict from v8?
Quoted 1 line
Quoted 24 lines
From: Albert Esteve <[email protected]>
Sent: 09 July 2025 03:31 PMQuoted 19 lines · Parav Pandit
Quoted 18 lines
From: Matias Ezequiel Vara Larsen <[email protected]>
Sent: 09 July 2025 03:14 PMQuoted 4 lines · Albert Esteve
New attempt of including virtio-media device specification.
v8->v9:
- Rebased to virtio-1.4I wonder if it wouldn't be better to just send a patch that fixes the series instead of the whole series again. Such a change would be a non editorial change thus not requiring to vote again. The v8 has been already voted and accepted.
Matias
Quoted 7 lines
Quoted 2 lines
Right. I was able to apply v9 without merge conflict.
Even though, I prefer what Matias suggested to send the fixup patch so thatv9 does not need to go through the review and vote again.
I'm ok with that. I assumed that as it had no changes from v8 whatsoever, it would not require a vote. If it does, let's disregard this new version.
If its editorial change, it does not require a vote.
Quoted 3 lines
What do you mean for a fixup patch? New patch series with just the changeset required to eliminate the conflict from v8?
I was looking for something that I apply locally to resolve the merge conflict myself. That way, its basically just v8 + fixup.
So basically a short diff that you did manually, which I can also do before merging it.
Quoted 63 lines · Albert Esteve
New attempt of including virtio-media
device specification.
v8->v9:
- Rebased to virtio-1.4
v7->v8:
- Merged DQBUF and DQEVENT into the same
item for unsupported ioctls
- Rewrote/clarified a couple paragraphs
based on previous version reviews.Virtio-media came from a discussion on virtio-dev mailing list, which lead to Alex Courbot presenting virtio-v4l2[1] specification as an alternative to virtio-video.
Later, virtio-v4l2 was renamed to virtio-media[2] and published at:
https://github.com/chromeos/virtio-media
The repository above includes a virtio-media driver able to pass v4l2-compliance when proxying the vivid/vicodec virtual devices or an actual UVC camera using the V4L2 vhost device (available in the repository). It also includes a FFmpeg-based video encoder device. Steps to reproduce are also detailed[3].
Recently, virtio-media has landed in AOSP[4]. Also the driver patch has been sent to the kernel, currently in its v2 [5].
Furthermore, virtio-media got a proposal to reserve device ID 48, which was finally approved for inclusion in v1.4.
There is some overlap with virtio-video in regards to which devices it can handle. However, they take different approaches, potentially making them the preferable choice for different scenarios. Furthermore, virtio-media has matured since it was first presented, and there are public efforts to have upstream driver and devices.
But mainly, given that virtio-media will be the virtualization solution for media devices for ChromeOS, Android, and possibly others, I think is important that it gets standardized and included in the specification, despite the aforementioned overlap.
Full PDF: https://drive.google.com/file/d/1sDZXWVlvPbHKKguhekCXMzkpNZZ0g3hV/view?usp=sharing
PDF with the media section only: https://drive.google.com/file/d/1JjOIzecnkFFlpFBRjPQEUyAOD3BDHAdw/view?usp=sharing[1] https://mail.google.com/mail/u/0?ui=2&ik=73ebd65ebd&attid=0.1&permmsgid=msg-f:1767388565327924962&th=1887068940754ee2&view=att&disp=inline&realattid=f_libalimc0
[2] https://www.mail-archive.com/[email protected]/msg12665.html
[3] https://github.com/chromeos/virtio-media/blob/main/TRY_IT_OUT.md
[4] https://cs.android.com/android/platform/superproject/main/+/main:external/virtio-media/
[5] https://lore.kernel.org/lkml/[email protected]/T/Albert Esteve (1):
virtio-media: Add virtio media device specification
Quoted 9 lines
conformance.tex | 6 + content.tex | 1 + device-types/media/description.tex | 639 ++++++++++++++++++++++ device-types/media/device-conformance.tex | 15 + device-types/media/driver-conformance.tex | 11 + 5 files changed, 672 insertions(+) create mode 100644 device-types/media/description.tex create mode 100644 device-types/media/device-conformance.tex create mode 100644 device-types/media/driver-conformance.tex
It's probably too late, but I have some serious concerns about this device. Specifically:
1. I don't see a reasonable way to support untrusted virtio-media devices. v4l2 has too many ioctls and I can't realistically see a way to enforce that the return values from all of them are consistent with each other. It is possible to only allow a fixed format (such as uncompressed ARGB) and only allow the video source and/or sink to provide resolutions and frame rates, but this means losing a lot of performance and abandoning any attempt at zero-copy.
The use-case for untrusted virtio-media devices is not confidential computing, but rather disaggregated systems where video sources (like webcams) may be attached to untrusted virtual machines.
2. v4l2 is a chatty protocol and using it implies a lot of guest <=> host round-trips. This is bad for performance. Is this overhead actually significant, and if so, are there plans to reduce it?
Signature · 2 lines
Sincerely, Demi Marie Obenour (she/her/hers)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 20 lines · Nicolas Dufresne
Hi Demi,
Le mercredi 09 juillet 2025 à 13:26 -0400, Demi Marie Obenour a écrit :
Quoted 2 lines · Albert Esteve
Quoted 1 line
[...]
Quoted 13 lines
It's probably too late, but I have some serious concerns about this device. Specifically:
1. I don't see a reasonable way to support untrusted virtio-media devices. v4l2 has too many ioctls and I can't realistically see a way to enforce that the return values from all of them are consistent with each other. It is possible to only allow a fixed format (such as uncompressed ARGB) and only allow the video source and/or sink to provide resolutions and frame rates, but this means losing a lot of performance and abandoning any attempt at zero-copy.
Quoted 7 lines
Quoted 4 lines
The use-case for untrusted virtio-media devices is not confidential computing, but rather disaggregated systems where video sources (like webcams) may be attached to untrusted virtual machines.
Please consider that you are posting broadly and understanding of what you mean by untrusted virtio-media can be difficult for non-virtio devs like me.
My use-case is this:
1. There is a VM that has all USB devices.
2. Some of those USB devices, like untrusted webcams, may be passed through to another VM via usbip.
3. The VM with the usbip device provides a video device to yet another VM.
The VM with the webcam and the VM receiving the video stream do not trust each other.
Quoted 4 lines
With the enclosed information, I believe implementer are not forced to map to a host V4L2 driver. You can implement a userspace layer in between, and possibly even implement reclaiming by simulation some hot-unplug of the devices, or by not scheduling any work for M2M.
I expect that any implementation that cares about security is going to do this.
Quoted 4 lines
I don't really see the relation between "trust" and the pixel format selection. Nor why preventing zero-copy would help handling untrusted virtual machines. I do believe you should not attempt this if you don't have the required knowledge though, as its unlikely the end-result will be safe.
How can I validate the various parameters provided by a backend?
Quoted 3 lines
For me the main issue for untrusted VM is the lack of control measure, such as cgroup for resource usage. This is being worked on though, Maxime Ripard have given an update at the Linux Media Summit in Nice.
Denial of service is not a concern in my use-cases.
Quoted 9 lines
Quoted 4 lines
2. v4l2 is a chatty protocol and using it implies a lot of guest <=> host round-trips. This is bad for performance. Is this overhead actually significant, and if so, are there plans to reduce it?
You should try and demonstrate this. Virtio Video tried to make their own and endup with something kind of similar with about the same level of back and forth. We already had few concerns with
ioctl()overhead, and some measure have been implemented, notablys_ctrl_ext.
I'll leave this for later.
Signature · 2 lines
Sincerely, Demi Marie Obenour (she/her/hers)