virtio-media: Add device specification
v4 cover virtio-media: Add device specification
New attempt of including virtio-media
device specification.
v3->v4:
- Updated device ID to 48
(moved to the first free id)
- Specified endianess on all
structures defined through
the spec (leXX)
- Slightly improved the explanation
of the size (and endianess)
of the exchanged v4l2 structs
as part of the ioctl command
payload
- Changed cmd_mmap offset size to
match the V4L2_QUERYBUF offset field
- Fixed most comments on previous
version for small typos/rewrites.
v2->v3:
- Updated device ID to 49
- Renamed virtio memory types so that
they are differentiated from V4L2
memory types
- Memory types description slightly
rewritten, explicitely excluding
USERPTR support for guest userspace
v1->v2:
- Remove naming host/guest in the text
- Explicitly specify endian-ness of the device
- Change address by offset in the MMAP operation
- Specify SHM region for MMAP operation
Virtio-media came from a discussion on virtio-dev mailing list, which lead to presenting virtio-v4l2[1] specification as an alternative to virtio-video.
Later, virtio-v4l2 was renamed to virtio-media[2] and published at:
https://github.com/chromeos/virtio-media
The repository above includes a virtio-media driver able to pass v4l2-compliance when proxying the vivid/vicodec virtual devices or an actual UVC camera using the V4L2 vhost device (available in the repository). It also includes a FFmpeg-based video encoder device. Steps to reproduce are also detailed[3].
Recently, virtio-media has landed in AOSP[4].
Furthermore, virtio-media got a proposal to reserve device ID 48, which was finally approved for inclusion in v1.4.
There is some overlap with virtio-video in regards to which devices it can handle. However,
they take different approaches, making them
the preferable choice for different scenarios. Moreover, as virtio-media will likely be the virtualization solution for ChromeOS, Android, and possibly others, for media devices, it is justified the desire to include the specification in the next release despite
the aforementioned overlap.
Full PDF: https://drive.google.com/file/d/13es8492vk2ZSYCwVcGt1qGs6uLvESvoo/view?usp=sharing
PDF with the media section only: https://drive.google.com/file/d/1cILB6mKSyWVs2yGs_WKfMpp1PSDOFPp4/view?usp=sharing
[1] https://mail.google.com/mail/u/0?ui=2&ik=73ebd65ebd&attid=0.1&permmsgid=msg-f:1767388565327924962&th=1887068940754ee2&view=att&disp=inline&realattid=f_libalimc0
[2] https://www.mail-archive.com/[email protected]/msg12665.html
[3] https://github.com/chromeos/virtio-media/blob/main/TRY_IT_OUT.md
[4] https://cs.android.com/android/platform/superproject/main/+/main:external/virtio-media/
Albert Esteve (1):
virtio-media: Add virtio media device specification
conformance.tex | 13 +- content.tex | 1 + device-types/media/description.tex | 575 ++++++++++++++++++++++ device-types/media/device-conformance.tex | 10 + device-types/media/driver-conformance.tex | 8 + 5 files changed, 603 insertions(+), 4 deletions(-) create mode 100644 device-types/media/description.tex create mode 100644 device-types/media/device-conformance.tex create mode 100644 device-types/media/driver-conformance.tex
Signature · 1 line
2.47.0
PATCH v4 1/1 virtio-media: Add virtio media device specification
Virtio-media is an encapsulation of the V4L2 UAPI into virtio, able to virtualize any video device supported by V4L2.
Note that virtio-media does not require the use of a V4L2 device driver on the host or guest side - V4L2 is only used as a host-guest protocol, and both sides are free to convert it from/to any model that they wish to use.
--- conformance.tex | 13 +- content.tex | 1 + device-types/media/description.tex | 575 ++++++++++++++++++++++ device-types/media/device-conformance.tex | 10 + device-types/media/driver-conformance.tex | 8 + 5 files changed, 603 insertions(+), 4 deletions(-) create mode 100644 device-types/media/description.tex create mode 100644 device-types/media/device-conformance.tex create mode 100644 device-types/media/driver-conformance.tex
conformance.tex +9 -4
@@ -32,8 +32,10 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Driver Conformance / Memory Driver Conformance}, \ref{sec:Conformance / Driver Conformance / I2C Adapter Driver Conformance}, \ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance}, -\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance} or -\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}. +\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance}, +\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance} or +\ref{sec:Conformance / Driver Conformance / Media Driver Conformance}. + \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}
@@ -59,8 +61,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Device Conformance / Memory Device Conformance}, \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance}, \ref{sec:Conformance / Device Conformance / SCMI Device Conformance}, -\ref{sec:Conformance / Device Conformance / GPIO Device Conformance} or -\ref{sec:Conformance / Device Conformance / PMEM Device Conformance}. +\ref{sec:Conformance / Device Conformance / GPIO Device Conformance}, +\ref{sec:Conformance / Device Conformance / PMEM Device Conformance} or +\ref{sec:Conformance / Device Conformance / Media Device Conformance}. \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}
@@ -152,6 +155,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/driver-conformance.tex} \input{device-types/gpio/driver-conformance.tex} \input{device-types/pmem/driver-conformance.tex} +\input{device-types/media/driver-conformance.tex} \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}
@@ -238,6 +242,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/device-conformance.tex} \input{device-types/gpio/device-conformance.tex} \input{device-types/pmem/device-conformance.tex} +\input{device-types/media/device-conformance.tex} \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance} A conformant implementation MUST be either transitional or
content.tex +1 -0
@@ -767,6 +767,7 @@ \chapter{Device Types}\label{sec:Device Types} \input{device-types/scmi/description.tex} \input{device-types/gpio/description.tex} \input{device-types/pmem/description.tex} +\input{device-types/media/description.tex} \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
device-types/media/description.tex new file +575 -0
575 lines not on this page · raw message
device-types/media/device-conformance.tex new file +10 -0
@@ -0,0 +1,10 @@ +\conformance{\subsection}{Media Device Conformance}\label{sec:Conformance / Device Conformance / Media Device Conformance} + +A Media device MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Open device} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / V4L2 ioctls} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Unsupported ioctls} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Unmapping a MMAP buffer} +\end{itemize}
device-types/media/driver-conformance.tex new file +8 -0
@@ -0,0 +1,8 @@ +\conformance{\subsection}{Media Device Conformance}\label{sec:Conformance / Driver Conformance / Media Driver Conformance} + +A Media device MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{drivernormative:Device Types / Media Device / Device Operation / Command Virtqueue} +\item \ref{drivernormative:Device Types / Media Device / Device Operation / Close device} +\end{itemize}
Signature · 1 line
2.47.0
Re: [PATCH v4 1/1] virtio-media: Add virtio media device specification
Hello Albert,
I added some minor comments below. I may need to read it again since there are some points that I did not understand completely.
Quoted 138 lines · Albert Esteve
Virtio-media is an encapsulation of the V4L2 UAPI into virtio, able to virtualize any video device supported by V4L2.
Note that virtio-media does not require the use of a V4L2 device driver on the host or guest side - V4L2 is only used as a host-guest protocol, and both sides are free to convert it from/to any model that they wish to use.
Signed-off-by Albert Esteve <[email protected]>--- conformance.tex | 13 +- content.tex | 1 + device-types/media/description.tex | 575 ++++++++++++++++++++++ device-types/media/device-conformance.tex | 10 + device-types/media/driver-conformance.tex | 8 + 5 files changed, 603 insertions(+), 4 deletions(-) create mode 100644 device-types/media/description.tex create mode 100644 device-types/media/device-conformance.tex create mode 100644 device-types/media/driver-conformance.texconformance.tex +9
-4@@ -32,8 +32,10 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Driver Conformance / Memory Driver Conformance}, \ref{sec:Conformance / Driver Conformance / I2C Adapter Driver Conformance}, \ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance}, -\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance} or -\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}. +\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance}, +\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance} or +\ref{sec:Conformance / Driver Conformance / Media Driver Conformance}. + \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -59,8 +61,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Device Conformance / Memory Device Conformance}, \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance}, \ref{sec:Conformance / Device Conformance / SCMI Device Conformance}, -\ref{sec:Conformance / Device Conformance / GPIO Device Conformance} or -\ref{sec:Conformance / Device Conformance / PMEM Device Conformance}. +\ref{sec:Conformance / Device Conformance / GPIO Device Conformance}, +\ref{sec:Conformance / Device Conformance / PMEM Device Conformance} or +\ref{sec:Conformance / Device Conformance / Media Device Conformance}. \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -152,6 +155,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/driver-conformance.tex} \input{device-types/gpio/driver-conformance.tex} \input{device-types/pmem/driver-conformance.tex} +\input{device-types/media/driver-conformance.tex} \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}@@ -238,6 +242,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/device-conformance.tex} \input{device-types/gpio/device-conformance.tex} \input{device-types/pmem/device-conformance.tex} +\input{device-types/media/device-conformance.tex} \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance} A conformant implementation MUST be either transitional orcontent.tex +1
-0@@ -767,6 +767,7 @@ \chapter{Device Types}\label{sec:Device Types} \input{device-types/scmi/description.tex} \input{device-types/gpio/description.tex} \input{device-types/pmem/description.tex} +\input{device-types/media/description.tex} \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}device-types/media/description.tex new file +53
-0@@ -0,0 +1,575 @@ +\section{Media Device}\label{sec:Device Types / Media Device} + +The virtio media device follows the same model (and structures) as V4L2. It +can be used to virtualize cameras, codec devices, or any other device +supported by V4L2. The complete definition of V4L2 structures and ioctls can +be found under the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/index.html}{V4L2 UAPI documentation}. + +V4L2 is a UAPI that allows a less privileged entity (user-space) to use video +hardware exposed by a more privileged entity (the kernel). Virtio-media is an +encapsulation of this API into virtio, turning it into a virtualization API +for all classes of video devices supported by V4L2, where the device plays the +role of the kernel and the driver the role of user-space. + +The device is therefore responsible for presenting a virtual device that behaves +like an actual V4L2 device, which the driver can control. + +Note that virtio-media does not require the use of a V4L2 device driver or of +Linux on any side - V4L2 is only used as a transport protocol, +and both sides are free to convert it from/to any model that they wish to use. + +\subsection{Device ID}\label{sec:Device Types / Media Device / Device ID} + +48 + +\subsection{Virtqueues}\label{sec:Device Types / Media Device / Virtqueues} + +\begin{description} +\item[0] commandq - used for driver commands and device responses to these +commands. +\item[1] eventq - used for events sent by the device to the driver. +\end{description} + +\subsection{Feature Bits}\label{sec:Device Types / Media Device / Feature Bits} + +None + +\subsection{Device Configuration Layout}\label{sec:Device Types / Media Device / Device Configuration Layout} + +The video device configuration space uses the following layout: + +\begin{lstlisting} +struct virtio_media_config { + le32 device_caps; + le32 device_type; + le8 card[32]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{device_caps}] (driver-read-only) flags representing the device +capabilities as used in +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/vidioc-querycap.html#c.V4L.v4l2_capability}{struct v4l2_capability}.
device-types/media/description.tex line 63
Quoted 10 lines
device-types/media/description.tex new file +10
-0+It corresponds with the \field{device_caps} field in the \textit{struct video_device}. +\item[\field{device_type}] (driver-read-only) informs the driver of the type +of the video device. It corresponds with the \field{vfl_devnode_type} field of the device. +\item[\field{card}] (driver-read-only) name of the device, a NUL-terminated +UTF-8 string. It corresponds with the \field{card} field of the \textit{struct v4l2_capability}. +If all the characters of the field are used, it does not need to be NUL-terminated. +\end{description} + +\subsection{Device Initialization} +
Just to clarify, I would add a sentence here like:A driver executes the following sequence to initialize a device:
device-types/media/description.tex line 67
Quoted 4 lines
device-types/media/description.tex new file +4
-0+\begin{enumerate} +\item Read the \field{device_caps} and \field{device_type} fields +from the configuration layout to identify the device. +\item Set up the \field{commandq} and \field{eventq}.
Maybe add ... \field{commandq} and \field{eventq} virtqueues.
device-types/media/description.tex line 76
Quoted 9 lines
device-types/media/description.tex new file +9
-0+\item May open a session (see Section \ref{sec:Device Operation: Command Virtqueue: Sessions}) +to use the device and send V4L2 ioctls in order to receive more information +about the device, such as supported formats or controls. +\end{enumerate} + +\subsection{Device Operation}\label{sec:Device Types / Media Device / Device Operation} + +Commands are queued on the command queue by the driver for the device to +process. The errors returned by each command are standard
I would rewrite it as below but I do not have a strong opinion:
The driver enqueues commands in the command queue for the device to process them.
device-types/media/description.tex line 79
Quoted 3 lines
device-types/media/description.tex new file +3
-0+\href{https://www.kernel.org/doc/html/latest/userspace-api/media/gen-errors.html}{Linux kernel error codes}. +For instance, a command that contains invalid options will return \textit{EINVAL}. +
I think here is not the command that returns EINVAL but the device when trying to process a command that has an invalid option.
device-types/media/description.tex line 80
Quoted 1 line
device-types/media/description.tex new file +1
-0+Events are sent on the event queue by the device for the driver to handle.
I would rewrite it as below but I do not have a strong opinion:
The device enqueues events in the event queue for the driver to process them.
Quoted 31 lines
device-types/media/description.tex new file +31
-0+ +\subsubsection{Command Virtqueue} + +\paragraph{Device Operation: Command headers} + +\begin{lstlisting} +#define VIRTIO_MEDIA_CMD_OPEN 1 +#define VIRTIO_MEDIA_CMD_CLOSE 2 +#define VIRTIO_MEDIA_CMD_IOCTL 3 +#define VIRTIO_MEDIA_CMD_MMAP 4 +#define VIRTIO_MEDIA_CMD_MUNMAP 5 + +/* Header for all virtio commands from the driver to the device on the commandq. */ +struct virtio_media_cmd_header { + le32 cmd; + le32 __reserved; +}; + +/* Header for all virtio responses from the device to the driver on the commandq. */ +struct virtio_media_resp_header { + le32 status; + le32 __reserved; +}; +\end{lstlisting} + +A command consists of a command header \textit{virtio_media_cmd_header} +containing the following device-readable field: + +\begin{description} +\item[\field{cmd}] specifies a device request type (VIRTIO_MEDIA_CMD_*). +\end{description}
device-types/media/description.tex line 121
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +A response consists of a response header \textit{virtio_media_resp_header} +containing the following device-writable field: + +\begin{description} +\item[\field{status}] indicates a device request status. +\end{description} + +The status field can take 0 if the command was successful, or one of the +standard Linux error codes if it was not.
I would rewrite it as but I do not have a strong opinion about it:
`When the device executes the command successfully, the value of the status field is 0. When the device fails to execute the command, the value of the status field is one of the standard Linux error codes.`
Quoted 51 lines
device-types/media/description.tex new file +51
-0+ +\drivernormative{\paragraph}{Device Operation: Command Virtqueue: Sessions}{Device Types / Media Device / Device Operation / Command Virtqueue} + +Sessions are how the device is multiplexed, allowing several distinct works to +take place simultaneously. Before start operating, the driver needs to open a +session. This is equivalent to opening the \textit{/dev/videoX} file of the +V4L2 device. Each session gets a unique ID assigned, which can be then used +to perform actions on it. + +\paragraph{Device Operation: Open device} + +\textbf{VIRTIO_MEDIA_CMD_OPEN} Command for creating a new session. + +This is the equivalent of calling \textit{open} on a V4L2 device node. +The driver uses \textit{virtio_media_cmd_open} to send an open request. + +\begin{lstlisting} +struct virtio_media_cmd_open { + struct virtio_media_cmd_header hdr; +}; +\end{lstlisting} + +The device responds to \textit{VIRTIO_MEDIA_CMD_OPEN} with \textit{virtio_media_resp_open}. + +\begin{lstlisting} +struct virtio_media_resp_open { + struct virtio_media_resp_header hdr; + le32 session_id; + le32 __reserved; +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the current session, which is used for +other commands, predominantly ioctls. +\end{description} + +\devicenormative{\subparagraph}{Device Operation: Open device}{Device Types / Media Device / Device Operation / Open device} + +Upon success, the device MUST set a \field{session_id} in \textit{virtio_media_resp_open} +to an integer that is NOT used by any other open session. + +\paragraph{Device Operation: Close device} + +\textbf{VIRTIO_MEDIA_CMD_CLOSE} Command for closing an active session. + +This is the equivalent of calling \textit{close} on a previously opened V4L2 +device node. All resources associated with this session will be freed. + +This command does not require a response from the device. +
device-types/media/description.tex line 182
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\begin{lstlisting} +struct virtio_media_cmd_close { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 __reserved; +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] specifies an identifier for the session to close.
I would rewrite it shorter as:
session_id identifies the session to close.
Quoted 94 lines
device-types/media/description.tex new file +94
-0+\end{description} + +\drivernormative{\subparagraph}{Device Operation: Close device}{Device Types / Media Device / Device Operation / Close device} + +The session ID SHALL NOT be used again after queueing this command. + +\paragraph{Device Operation: V4L2 ioctls} + +\textbf{VIRTIO_MEDIA_CMD_IOCTL} Command for executing an ioctl on an open +session. + +This command tells the device to run one of the `VIDIOC_*` ioctls on the +session identified by \textit{session_id}. + +\begin{lstlisting} +struct virtio_media_cmd_ioctl { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 code; + /* Followed by the relevant ioctl payload as defined in the macro */ +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the session to run the ioctl on. +\item[\field{code}] specifies the code of the \field{VIDIOC_*} ioctl to run. +\end{description} + +The code is extracted from the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/videodev.html}{videodev2.h}, +header file. The file defines the ioctl's codes, type of payload, and +direction. The code consists of the second argument of the \field{_IO*} macro. + +For example, the \textit{VIDIOC_G_FMT} is defined as follows: + +\begin{lstlisting} +#define VIDIOC_G_FMT _IOWR('V', 4, struct v4l2_format) +\end{lstlisting} + +This means that its ioctl code is \textit{4}, its payload is a +\textit{struct v4l2_format}, and its direction is \textit{WR} (i.e., the +payload is written by both the driver and the device). +See Section \ref{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} +for more information about the direction of ioctls. +Note that although most architectures use this format, there +are some architecture-specific encoding differences. +See \textit{include/ARCH/ioctl.h} for more details. + +The payload struct layout always matches the 64-bit, little-endian +representation of the corresponding V4L2 structure. For most structs, the +size is identical for both 32 and 64 bits versions. Otherwise, the driver +must translate them to the aforementioned size and endianess. + +The device responds to \textit{VIRTIO_MEDIA_CMD_IOCTL} with \textit{virtio_media_resp_ioctl}. + +\begin{lstlisting} +struct virtio_media_resp_ioctl { + struct virtio_media_resp_header hdr; + /* Followed by the ioctl payload as defined in the macro */ +}; +\end{lstlisting} + +\subparagraph{Ioctls payload}\label{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} + +Each ioctl has a payload, which is defined by the third argument of the +\field{_IO*} macro. + +The payload of an ioctl in the descriptor chain follows the command structure, +the response structure, or both depending on the direction: + +\begin{itemize} +\item \textbf{_IOR} is read-only for the driver, meaning the payload +follows the response in the device-writable section of the descriptor chain. +\item \textbf{_IOW} is read-only for the device, meaning the payload +follows the command in the driver-writable section of the descriptor chain. +\item \textbf{_IOWR} is writable by both the device and driver, +meaning the payload must follow both the command in the driver-writable section +of the descriptor chain, and the response in the device-writable section. +\end{itemize} + +A common optimization for \textit{WR} ioctls is to provide the payload using +descriptors that both point to the same buffer. This mimics the behavior of +V4L2 ioctls where the data is only passed once and used as both input and +output by the kernel. + +\devicenormative{\subparagraph}{Device Operation: V4L2 ioctls}{Device Types / Media Device / Device Operation / V4L2 ioctls} + +In case of success of a device-writable ioctl, the device MUST always write the +payload in the device-writable part of the descriptor chain. + +In case of failure of a device-writable ioctl, the device is free to write the +payload in the device-writable part of the descriptor chain or not. Some errors +may still result in the payload being updated, and in this case the device is +expected to write the updated payload. If the device has not written the
device-types/media/description.tex line 286
Quoted 10 lines
device-types/media/description.tex new file +10
-0+payload after an error (i.e., only the header is returned), the driver MUST +assume that the payload has not been modified. + +\subparagraph{Handling of pointers in ioctl payload} + +A few structures used as ioctl payloads contain pointers to further +data needed for the ioctl. There are notably: + +\begin{itemize} +\item The \field{planes} pointer of
Do you mean points to?
Quoted 108 lines
device-types/media/description.tex new file +108
-0+\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-buffer}{struct v4l2_buffer}, +which size is determined by the length member. +\item The \field{controls} pointer of \textit{struct v4l2_ext_controls}, which +size is determined by the count member. +\end{itemize} + +If the size of the pointed area is determined to be non-zero, then the main +payload is immediately followed by the pointed data in their order of +appearance in the structure, and the pointer value itself is ignored by the +device, which must also return the value initially passed by the driver. + +\subparagraph{Handling of pointers to userspace memory} +\label{sec:Device Types / Media Device / V4L2 ioctls / Userspace memory} + +A few pointers are special in that they point to userspace memory in the +original V4L2 specification. They are: + +\begin{itemize} +\item The \field{m.userptr} member of \textit{struct v4l2_buffer} and +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-plane}{struct v4l2_plane} +(technically an unsigned long, but designated a userspace address). +\item The \field{ptr} member of \textit{struct v4l2_ext_ctrl}. +\end{itemize} + +These pointers can cover large areas of scattered memory, which has the +potential to require more descriptors than the virtio queue can provide. For +these particular pointers only, a list of \textit{struct virtio_media_sg_entry} +that covers the needed amount of memory for the pointer is used instead of +using descriptors to map the pointed memory directly. + +\begin{lstlisting} +struct virtio_media_sg_entry { + le64 start; + le32 len; + le32 __reserved; +}; +\end{lstlisting} + +For each such pointer to read, the device reads as many SG entries as needed +to cover the length of the pointed buffer, as described by its parent +structure (\field{length} member of \textit{struct v4l2_buffer} or +\textit{struct v4l2_plane} for buffer memory, and \field{size} member of +\textit{struct v4l2_ext_control} for control data). + +Since the device never needs to modify the list of SG entries, it is only +provided by the driver in the device-readable section of the descriptor chain, +and not repeated in the device-writable section, even for WR ioctls. + +\subparagraph{Unsupported ioctls} + +A few ioctls are replaced by other, more suitable mechanisms. + +\begin{itemize} +\item \textit{VIDIOC_QUERYCAP} is replaced by reading the configuration area +(see \ref{sec:Device Types / Media Device / Device Configuration Layout}). +\item \textit{VIDIOC_DQBUF} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Dequeue buffer}). +\item \textit{VIDIOC_DQEVENT} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Emit an event}). +\item \textit{VIDIOC_G_JPEGCOMP} and \textit{VIDIOC_S_JPEGCOMP} are deprecated +and replaced by the controls of the JPEG class. +\item \textit{VIDIOC_LOG_STATUS} is a driver-only operation and shall not be +implemented by the device. +\end{itemize} + +\devicenormative{\subparagraph}{Device Operation: Unsupported ioctls}{Device Types / Media Device / Device Operation / Unsupported ioctls} + +When a request is not supported, the device MUST return \textit{ENOTTY}, +which corresponds with the response for unknown ioctls. + +\paragraph{Device Operation: Mapping a MMAP buffer} + +\textbf{VIRTIO_MEDIA_CMD_MMAP} Command for mapping a MMAP buffer into the +driver's address space. + +Shared memory region ID 0 is used to map MMAP buffers with +the \textit{VIRTIO_MEDIA_CMD_MMAP} command. + +\begin{lstlisting} +#define VIRTIO_MEDIA_MMAP_FLAG_RW (1 << 0) + +struct virtio_media_cmd_mmap { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 flags; + le32 offset; +}; +\end{lstlisting} + +\begin{description} +\item[\field{flags}] is the set of flags for the mapping. \field{VIRTIO_MEDIA_MMAP_FLAG_RW} +can be set if a read-write mapping is desired. Without this flag the mapping +will be read-only. +\item[\field{offset}] corresponds to the \field{mem_offset} field of the +\textit{union v4l2_plane} for the plane to map. This field can be obtained +using the \textit{VIDIOC_QUERYBUF} ioctl. +\end{description} + +The device responds to \textit{VIRTIO_MEDIA_CMD_MMAP} with \textit{virtio_media_resp_mmap}. + +\begin{lstlisting} +struct virtio_media_resp_mmap { + struct virtio_media_resp_header hdr; + le64 driver_addr; + le64 len; +}; +\end{lstlisting} +
device-types/media/description.tex line 404
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\begin{description} +\item[\field{driver_addr}] offset into SHM region ID 0 of the start of the mapping. +\item[\field{len}] length of the mapping as indicated by the \textit{struct v4l2_plane} +the buffer belongs to. +\end{description} + +\paragraph{Device Operation: Unmapping a MMAP buffer} + +\textbf{VIRTIO_MEDIA_CMD_MUNMAP} Unmap a MMAP buffer previously mapped using \field{VIRTIO_MEDIA_CMD_MMAP}. +
s/Unmap/unmaps
Quoted 40 lines
device-types/media/description.tex new file +40
-0+\begin{lstlisting} +struct virtio_media_cmd_munmap { + struct virtio_media_cmd_header hdr; + le64 driver_addr; +}; +\end{lstlisting} + +\begin{description} +\item[\field{driver_addr}] offset into SHM region ID 0 previously returned by +\textit{VIRTIO_MEDIA_CMD_MMAP} at which the buffer has been previously mapped. +\end{description} + +The device responds to \textit{VIRTIO_MEDIA_CMD_MUNMAP} with \textit{virtio_media_resp_munmap}. + +\begin{lstlisting} +struct virtio_media_resp_munmap { + struct virtio_media_resp_header hdr; +}; +\end{lstlisting} + +\devicenormative{\subparagraph}{Device Operation: Unmapping a MMAP buffer}{Device Types / Media Device / Device Operation / Unmapping a MMAP buffer} + +The device MUST keep mappings performed using \textit{VIRTIO_MEDIA_CMD_MMAP} +valid until \textit{VIRTIO_MEDIA_CMD_MUNMAP} is called, even if the buffers or +session they belong to are released or closed by the driver. + +\paragraph{Device Operation: Memory Types} + +The semantics of the three V4L2 memory types (\textit{MMAP}, \textit{USERPTR} +and \textit{DMABUF}) can easily be mapped to both driver and device context. + +However, the driver shall only advertise support for \textit{MMAP} and +\textit{DMABUF} to the guest userspace. \textit{USERPTR} is merely employed to +discern \textit{SHARED_PAGES} buffers, similar to how \textit{DMABUF} is used +to signify \textit{VIRTIO_OBJECT} buffers in the driver and device context. + +\subparagraph{MMAP} + +In virtio-media, \textit{MMAP} buffers are provisioned by the device, just like +they are by the kernel in regular V4L2. Similarly to how userspace can map a
device-types/media/description.tex line 454
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\textit{MMAP} buffer into its address space using mmap and munmap, the +virtio-media driver can map device buffers into the driver space by queueing the +\textit{struct virtio_media_cmd_mmap} and \textit{struct virtio_media_cmd_munmap} +commands to the commandq. + +\subparagraph{SHARED_PAGES} + +In virtio-media, \textit{SHARED_PAGES} buffers are provisioned by the driver, +and use guest physical addresses. Instances of \textit{struct v4l2_buffer} +and \textit{struct v4l2_plane} of this type are followed by a list of
I think here is these types.
Quoted 9 lines
device-types/media/description.tex new file +9
-0+\textit{struct virtio_media_sg_entry}. For more information, see +\ref{sec:Device Types / Media Device / V4L2 ioctls / Userspace memory} + +The device must not alter the pointer values provided by the driver, i.e. +\field{the m.userptr} member of \textit{struct v4l2_buffer} and +\textit{struct v4l2_plane} must be returned to the driver with the same value +as it was provided. + +\subparagraph{VIRTIO_OBJECT}
device-types/media/description.tex line 473
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +In virtio-media, \textit{VIRTIO_OBJECT} buffers are provisioned by a virtio +object, just like they are by a \textit{DMABUF} in regular V4L2. Virtio objects +are 16-bytes UUIDs and do not fit in the placeholders for file descriptors, so +they follow their embedding data structure as needed and the device must +leave the V4L2 structure placeholder unchanged. + +Contrary to \textit{USERPTR} buffers, virtio objects UUIDs need to be added in +both the device-readable and device-writable section of the descriptor chain. +
I think you should use Conversely to.
device-types/media/description.tex line 482
Quoted 9 lines
device-types/media/description.tex new file +9
-0+Device-allocated buffers with the \textit{V4L2_MEMORY_MMAP} memory type can also +be exported as virtio objects for use with another virtio device using the +\textit{VIDIOC_EXPBUF} ioctl. The fd placefolder of \textit{v4l2_exportbuffer} +means that space for the UUID needs to be reserved right after that structure + +\subsubsection{Event Virtqueue} + +Events are a way for the device to inform the driver about asynchronous events +that it should know about. In virtio-media, they are used as a replacement for
I think you can rewrite the first sentence as (I do not have a strong opinion though):
Events are asynchronous notifications to the driver.
Quoted 34 lines
device-types/media/description.tex new file +34
-0+the \textit{VIDIOC_DQBUF} and \textit{VIDIOC_DQEVENT} ioctls and the polling +mechanism, which would be impractical to implement on top of virtio. + +\paragraph{Device Operation: Event header} + +\begin{lstlisting} +#define VIRTIO_MEDIA_EVT_ERROR 0 +#define VIRTIO_MEDIA_EVT_DQBUF 1 +#define VIRTIO_MEDIA_EVT_EVENT 2 + +/* Header for events queued by the device for the driver on the eventq. */ +struct virtio_media_event_header { + le32 event; + le32 session_id; +}; +\end{lstlisting} + +\begin{description} +\item[\field{event}] one of \field{VIRTIO_MEDIA_EVT_*}. +\item[\field{session_id}] ID of the session the event applies to. +\end{description} + +\paragraph{Device Operation: Device-side error} + +\textbf{VIRTIO_MEDIA_EVT_ERROR} Upon receiving this event, the session +mentioned in the header is considered corrupted and automatically closed by +the device. + +\begin{lstlisting} +struct virtio_media_event_error { + struct virtio_media_event_header hdr; + le32 errno; + le32 __reserved; +};
device-types/media/description.tex line 526
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\end{lstlisting} + +\begin{description} +\item[\field{errno}] error code describing the kind of error that occurred. +\end{description} + +\paragraph{Device Operation: Dequeue buffer} +\label{sec:Device Types / Media Device / Device Operation / Dequeue buffer} + +\textbf{VIRTIO_MEDIA_EVT_DQBUF} Signals that a buffer is not being used anymore
s/Signals/signals
device-types/media/description.tex line 533
Quoted 7 lines
device-types/media/description.tex new file +7
-0+by the device and is returned to the driver. + +A \textit{struct virtio_media_event_dqbuf} event is queued on the eventq by the +device every time a buffer previously queued using the \textit{VIDIOC_QBUF} +ioctl is done being processed and can be used by the driver again. This is like +an implicit \textit{VIDIOC_DQBUF} ioctl. +
I think this paragraph could be rewritten. If I understand correctly, every time a buffer is queued using the VIDIOC_QBUF ioctl, the device queues a virtio_media_event_dqbuf event in the eventq to indicate the driver that the buffer can be used again.
Quoted 78 lines
device-types/media/description.tex new file +42
-0+\begin{lstlisting} +struct virtio_media_event_dqbuf { + struct virtio_media_event_header hdr; + struct v4l2_buffer buffer; + struct v4l2_plane planes[VIDEO_MAX_PLANES]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{buffer}] \textit{struct v4l2_buffer} describing the buffer that has been dequeued. +\item[\field{planes}] array of \textit{struct v4l2_plane} containing the plane information for multi-planar buffers. +\end{description} + +Pointer values in the \textit{struct v4l2_buffer} and \textit{struct v4l2_plane} +are meaningless and must be ignored by the driver. It is recommended that the +device sets them to NULL in order to avoid leaking potential device addresses. + +Note that in the case of a \field{USERPTR} buffer, the \textit{struct v4l2_buffer} +used as event payload is not followed by the buffer memory: since that memory +is the same that the driver submitted with the \textit{VIDIOC_QBUF}, it would +be redundant to have it here. + +\paragraph{Device Operation: Emit an event} +\label{sec:Device Types / Media Device / Device Operation / Emit an event} + +\textbf{VIRTIO_MEDIA_EVT_EVENT} Signals that a V4L2 event has been emitted for a session. + +A \textit{struct virtio_media_event_event} event is queued on the eventq by the +device every time an event the driver previously subscribed to using the +\textit{VIDIOC_SUBSCRIBE_EVENT} ioctl has been signaled. This is like an +implicit \textit{VIDIOC_DQEVENT} ioctl. + +\begin{lstlisting} +struct virtio_media_event_event { + struct virtio_media_event_header hdr; + struct v4l2_event event; +}; +\end{lstlisting} + +\begin{description} +\item[\field{event}] \textit{struct v4l2_event} describing the event that occurred. +\end{description}device-types/media/device-conformance.tex new file +10
-0@@ -0,0 +1,10 @@ +\conformance{\subsection}{Media Device Conformance}\label{sec:Conformance / Device Conformance / Media Device Conformance} + +A Media device MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Open device} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / V4L2 ioctls} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Unsupported ioctls} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Unmapping a MMAP buffer} +\end{itemize}device-types/media/driver-conformance.tex new file +8
-0@@ -0,0 +1,8 @@ +\conformance{\subsection}{Media Device Conformance}\label{sec:Conformance / Driver Conformance / Media Driver Conformance} + +A Media device MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{drivernormative:Device Types / Media Device / Device Operation / Command Virtqueue} +\item \ref{drivernormative:Device Types / Media Device / Device Operation / Close device} +\end{itemize}Signature · 1 line
2.47.0
Re: [PATCH v4 1/1] virtio-media: Add virtio media device specification
Hi Matias,
Quoted 407 lines · Matias Ezequiel Vara Larsen
Hello Albert,
I added some minor comments below. I may need to read it again since there are some points that I did not understand completely.
Quoted 138 lines · Albert Esteve
Virtio-media is an encapsulation of the V4L2 UAPI into virtio, able to virtualize any video device supported by V4L2.
Note that virtio-media does not require the use of a V4L2 device driver on the host or guest side - V4L2 is only used as a host-guest protocol, and both sides are free to convert it from/to any model that they wish to use.
Signed-off-by Albert Esteve <[email protected]>--- conformance.tex | 13 +- content.tex | 1 + device-types/media/description.tex | 575 ++++++++++++++++++++++ device-types/media/device-conformance.tex | 10 + device-types/media/driver-conformance.tex | 8 + 5 files changed, 603 insertions(+), 4 deletions(-) create mode 100644 device-types/media/description.tex create mode 100644 device-types/media/device-conformance.tex create mode 100644 device-types/media/driver-conformance.texconformance.tex +9
-4@@ -32,8 +32,10 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Driver Conformance / Memory Driver Conformance}, \ref{sec:Conformance / Driver Conformance / I2C Adapter Driver Conformance}, \ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance}, -\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance} or -\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}. +\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance}, +\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance} or +\ref{sec:Conformance / Driver Conformance / Media Driver Conformance}. + \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -59,8 +61,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Device Conformance / Memory Device Conformance}, \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance}, \ref{sec:Conformance / Device Conformance / SCMI Device Conformance}, -\ref{sec:Conformance / Device Conformance / GPIO Device Conformance} or -\ref{sec:Conformance / Device Conformance / PMEM Device Conformance}. +\ref{sec:Conformance / Device Conformance / GPIO Device Conformance}, +\ref{sec:Conformance / Device Conformance / PMEM Device Conformance} or +\ref{sec:Conformance / Device Conformance / Media Device Conformance}. \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -152,6 +155,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/driver-conformance.tex} \input{device-types/gpio/driver-conformance.tex} \input{device-types/pmem/driver-conformance.tex} +\input{device-types/media/driver-conformance.tex} \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}@@ -238,6 +242,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/device-conformance.tex} \input{device-types/gpio/device-conformance.tex} \input{device-types/pmem/device-conformance.tex} +\input{device-types/media/device-conformance.tex} \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance} A conformant implementation MUST be either transitional orcontent.tex +1
-0@@ -767,6 +767,7 @@ \chapter{Device Types}\label{sec:Device Types} \input{device-types/scmi/description.tex} \input{device-types/gpio/description.tex} \input{device-types/pmem/description.tex} +\input{device-types/media/description.tex} \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}device-types/media/description.tex new file +53
-0@@ -0,0 +1,575 @@ +\section{Media Device}\label{sec:Device Types / Media Device} + +The virtio media device follows the same model (and structures) as V4L2. It +can be used to virtualize cameras, codec devices, or any other device +supported by V4L2. The complete definition of V4L2 structures and ioctls can +be found under the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/index.html}{V4L2 UAPI documentation}. + +V4L2 is a UAPI that allows a less privileged entity (user-space) to use video +hardware exposed by a more privileged entity (the kernel). Virtio-media is an +encapsulation of this API into virtio, turning it into a virtualization API +for all classes of video devices supported by V4L2, where the device plays the +role of the kernel and the driver the role of user-space. + +The device is therefore responsible for presenting a virtual device that behaves +like an actual V4L2 device, which the driver can control. + +Note that virtio-media does not require the use of a V4L2 device driver or of +Linux on any side - V4L2 is only used as a transport protocol, +and both sides are free to convert it from/to any model that they wish to use. + +\subsection{Device ID}\label{sec:Device Types / Media Device / Device ID} + +48 + +\subsection{Virtqueues}\label{sec:Device Types / Media Device / Virtqueues} + +\begin{description} +\item[0] commandq - used for driver commands and device responses to these +commands. +\item[1] eventq - used for events sent by the device to the driver. +\end{description} + +\subsection{Feature Bits}\label{sec:Device Types / Media Device / Feature Bits} + +None + +\subsection{Device Configuration Layout}\label{sec:Device Types / Media Device / Device Configuration Layout} + +The video device configuration space uses the following layout: + +\begin{lstlisting} +struct virtio_media_config { + le32 device_caps; + le32 device_type; + le8 card[32]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{device_caps}] (driver-read-only) flags representing the device +capabilities as used in +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/vidioc-querycap.html#c.V4L.v4l2_capability}{struct v4l2_capability}.device-types/media/description.tex line 63
Quoted 10 lines
device-types/media/description.tex new file +10
-0+It corresponds with the \field{device_caps} field in the \textit{struct video_device}. +\item[\field{device_type}] (driver-read-only) informs the driver of the type +of the video device. It corresponds with the \field{vfl_devnode_type} field of the device. +\item[\field{card}] (driver-read-only) name of the device, a NUL-terminated +UTF-8 string. It corresponds with the \field{card} field of the \textit{struct v4l2_capability}. +If all the characters of the field are used, it does not need to be NUL-terminated. +\end{description} + +\subsection{Device Initialization} +Just to clarify, I would add a sentence here like:
A driver executes the following sequence to initialize a device:device-types/media/description.tex line 67
Quoted 4 lines
device-types/media/description.tex new file +4
-0+\begin{enumerate} +\item Read the \field{device_caps} and \field{device_type} fields +from the configuration layout to identify the device. +\item Set up the \field{commandq} and \field{eventq}.Maybe add
... \field{commandq} and \field{eventq} virtqueues.device-types/media/description.tex line 76
Quoted 9 lines
device-types/media/description.tex new file +9
-0+\item May open a session (see Section \ref{sec:Device Operation: Command Virtqueue: Sessions}) +to use the device and send V4L2 ioctls in order to receive more information +about the device, such as supported formats or controls. +\end{enumerate} + +\subsection{Device Operation}\label{sec:Device Types / Media Device / Device Operation} + +Commands are queued on the command queue by the driver for the device to +process. The errors returned by each command are standardI would rewrite it as below but I do not have a strong opinion:
The driver enqueues commands in the command queue for the device to process them.device-types/media/description.tex line 79
Quoted 3 lines
device-types/media/description.tex new file +3
-0+\href{https://www.kernel.org/doc/html/latest/userspace-api/media/gen-errors.html}{Linux kernel error codes}. +For instance, a command that contains invalid options will return \textit{EINVAL}. +I think here is not the command that returns EINVAL but the device when trying to process a command that has an invalid option.
device-types/media/description.tex line 80
Quoted 1 line
device-types/media/description.tex new file +1
-0+Events are sent on the event queue by the device for the driver to handle.I would rewrite it as below but I do not have a strong opinion:
The device enqueues events in the event queue for the driver to process them.Quoted 31 lines
device-types/media/description.tex new file +31
-0+ +\subsubsection{Command Virtqueue} + +\paragraph{Device Operation: Command headers} + +\begin{lstlisting} +#define VIRTIO_MEDIA_CMD_OPEN 1 +#define VIRTIO_MEDIA_CMD_CLOSE 2 +#define VIRTIO_MEDIA_CMD_IOCTL 3 +#define VIRTIO_MEDIA_CMD_MMAP 4 +#define VIRTIO_MEDIA_CMD_MUNMAP 5 + +/* Header for all virtio commands from the driver to the device on the commandq. */ +struct virtio_media_cmd_header { + le32 cmd; + le32 __reserved; +}; + +/* Header for all virtio responses from the device to the driver on the commandq. */ +struct virtio_media_resp_header { + le32 status; + le32 __reserved; +}; +\end{lstlisting} + +A command consists of a command header \textit{virtio_media_cmd_header} +containing the following device-readable field: + +\begin{description} +\item[\field{cmd}] specifies a device request type (VIRTIO_MEDIA_CMD_*). +\end{description}device-types/media/description.tex line 121
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +A response consists of a response header \textit{virtio_media_resp_header} +containing the following device-writable field: + +\begin{description} +\item[\field{status}] indicates a device request status. +\end{description} + +The status field can take 0 if the command was successful, or one of the +standard Linux error codes if it was not.I would rewrite it as but I do not have a strong opinion about it:
`When the device executes the command successfully, the value of the status field is 0. When the device fails to execute the command, the value of the status field is one of the standard Linux error codes.`
Quoted 51 lines
device-types/media/description.tex new file +51
-0+ +\drivernormative{\paragraph}{Device Operation: Command Virtqueue: Sessions}{Device Types / Media Device / Device Operation / Command Virtqueue} + +Sessions are how the device is multiplexed, allowing several distinct works to +take place simultaneously. Before start operating, the driver needs to open a +session. This is equivalent to opening the \textit{/dev/videoX} file of the +V4L2 device. Each session gets a unique ID assigned, which can be then used +to perform actions on it. + +\paragraph{Device Operation: Open device} + +\textbf{VIRTIO_MEDIA_CMD_OPEN} Command for creating a new session. + +This is the equivalent of calling \textit{open} on a V4L2 device node. +The driver uses \textit{virtio_media_cmd_open} to send an open request. + +\begin{lstlisting} +struct virtio_media_cmd_open { + struct virtio_media_cmd_header hdr; +}; +\end{lstlisting} + +The device responds to \textit{VIRTIO_MEDIA_CMD_OPEN} with \textit{virtio_media_resp_open}. + +\begin{lstlisting} +struct virtio_media_resp_open { + struct virtio_media_resp_header hdr; + le32 session_id; + le32 __reserved; +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the current session, which is used for +other commands, predominantly ioctls. +\end{description} + +\devicenormative{\subparagraph}{Device Operation: Open device}{Device Types / Media Device / Device Operation / Open device} + +Upon success, the device MUST set a \field{session_id} in \textit{virtio_media_resp_open} +to an integer that is NOT used by any other open session. + +\paragraph{Device Operation: Close device} + +\textbf{VIRTIO_MEDIA_CMD_CLOSE} Command for closing an active session. + +This is the equivalent of calling \textit{close} on a previously opened V4L2 +device node. All resources associated with this session will be freed. + +This command does not require a response from the device. +device-types/media/description.tex line 182
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\begin{lstlisting} +struct virtio_media_cmd_close { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 __reserved; +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] specifies an identifier for the session to close.I would rewrite it shorter as:
session_id identifies the session to close.Quoted 98 lines
device-types/media/description.tex new file +98
-0+\end{description} + +\drivernormative{\subparagraph}{Device Operation: Close device}{Device Types / Media Device / Device Operation / Close device} + +The session ID SHALL NOT be used again after queueing this command. + +\paragraph{Device Operation: V4L2 ioctls} + +\textbf{VIRTIO_MEDIA_CMD_IOCTL} Command for executing an ioctl on an open +session. + +This command tells the device to run one of the `VIDIOC_*` ioctls on the +session identified by \textit{session_id}. + +\begin{lstlisting} +struct virtio_media_cmd_ioctl { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 code; + /* Followed by the relevant ioctl payload as defined in the macro */ +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the session to run the ioctl on. +\item[\field{code}] specifies the code of the \field{VIDIOC_*} ioctl to run. +\end{description} + +The code is extracted from the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/videodev.html}{videodev2.h}, +header file. The file defines the ioctl's codes, type of payload, and +direction. The code consists of the second argument of the \field{_IO*} macro. + +For example, the \textit{VIDIOC_G_FMT} is defined as follows: + +\begin{lstlisting} +#define VIDIOC_G_FMT _IOWR('V', 4, struct v4l2_format) +\end{lstlisting} + +This means that its ioctl code is \textit{4}, its payload is a +\textit{struct v4l2_format}, and its direction is \textit{WR} (i.e., the +payload is written by both the driver and the device). +See Section \ref{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} +for more information about the direction of ioctls. +Note that although most architectures use this format, there +are some architecture-specific encoding differences. +See \textit{include/ARCH/ioctl.h} for more details. + +The payload struct layout always matches the 64-bit, little-endian +representation of the corresponding V4L2 structure. For most structs, the +size is identical for both 32 and 64 bits versions. Otherwise, the driver +must translate them to the aforementioned size and endianess. + +The device responds to \textit{VIRTIO_MEDIA_CMD_IOCTL} with \textit{virtio_media_resp_ioctl}. + +\begin{lstlisting} +struct virtio_media_resp_ioctl { + struct virtio_media_resp_header hdr; + /* Followed by the ioctl payload as defined in the macro */ +}; +\end{lstlisting} + +\subparagraph{Ioctls payload}\label{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} + +Each ioctl has a payload, which is defined by the third argument of the +\field{_IO*} macro. + +The payload of an ioctl in the descriptor chain follows the command structure, +the response structure, or both depending on the direction: + +\begin{itemize} +\item \textbf{_IOR} is read-only for the driver, meaning the payload +follows the response in the device-writable section of the descriptor chain. +\item \textbf{_IOW} is read-only for the device, meaning the payload +follows the command in the driver-writable section of the descriptor chain. +\item \textbf{_IOWR} is writable by both the device and driver, +meaning the payload must follow both the command in the driver-writable section +of the descriptor chain, and the response in the device-writable section. +\end{itemize} + +A common optimization for \textit{WR} ioctls is to provide the payload using +descriptors that both point to the same buffer. This mimics the behavior of +V4L2 ioctls where the data is only passed once and used as both input and +output by the kernel. + +\devicenormative{\subparagraph}{Device Operation: V4L2 ioctls}{Device Types / Media Device / Device Operation / V4L2 ioctls} + +In case of success of a device-writable ioctl, the device MUST always write the +payload in the device-writable part of the descriptor chain. + +In case of failure of a device-writable ioctl, the device is free to write the +payload in the device-writable part of the descriptor chain or not. Some errors +may still result in the payload being updated, and in this case the device is +expected to write the updated payload. If the device has not written the +payload after an error (i.e., only the header is returned), the driver MUST +assume that the payload has not been modified. + +\subparagraph{Handling of pointers in ioctl payload}
Quoted 8 lines
device-types/media/description.tex line 286
Quoted 6 lines
device-types/media/description.tex new file +6
-0+ +A few structures used as ioctl payloads contain pointers to further +data needed for the ioctl. There are notably: + +\begin{itemize} +\item The \field{planes} pointer ofDo you mean
points to?
Here we specify the type ""of"" the data the pointer is pointing to. I feel this construction is correct as is, although 'points to' would mean the same, and be equally valid. I will keep it as is unless there is a strong preference.
Quoted 166 lines
Quoted 109 lines
device-types/media/description.tex new file +108
-0+\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-buffer}{struct v4l2_buffer}, +which size is determined by the length member. +\item The \field{controls} pointer of \textit{struct v4l2_ext_controls}, which +size is determined by the count member. +\end{itemize} + +If the size of the pointed area is determined to be non-zero, then the main +payload is immediately followed by the pointed data in their order of +appearance in the structure, and the pointer value itself is ignored by the +device, which must also return the value initially passed by the driver. + +\subparagraph{Handling of pointers to userspace memory} +\label{sec:Device Types / Media Device / V4L2 ioctls / Userspace memory} + +A few pointers are special in that they point to userspace memory in the +original V4L2 specification. They are: + +\begin{itemize} +\item The \field{m.userptr} member of \textit{struct v4l2_buffer} and +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-plane}{struct v4l2_plane} +(technically an unsigned long, but designated a userspace address). +\item The \field{ptr} member of \textit{struct v4l2_ext_ctrl}. +\end{itemize} + +These pointers can cover large areas of scattered memory, which has the +potential to require more descriptors than the virtio queue can provide. For +these particular pointers only, a list of \textit{struct virtio_media_sg_entry} +that covers the needed amount of memory for the pointer is used instead of +using descriptors to map the pointed memory directly. + +\begin{lstlisting} +struct virtio_media_sg_entry { + le64 start; + le32 len; + le32 __reserved; +}; +\end{lstlisting} + +For each such pointer to read, the device reads as many SG entries as needed +to cover the length of the pointed buffer, as described by its parent +structure (\field{length} member of \textit{struct v4l2_buffer} or +\textit{struct v4l2_plane} for buffer memory, and \field{size} member of +\textit{struct v4l2_ext_control} for control data). + +Since the device never needs to modify the list of SG entries, it is only +provided by the driver in the device-readable section of the descriptor chain, +and not repeated in the device-writable section, even for WR ioctls. + +\subparagraph{Unsupported ioctls} + +A few ioctls are replaced by other, more suitable mechanisms. + +\begin{itemize} +\item \textit{VIDIOC_QUERYCAP} is replaced by reading the configuration area +(see \ref{sec:Device Types / Media Device / Device Configuration Layout}). +\item \textit{VIDIOC_DQBUF} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Dequeue buffer}). +\item \textit{VIDIOC_DQEVENT} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Emit an event}). +\item \textit{VIDIOC_G_JPEGCOMP} and \textit{VIDIOC_S_JPEGCOMP} are deprecated +and replaced by the controls of the JPEG class. +\item \textit{VIDIOC_LOG_STATUS} is a driver-only operation and shall not be +implemented by the device. +\end{itemize} + +\devicenormative{\subparagraph}{Device Operation: Unsupported ioctls}{Device Types / Media Device / Device Operation / Unsupported ioctls} + +When a request is not supported, the device MUST return \textit{ENOTTY}, +which corresponds with the response for unknown ioctls. + +\paragraph{Device Operation: Mapping a MMAP buffer} + +\textbf{VIRTIO_MEDIA_CMD_MMAP} Command for mapping a MMAP buffer into the +driver's address space. + +Shared memory region ID 0 is used to map MMAP buffers with +the \textit{VIRTIO_MEDIA_CMD_MMAP} command. + +\begin{lstlisting} +#define VIRTIO_MEDIA_MMAP_FLAG_RW (1 << 0) + +struct virtio_media_cmd_mmap { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 flags; + le32 offset; +}; +\end{lstlisting} + +\begin{description} +\item[\field{flags}] is the set of flags for the mapping. \field{VIRTIO_MEDIA_MMAP_FLAG_RW} +can be set if a read-write mapping is desired. Without this flag the mapping +will be read-only. +\item[\field{offset}] corresponds to the \field{mem_offset} field of the +\textit{union v4l2_plane} for the plane to map. This field can be obtained +using the \textit{VIDIOC_QUERYBUF} ioctl. +\end{description} + +The device responds to \textit{VIRTIO_MEDIA_CMD_MMAP} with \textit{virtio_media_resp_mmap}. + +\begin{lstlisting} +struct virtio_media_resp_mmap { + struct virtio_media_resp_header hdr; + le64 driver_addr; + le64 len; +}; +\end{lstlisting} +device-types/media/description.tex line 404
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\begin{description} +\item[\field{driver_addr}] offset into SHM region ID 0 of the start of the mapping. +\item[\field{len}] length of the mapping as indicated by the \textit{struct v4l2_plane} +the buffer belongs to. +\end{description} + +\paragraph{Device Operation: Unmapping a MMAP buffer} + +\textbf{VIRTIO_MEDIA_CMD_MUNMAP} Unmap a MMAP buffer previously mapped using \field{VIRTIO_MEDIA_CMD_MMAP}. +s/Unmap/unmaps
Quoted 44 lines
device-types/media/description.tex new file +44
-0+\begin{lstlisting} +struct virtio_media_cmd_munmap { + struct virtio_media_cmd_header hdr; + le64 driver_addr; +}; +\end{lstlisting} + +\begin{description} +\item[\field{driver_addr}] offset into SHM region ID 0 previously returned by +\textit{VIRTIO_MEDIA_CMD_MMAP} at which the buffer has been previously mapped. +\end{description} + +The device responds to \textit{VIRTIO_MEDIA_CMD_MUNMAP} with \textit{virtio_media_resp_munmap}. + +\begin{lstlisting} +struct virtio_media_resp_munmap { + struct virtio_media_resp_header hdr; +}; +\end{lstlisting} + +\devicenormative{\subparagraph}{Device Operation: Unmapping a MMAP buffer}{Device Types / Media Device / Device Operation / Unmapping a MMAP buffer} + +The device MUST keep mappings performed using \textit{VIRTIO_MEDIA_CMD_MMAP} +valid until \textit{VIRTIO_MEDIA_CMD_MUNMAP} is called, even if the buffers or +session they belong to are released or closed by the driver. + +\paragraph{Device Operation: Memory Types} + +The semantics of the three V4L2 memory types (\textit{MMAP}, \textit{USERPTR} +and \textit{DMABUF}) can easily be mapped to both driver and device context. + +However, the driver shall only advertise support for \textit{MMAP} and +\textit{DMABUF} to the guest userspace. \textit{USERPTR} is merely employed to +discern \textit{SHARED_PAGES} buffers, similar to how \textit{DMABUF} is used +to signify \textit{VIRTIO_OBJECT} buffers in the driver and device context. + +\subparagraph{MMAP} + +In virtio-media, \textit{MMAP} buffers are provisioned by the device, just like +they are by the kernel in regular V4L2. Similarly to how userspace can map a +\textit{MMAP} buffer into its address space using mmap and munmap, the +virtio-media driver can map device buffers into the driver space by queueing the +\textit{struct virtio_media_cmd_mmap} and \textit{struct virtio_media_cmd_munmap} +commands to the commandq.
Quoted 8 lines
device-types/media/description.tex line 454
Quoted 6 lines
device-types/media/description.tex new file +6
-0+ +\subparagraph{SHARED_PAGES} + +In virtio-media, \textit{SHARED_PAGES} buffers are provisioned by the driver, +and use guest physical addresses. Instances of \textit{struct v4l2_buffer} +and \textit{struct v4l2_plane} of this type are followed by a list ofI think here is
these types.
It refers to the memory type. I will clarify in the text.
Quoted 84 lines
Quoted 10 lines
device-types/media/description.tex new file +9
-0+\textit{struct virtio_media_sg_entry}. For more information, see +\ref{sec:Device Types / Media Device / V4L2 ioctls / Userspace memory} + +The device must not alter the pointer values provided by the driver, i.e. +\field{the m.userptr} member of \textit{struct v4l2_buffer} and +\textit{struct v4l2_plane} must be returned to the driver with the same value +as it was provided. + +\subparagraph{VIRTIO_OBJECT}device-types/media/description.tex line 473
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +In virtio-media, \textit{VIRTIO_OBJECT} buffers are provisioned by a virtio +object, just like they are by a \textit{DMABUF} in regular V4L2. Virtio objects +are 16-bytes UUIDs and do not fit in the placeholders for file descriptors, so +they follow their embedding data structure as needed and the device must +leave the V4L2 structure placeholder unchanged. + +Contrary to \textit{USERPTR} buffers, virtio objects UUIDs need to be added in +both the device-readable and device-writable section of the descriptor chain. +I think you should use
Conversely to.device-types/media/description.tex line 482
Quoted 9 lines
device-types/media/description.tex new file +9
-0+Device-allocated buffers with the \textit{V4L2_MEMORY_MMAP} memory type can also +be exported as virtio objects for use with another virtio device using the +\textit{VIDIOC_EXPBUF} ioctl. The fd placefolder of \textit{v4l2_exportbuffer} +means that space for the UUID needs to be reserved right after that structure + +\subsubsection{Event Virtqueue} + +Events are a way for the device to inform the driver about asynchronous events +that it should know about. In virtio-media, they are used as a replacement forI think you can rewrite the first sentence as (I do not have a strong opinion though):
Events are asynchronous notifications to the driver.Quoted 34 lines
device-types/media/description.tex new file +34
-0+the \textit{VIDIOC_DQBUF} and \textit{VIDIOC_DQEVENT} ioctls and the polling +mechanism, which would be impractical to implement on top of virtio. + +\paragraph{Device Operation: Event header} + +\begin{lstlisting} +#define VIRTIO_MEDIA_EVT_ERROR 0 +#define VIRTIO_MEDIA_EVT_DQBUF 1 +#define VIRTIO_MEDIA_EVT_EVENT 2 + +/* Header for events queued by the device for the driver on the eventq. */ +struct virtio_media_event_header { + le32 event; + le32 session_id; +}; +\end{lstlisting} + +\begin{description} +\item[\field{event}] one of \field{VIRTIO_MEDIA_EVT_*}. +\item[\field{session_id}] ID of the session the event applies to. +\end{description} + +\paragraph{Device Operation: Device-side error} + +\textbf{VIRTIO_MEDIA_EVT_ERROR} Upon receiving this event, the session +mentioned in the header is considered corrupted and automatically closed by +the device. + +\begin{lstlisting} +struct virtio_media_event_error { + struct virtio_media_event_header hdr; + le32 errno; + le32 __reserved; +};device-types/media/description.tex line 526
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\end{lstlisting} + +\begin{description} +\item[\field{errno}] error code describing the kind of error that occurred. +\end{description} + +\paragraph{Device Operation: Dequeue buffer} +\label{sec:Device Types / Media Device / Device Operation / Dequeue buffer} + +\textbf{VIRTIO_MEDIA_EVT_DQBUF} Signals that a buffer is not being used anymores/Signals/signals
Quoted 12 lines
device-types/media/description.tex line 533
Quoted 7 lines
device-types/media/description.tex new file +7
-0+by the device and is returned to the driver. + +A \textit{struct virtio_media_event_dqbuf} event is queued on the eventq by the +device every time a buffer previously queued using the \textit{VIDIOC_QBUF} +ioctl is done being processed and can be used by the driver again. This is like +an implicit \textit{VIDIOC_DQBUF} ioctl. +I think this paragraph could be rewritten. If I understand correctly, every time a buffer is queued using the
VIDIOC_QBUFioctl, the device queues avirtio_media_event_dqbufevent in the eventq to indicate the driver that the buffer can be used again.
Yes, that is more or less the idea. Once the buffer is filled (or displayed), and
is ready to be reused, the device sends this event to the driver. I will rewrite this.
Also complied with all the other changes in your comment.
Thanks for taking the time to review!
Quoted 80 lines
Quoted 79 lines
device-types/media/description.tex new file +42
-0+\begin{lstlisting} +struct virtio_media_event_dqbuf { + struct virtio_media_event_header hdr; + struct v4l2_buffer buffer; + struct v4l2_plane planes[VIDEO_MAX_PLANES]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{buffer}] \textit{struct v4l2_buffer} describing the buffer that has been dequeued. +\item[\field{planes}] array of \textit{struct v4l2_plane} containing the plane information for multi-planar buffers. +\end{description} + +Pointer values in the \textit{struct v4l2_buffer} and \textit{struct v4l2_plane} +are meaningless and must be ignored by the driver. It is recommended that the +device sets them to NULL in order to avoid leaking potential device addresses. + +Note that in the case of a \field{USERPTR} buffer, the \textit{struct v4l2_buffer} +used as event payload is not followed by the buffer memory: since that memory +is the same that the driver submitted with the \textit{VIDIOC_QBUF}, it would +be redundant to have it here. + +\paragraph{Device Operation: Emit an event} +\label{sec:Device Types / Media Device / Device Operation / Emit an event} + +\textbf{VIRTIO_MEDIA_EVT_EVENT} Signals that a V4L2 event has been emitted for a session. + +A \textit{struct virtio_media_event_event} event is queued on the eventq by the +device every time an event the driver previously subscribed to using the +\textit{VIDIOC_SUBSCRIBE_EVENT} ioctl has been signaled. This is like an +implicit \textit{VIDIOC_DQEVENT} ioctl. + +\begin{lstlisting} +struct virtio_media_event_event { + struct virtio_media_event_header hdr; + struct v4l2_event event; +}; +\end{lstlisting} + +\begin{description} +\item[\field{event}] \textit{struct v4l2_event} describing the event that occurred. +\end{description}device-types/media/device-conformance.tex new file +10
-0@@ -0,0 +1,10 @@ +\conformance{\subsection}{Media Device Conformance}\label{sec:Conformance / Device Conformance / Media Device Conformance} + +A Media device MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Open device} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / V4L2 ioctls} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Unsupported ioctls} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Unmapping a MMAP buffer} +\end{itemize}device-types/media/driver-conformance.tex new file +8
-1@@ -0,0 +1,8 @@ +\conformance{\subsection}{Media Device Conformance}\label{sec:Conformance / Driver Conformance / Media Driver Conformance} + +A Media device MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{drivernormative:Device Types / Media Device / Device Operation / Command Virtqueue} +\item \ref{drivernormative:Device Types / Media Device / Device Operation / Close device} +\end{itemize} --2.47.0
Re: [PATCH v4 1/1] virtio-media: Add virtio media device specification
Quoted 390 lines · Albert Esteve
Hi Matias,
Quoted 386 lines · Matias Ezequiel Vara Larsen
Hello Albert,
I added some minor comments below. I may need to read it again since there are some points that I did not understand completely.
Quoted 138 lines · Albert Esteve
Virtio-media is an encapsulation of the V4L2 UAPI into virtio, able to virtualize any video device supported by V4L2.
Note that virtio-media does not require the use of a V4L2 device driver on the host or guest side - V4L2 is only used as a host-guest protocol, and both sides are free to convert it from/to any model that they wish to use.
Signed-off-by Albert Esteve <[email protected]>--- conformance.tex | 13 +- content.tex | 1 + device-types/media/description.tex | 575 ++++++++++++++++++++++ device-types/media/device-conformance.tex | 10 + device-types/media/driver-conformance.tex | 8 + 5 files changed, 603 insertions(+), 4 deletions(-) create mode 100644 device-types/media/description.tex create mode 100644 device-types/media/device-conformance.tex create mode 100644 device-types/media/driver-conformance.texconformance.tex +9
-4@@ -32,8 +32,10 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Driver Conformance / Memory Driver Conformance}, \ref{sec:Conformance / Driver Conformance / I2C Adapter Driver Conformance}, \ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance}, -\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance} or -\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}. +\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance}, +\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance} or +\ref{sec:Conformance / Driver Conformance / Media Driver Conformance}. + \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -59,8 +61,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Device Conformance / Memory Device Conformance}, \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance}, \ref{sec:Conformance / Device Conformance / SCMI Device Conformance}, -\ref{sec:Conformance / Device Conformance / GPIO Device Conformance} or -\ref{sec:Conformance / Device Conformance / PMEM Device Conformance}. +\ref{sec:Conformance / Device Conformance / GPIO Device Conformance}, +\ref{sec:Conformance / Device Conformance / PMEM Device Conformance} or +\ref{sec:Conformance / Device Conformance / Media Device Conformance}. \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -152,6 +155,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/driver-conformance.tex} \input{device-types/gpio/driver-conformance.tex} \input{device-types/pmem/driver-conformance.tex} +\input{device-types/media/driver-conformance.tex} \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}@@ -238,6 +242,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/device-conformance.tex} \input{device-types/gpio/device-conformance.tex} \input{device-types/pmem/device-conformance.tex} +\input{device-types/media/device-conformance.tex} \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance} A conformant implementation MUST be either transitional orcontent.tex +1
-0@@ -767,6 +767,7 @@ \chapter{Device Types}\label{sec:Device Types} \input{device-types/scmi/description.tex} \input{device-types/gpio/description.tex} \input{device-types/pmem/description.tex} +\input{device-types/media/description.tex} \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}device-types/media/description.tex new file +53
-0@@ -0,0 +1,575 @@ +\section{Media Device}\label{sec:Device Types / Media Device} + +The virtio media device follows the same model (and structures) as V4L2. It +can be used to virtualize cameras, codec devices, or any other device +supported by V4L2. The complete definition of V4L2 structures and ioctls can +be found under the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/index.html}{V4L2 UAPI documentation}. + +V4L2 is a UAPI that allows a less privileged entity (user-space) to use video +hardware exposed by a more privileged entity (the kernel). Virtio-media is an +encapsulation of this API into virtio, turning it into a virtualization API +for all classes of video devices supported by V4L2, where the device plays the +role of the kernel and the driver the role of user-space. + +The device is therefore responsible for presenting a virtual device that behaves +like an actual V4L2 device, which the driver can control. + +Note that virtio-media does not require the use of a V4L2 device driver or of +Linux on any side - V4L2 is only used as a transport protocol, +and both sides are free to convert it from/to any model that they wish to use. + +\subsection{Device ID}\label{sec:Device Types / Media Device / Device ID} + +48 + +\subsection{Virtqueues}\label{sec:Device Types / Media Device / Virtqueues} + +\begin{description} +\item[0] commandq - used for driver commands and device responses to these +commands. +\item[1] eventq - used for events sent by the device to the driver. +\end{description} + +\subsection{Feature Bits}\label{sec:Device Types / Media Device / Feature Bits} + +None + +\subsection{Device Configuration Layout}\label{sec:Device Types / Media Device / Device Configuration Layout} + +The video device configuration space uses the following layout: + +\begin{lstlisting} +struct virtio_media_config { + le32 device_caps; + le32 device_type; + le8 card[32]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{device_caps}] (driver-read-only) flags representing the device +capabilities as used in +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/vidioc-querycap.html#c.V4L.v4l2_capability}{struct v4l2_capability}.device-types/media/description.tex line 63
Quoted 10 lines
device-types/media/description.tex new file +10
-0+It corresponds with the \field{device_caps} field in the \textit{struct video_device}. +\item[\field{device_type}] (driver-read-only) informs the driver of the type +of the video device. It corresponds with the \field{vfl_devnode_type} field of the device. +\item[\field{card}] (driver-read-only) name of the device, a NUL-terminated +UTF-8 string. It corresponds with the \field{card} field of the \textit{struct v4l2_capability}. +If all the characters of the field are used, it does not need to be NUL-terminated. +\end{description} + +\subsection{Device Initialization} +Just to clarify, I would add a sentence here like:
A driver executes the following sequence to initialize a device:device-types/media/description.tex line 67
Quoted 4 lines
device-types/media/description.tex new file +4
-0+\begin{enumerate} +\item Read the \field{device_caps} and \field{device_type} fields +from the configuration layout to identify the device. +\item Set up the \field{commandq} and \field{eventq}.Maybe add
... \field{commandq} and \field{eventq} virtqueues.device-types/media/description.tex line 76
Quoted 9 lines
device-types/media/description.tex new file +9
-0+\item May open a session (see Section \ref{sec:Device Operation: Command Virtqueue: Sessions}) +to use the device and send V4L2 ioctls in order to receive more information +about the device, such as supported formats or controls. +\end{enumerate} + +\subsection{Device Operation}\label{sec:Device Types / Media Device / Device Operation} + +Commands are queued on the command queue by the driver for the device to +process. The errors returned by each command are standardI would rewrite it as below but I do not have a strong opinion:
The driver enqueues commands in the command queue for the device to process them.device-types/media/description.tex line 79
Quoted 3 lines
device-types/media/description.tex new file +3
-0+\href{https://www.kernel.org/doc/html/latest/userspace-api/media/gen-errors.html}{Linux kernel error codes}. +For instance, a command that contains invalid options will return \textit{EINVAL}. +I think here is not the command that returns EINVAL but the device when trying to process a command that has an invalid option.
device-types/media/description.tex line 80
Quoted 1 line
device-types/media/description.tex new file +1
-0+Events are sent on the event queue by the device for the driver to handle.I would rewrite it as below but I do not have a strong opinion:
The device enqueues events in the event queue for the driver to process them.Quoted 31 lines
device-types/media/description.tex new file +31
-0+ +\subsubsection{Command Virtqueue} + +\paragraph{Device Operation: Command headers} + +\begin{lstlisting} +#define VIRTIO_MEDIA_CMD_OPEN 1 +#define VIRTIO_MEDIA_CMD_CLOSE 2 +#define VIRTIO_MEDIA_CMD_IOCTL 3 +#define VIRTIO_MEDIA_CMD_MMAP 4 +#define VIRTIO_MEDIA_CMD_MUNMAP 5 + +/* Header for all virtio commands from the driver to the device on the commandq. */ +struct virtio_media_cmd_header { + le32 cmd; + le32 __reserved; +}; + +/* Header for all virtio responses from the device to the driver on the commandq. */ +struct virtio_media_resp_header { + le32 status; + le32 __reserved; +}; +\end{lstlisting} + +A command consists of a command header \textit{virtio_media_cmd_header} +containing the following device-readable field: + +\begin{description} +\item[\field{cmd}] specifies a device request type (VIRTIO_MEDIA_CMD_*). +\end{description}device-types/media/description.tex line 121
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +A response consists of a response header \textit{virtio_media_resp_header} +containing the following device-writable field: + +\begin{description} +\item[\field{status}] indicates a device request status. +\end{description} + +The status field can take 0 if the command was successful, or one of the +standard Linux error codes if it was not.I would rewrite it as but I do not have a strong opinion about it:
`When the device executes the command successfully, the value of the status field is 0. When the device fails to execute the command, the value of the status field is one of the standard Linux error codes.`
Quoted 51 lines
device-types/media/description.tex new file +51
-0+ +\drivernormative{\paragraph}{Device Operation: Command Virtqueue: Sessions}{Device Types / Media Device / Device Operation / Command Virtqueue} + +Sessions are how the device is multiplexed, allowing several distinct works to +take place simultaneously. Before start operating, the driver needs to open a +session. This is equivalent to opening the \textit{/dev/videoX} file of the +V4L2 device. Each session gets a unique ID assigned, which can be then used +to perform actions on it. + +\paragraph{Device Operation: Open device} + +\textbf{VIRTIO_MEDIA_CMD_OPEN} Command for creating a new session. + +This is the equivalent of calling \textit{open} on a V4L2 device node. +The driver uses \textit{virtio_media_cmd_open} to send an open request. + +\begin{lstlisting} +struct virtio_media_cmd_open { + struct virtio_media_cmd_header hdr; +}; +\end{lstlisting} + +The device responds to \textit{VIRTIO_MEDIA_CMD_OPEN} with \textit{virtio_media_resp_open}. + +\begin{lstlisting} +struct virtio_media_resp_open { + struct virtio_media_resp_header hdr; + le32 session_id; + le32 __reserved; +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the current session, which is used for +other commands, predominantly ioctls. +\end{description} + +\devicenormative{\subparagraph}{Device Operation: Open device}{Device Types / Media Device / Device Operation / Open device} + +Upon success, the device MUST set a \field{session_id} in \textit{virtio_media_resp_open} +to an integer that is NOT used by any other open session. + +\paragraph{Device Operation: Close device} + +\textbf{VIRTIO_MEDIA_CMD_CLOSE} Command for closing an active session. + +This is the equivalent of calling \textit{close} on a previously opened V4L2 +device node. All resources associated with this session will be freed. + +This command does not require a response from the device. +device-types/media/description.tex line 182
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\begin{lstlisting} +struct virtio_media_cmd_close { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 __reserved; +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] specifies an identifier for the session to close.I would rewrite it shorter as:
session_id identifies the session to close.Quoted 77 lines
device-types/media/description.tex new file +77
-0+\end{description} + +\drivernormative{\subparagraph}{Device Operation: Close device}{Device Types / Media Device / Device Operation / Close device} + +The session ID SHALL NOT be used again after queueing this command. + +\paragraph{Device Operation: V4L2 ioctls} + +\textbf{VIRTIO_MEDIA_CMD_IOCTL} Command for executing an ioctl on an open +session. + +This command tells the device to run one of the `VIDIOC_*` ioctls on the +session identified by \textit{session_id}. + +\begin{lstlisting} +struct virtio_media_cmd_ioctl { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 code; + /* Followed by the relevant ioctl payload as defined in the macro */ +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the session to run the ioctl on. +\item[\field{code}] specifies the code of the \field{VIDIOC_*} ioctl to run. +\end{description} + +The code is extracted from the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/videodev.html}{videodev2.h}, +header file. The file defines the ioctl's codes, type of payload, and +direction. The code consists of the second argument of the \field{_IO*} macro. + +For example, the \textit{VIDIOC_G_FMT} is defined as follows: + +\begin{lstlisting} +#define VIDIOC_G_FMT _IOWR('V', 4, struct v4l2_format) +\end{lstlisting} + +This means that its ioctl code is \textit{4}, its payload is a +\textit{struct v4l2_format}, and its direction is \textit{WR} (i.e., the +payload is written by both the driver and the device). +See Section \ref{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} +for more information about the direction of ioctls. +Note that although most architectures use this format, there +are some architecture-specific encoding differences. +See \textit{include/ARCH/ioctl.h} for more details. + +The payload struct layout always matches the 64-bit, little-endian +representation of the corresponding V4L2 structure. For most structs, the +size is identical for both 32 and 64 bits versions. Otherwise, the driver +must translate them to the aforementioned size and endianess. + +The device responds to \textit{VIRTIO_MEDIA_CMD_IOCTL} with \textit{virtio_media_resp_ioctl}. + +\begin{lstlisting} +struct virtio_media_resp_ioctl { + struct virtio_media_resp_header hdr; + /* Followed by the ioctl payload as defined in the macro */ +}; +\end{lstlisting} + +\subparagraph{Ioctls payload}\label{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} + +Each ioctl has a payload, which is defined by the third argument of the +\field{_IO*} macro. + +The payload of an ioctl in the descriptor chain follows the command structure, +the response structure, or both depending on the direction: + +\begin{itemize} +\item \textbf{_IOR} is read-only for the driver, meaning the payload +follows the response in the device-writable section of the descriptor chain. +\item \textbf{_IOW} is read-only for the device, meaning the payload +follows the command in the driver-writable section of the descriptor chain. +\item \textbf{_IOWR} is writable by both the device and driver, +meaning the payload must follow both the command in the driver-writable section
Quoted 8 lines
Quoted 8 lines
device-types/media/description.tex line 267
Quoted 8 lines
device-types/media/description.tex new file +8
-0+of the descriptor chain, and the response in the device-writable section. +\end{itemize} + +A common optimization for \textit{WR} ioctls is to provide the payload using +descriptors that both point to the same buffer. This mimics the behavior of +V4L2 ioctls where the data is only passed once and used as both input and +output by the kernel. +
I think you're suggesting that two descriptors in the descriptor ring point to the same memory address. For instance, if the chain consists of only two descriptors, the first one would be read-only and the second one would be write-only, both pointing to the same memory location. According to my understanding, this behavior is allowed by the specification as long as two separate descriptors are used. However, I think it's worth noting that the specification does not explicitly define a descriptor for simultaneous reading and writing.
It may be helpful to add a note in a new patch to inform other developers about this potential optimization. Nevertheless, it's crucial for both the driver and the device to understand it when accessing the data to avoid any issues.
Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
device-types/media/description.tex new file +13
-0+\devicenormative{\subparagraph}{Device Operation: V4L2 ioctls}{Device Types / Media Device / Device Operation / V4L2 ioctls} + +In case of success of a device-writable ioctl, the device MUST always write the +payload in the device-writable part of the descriptor chain. + +In case of failure of a device-writable ioctl, the device is free to write the +payload in the device-writable part of the descriptor chain or not. Some errors +may still result in the payload being updated, and in this case the device is +expected to write the updated payload. If the device has not written the +payload after an error (i.e., only the header is returned), the driver MUST +assume that the payload has not been modified. + +\subparagraph{Handling of pointers in ioctl payload}Quoted 8 lines
device-types/media/description.tex line 286
Quoted 6 lines
device-types/media/description.tex new file +6
-0+ +A few structures used as ioctl payloads contain pointers to further +data needed for the ioctl. There are notably: + +\begin{itemize} +\item The \field{planes} pointer ofDo you mean
points to?
Quoted 5 lines
Here we specify the type ""of"" the data the pointer is pointing to. I feel this construction is correct as is, although 'points to' would mean the same, and be equally valid. I will keep it as is unless there is a strong preference.
Oh, I see. I missunderstood the sentence.
Quoted 366 lines
Quoted 167 lines
Quoted 110 lines
device-types/media/description.tex new file +108
-0+\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-buffer}{struct v4l2_buffer}, +which size is determined by the length member. +\item The \field{controls} pointer of \textit{struct v4l2_ext_controls}, which +size is determined by the count member. +\end{itemize} + +If the size of the pointed area is determined to be non-zero, then the main +payload is immediately followed by the pointed data in their order of +appearance in the structure, and the pointer value itself is ignored by the +device, which must also return the value initially passed by the driver. + +\subparagraph{Handling of pointers to userspace memory} +\label{sec:Device Types / Media Device / V4L2 ioctls / Userspace memory} + +A few pointers are special in that they point to userspace memory in the +original V4L2 specification. They are: + +\begin{itemize} +\item The \field{m.userptr} member of \textit{struct v4l2_buffer} and +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-plane}{struct v4l2_plane} +(technically an unsigned long, but designated a userspace address). +\item The \field{ptr} member of \textit{struct v4l2_ext_ctrl}. +\end{itemize} + +These pointers can cover large areas of scattered memory, which has the +potential to require more descriptors than the virtio queue can provide. For +these particular pointers only, a list of \textit{struct virtio_media_sg_entry} +that covers the needed amount of memory for the pointer is used instead of +using descriptors to map the pointed memory directly. + +\begin{lstlisting} +struct virtio_media_sg_entry { + le64 start; + le32 len; + le32 __reserved; +}; +\end{lstlisting} + +For each such pointer to read, the device reads as many SG entries as needed +to cover the length of the pointed buffer, as described by its parent +structure (\field{length} member of \textit{struct v4l2_buffer} or +\textit{struct v4l2_plane} for buffer memory, and \field{size} member of +\textit{struct v4l2_ext_control} for control data). + +Since the device never needs to modify the list of SG entries, it is only +provided by the driver in the device-readable section of the descriptor chain, +and not repeated in the device-writable section, even for WR ioctls. + +\subparagraph{Unsupported ioctls} + +A few ioctls are replaced by other, more suitable mechanisms. + +\begin{itemize} +\item \textit{VIDIOC_QUERYCAP} is replaced by reading the configuration area +(see \ref{sec:Device Types / Media Device / Device Configuration Layout}). +\item \textit{VIDIOC_DQBUF} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Dequeue buffer}). +\item \textit{VIDIOC_DQEVENT} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Emit an event}). +\item \textit{VIDIOC_G_JPEGCOMP} and \textit{VIDIOC_S_JPEGCOMP} are deprecated +and replaced by the controls of the JPEG class. +\item \textit{VIDIOC_LOG_STATUS} is a driver-only operation and shall not be +implemented by the device. +\end{itemize} + +\devicenormative{\subparagraph}{Device Operation: Unsupported ioctls}{Device Types / Media Device / Device Operation / Unsupported ioctls} + +When a request is not supported, the device MUST return \textit{ENOTTY}, +which corresponds with the response for unknown ioctls. + +\paragraph{Device Operation: Mapping a MMAP buffer} + +\textbf{VIRTIO_MEDIA_CMD_MMAP} Command for mapping a MMAP buffer into the +driver's address space. + +Shared memory region ID 0 is used to map MMAP buffers with +the \textit{VIRTIO_MEDIA_CMD_MMAP} command. + +\begin{lstlisting} +#define VIRTIO_MEDIA_MMAP_FLAG_RW (1 << 0) + +struct virtio_media_cmd_mmap { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 flags; + le32 offset; +}; +\end{lstlisting} + +\begin{description} +\item[\field{flags}] is the set of flags for the mapping. \field{VIRTIO_MEDIA_MMAP_FLAG_RW} +can be set if a read-write mapping is desired. Without this flag the mapping +will be read-only. +\item[\field{offset}] corresponds to the \field{mem_offset} field of the +\textit{union v4l2_plane} for the plane to map. This field can be obtained +using the \textit{VIDIOC_QUERYBUF} ioctl. +\end{description} + +The device responds to \textit{VIRTIO_MEDIA_CMD_MMAP} with \textit{virtio_media_resp_mmap}. + +\begin{lstlisting} +struct virtio_media_resp_mmap { + struct virtio_media_resp_header hdr; + le64 driver_addr; + le64 len; +}; +\end{lstlisting} +device-types/media/description.tex line 404
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\begin{description} +\item[\field{driver_addr}] offset into SHM region ID 0 of the start of the mapping. +\item[\field{len}] length of the mapping as indicated by the \textit{struct v4l2_plane} +the buffer belongs to. +\end{description} + +\paragraph{Device Operation: Unmapping a MMAP buffer} + +\textbf{VIRTIO_MEDIA_CMD_MUNMAP} Unmap a MMAP buffer previously mapped using \field{VIRTIO_MEDIA_CMD_MMAP}. +s/Unmap/unmaps
Quoted 44 lines
device-types/media/description.tex new file +44
-0+\begin{lstlisting} +struct virtio_media_cmd_munmap { + struct virtio_media_cmd_header hdr; + le64 driver_addr; +}; +\end{lstlisting} + +\begin{description} +\item[\field{driver_addr}] offset into SHM region ID 0 previously returned by +\textit{VIRTIO_MEDIA_CMD_MMAP} at which the buffer has been previously mapped. +\end{description} + +The device responds to \textit{VIRTIO_MEDIA_CMD_MUNMAP} with \textit{virtio_media_resp_munmap}. + +\begin{lstlisting} +struct virtio_media_resp_munmap { + struct virtio_media_resp_header hdr; +}; +\end{lstlisting} + +\devicenormative{\subparagraph}{Device Operation: Unmapping a MMAP buffer}{Device Types / Media Device / Device Operation / Unmapping a MMAP buffer} + +The device MUST keep mappings performed using \textit{VIRTIO_MEDIA_CMD_MMAP} +valid until \textit{VIRTIO_MEDIA_CMD_MUNMAP} is called, even if the buffers or +session they belong to are released or closed by the driver. + +\paragraph{Device Operation: Memory Types} + +The semantics of the three V4L2 memory types (\textit{MMAP}, \textit{USERPTR} +and \textit{DMABUF}) can easily be mapped to both driver and device context. + +However, the driver shall only advertise support for \textit{MMAP} and +\textit{DMABUF} to the guest userspace. \textit{USERPTR} is merely employed to +discern \textit{SHARED_PAGES} buffers, similar to how \textit{DMABUF} is used +to signify \textit{VIRTIO_OBJECT} buffers in the driver and device context. + +\subparagraph{MMAP} + +In virtio-media, \textit{MMAP} buffers are provisioned by the device, just like +they are by the kernel in regular V4L2. Similarly to how userspace can map a +\textit{MMAP} buffer into its address space using mmap and munmap, the +virtio-media driver can map device buffers into the driver space by queueing the +\textit{struct virtio_media_cmd_mmap} and \textit{struct virtio_media_cmd_munmap} +commands to the commandq.Quoted 8 lines
device-types/media/description.tex line 454
Quoted 6 lines
device-types/media/description.tex new file +6
-0+ +\subparagraph{SHARED_PAGES} + +In virtio-media, \textit{SHARED_PAGES} buffers are provisioned by the driver, +and use guest physical addresses. Instances of \textit{struct v4l2_buffer} +and \textit{struct v4l2_plane} of this type are followed by a list ofI think here is
these types.It refers to the memory type. I will clarify in the text.
Quoted 84 lines
Quoted 10 lines
device-types/media/description.tex new file +9
-0+\textit{struct virtio_media_sg_entry}. For more information, see +\ref{sec:Device Types / Media Device / V4L2 ioctls / Userspace memory} + +The device must not alter the pointer values provided by the driver, i.e. +\field{the m.userptr} member of \textit{struct v4l2_buffer} and +\textit{struct v4l2_plane} must be returned to the driver with the same value +as it was provided. + +\subparagraph{VIRTIO_OBJECT}device-types/media/description.tex line 473
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +In virtio-media, \textit{VIRTIO_OBJECT} buffers are provisioned by a virtio +object, just like they are by a \textit{DMABUF} in regular V4L2. Virtio objects +are 16-bytes UUIDs and do not fit in the placeholders for file descriptors, so +they follow their embedding data structure as needed and the device must +leave the V4L2 structure placeholder unchanged. + +Contrary to \textit{USERPTR} buffers, virtio objects UUIDs need to be added in +both the device-readable and device-writable section of the descriptor chain. +I think you should use
Conversely to.device-types/media/description.tex line 482
Quoted 9 lines
device-types/media/description.tex new file +9
-0+Device-allocated buffers with the \textit{V4L2_MEMORY_MMAP} memory type can also +be exported as virtio objects for use with another virtio device using the +\textit{VIDIOC_EXPBUF} ioctl. The fd placefolder of \textit{v4l2_exportbuffer} +means that space for the UUID needs to be reserved right after that structure + +\subsubsection{Event Virtqueue} + +Events are a way for the device to inform the driver about asynchronous events +that it should know about. In virtio-media, they are used as a replacement forI think you can rewrite the first sentence as (I do not have a strong opinion though):
Events are asynchronous notifications to the driver.Quoted 34 lines
device-types/media/description.tex new file +34
-0+the \textit{VIDIOC_DQBUF} and \textit{VIDIOC_DQEVENT} ioctls and the polling +mechanism, which would be impractical to implement on top of virtio. + +\paragraph{Device Operation: Event header} + +\begin{lstlisting} +#define VIRTIO_MEDIA_EVT_ERROR 0 +#define VIRTIO_MEDIA_EVT_DQBUF 1 +#define VIRTIO_MEDIA_EVT_EVENT 2 + +/* Header for events queued by the device for the driver on the eventq. */ +struct virtio_media_event_header { + le32 event; + le32 session_id; +}; +\end{lstlisting} + +\begin{description} +\item[\field{event}] one of \field{VIRTIO_MEDIA_EVT_*}. +\item[\field{session_id}] ID of the session the event applies to. +\end{description} + +\paragraph{Device Operation: Device-side error} + +\textbf{VIRTIO_MEDIA_EVT_ERROR} Upon receiving this event, the session +mentioned in the header is considered corrupted and automatically closed by +the device. + +\begin{lstlisting} +struct virtio_media_event_error { + struct virtio_media_event_header hdr; + le32 errno; + le32 __reserved; +};device-types/media/description.tex line 526
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\end{lstlisting} + +\begin{description} +\item[\field{errno}] error code describing the kind of error that occurred. +\end{description} + +\paragraph{Device Operation: Dequeue buffer} +\label{sec:Device Types / Media Device / Device Operation / Dequeue buffer} + +\textbf{VIRTIO_MEDIA_EVT_DQBUF} Signals that a buffer is not being used anymores/Signals/signals
Quoted 12 lines
device-types/media/description.tex line 533
Quoted 7 lines
device-types/media/description.tex new file +7
-0+by the device and is returned to the driver. + +A \textit{struct virtio_media_event_dqbuf} event is queued on the eventq by the +device every time a buffer previously queued using the \textit{VIDIOC_QBUF} +ioctl is done being processed and can be used by the driver again. This is like +an implicit \textit{VIDIOC_DQBUF} ioctl. +I think this paragraph could be rewritten. If I understand correctly, every time a buffer is queued using the
VIDIOC_QBUFioctl, the device queues avirtio_media_event_dqbufevent in the eventq to indicate the driver that the buffer can be used again.Yes, that is more or less the idea. Once the buffer is filled (or displayed), and
is ready to be reused, the device sends this event to the driver. I will rewrite this.Also complied with all the other changes in your comment.
Thanks for taking the time to review!
Quoted 80 lines
Quoted 79 lines
device-types/media/description.tex new file +42
-0+\begin{lstlisting} +struct virtio_media_event_dqbuf { + struct virtio_media_event_header hdr; + struct v4l2_buffer buffer; + struct v4l2_plane planes[VIDEO_MAX_PLANES]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{buffer}] \textit{struct v4l2_buffer} describing the buffer that has been dequeued. +\item[\field{planes}] array of \textit{struct v4l2_plane} containing the plane information for multi-planar buffers. +\end{description} + +Pointer values in the \textit{struct v4l2_buffer} and \textit{struct v4l2_plane} +are meaningless and must be ignored by the driver. It is recommended that the +device sets them to NULL in order to avoid leaking potential device addresses. + +Note that in the case of a \field{USERPTR} buffer, the \textit{struct v4l2_buffer} +used as event payload is not followed by the buffer memory: since that memory +is the same that the driver submitted with the \textit{VIDIOC_QBUF}, it would +be redundant to have it here. + +\paragraph{Device Operation: Emit an event} +\label{sec:Device Types / Media Device / Device Operation / Emit an event} + +\textbf{VIRTIO_MEDIA_EVT_EVENT} Signals that a V4L2 event has been emitted for a session. + +A \textit{struct virtio_media_event_event} event is queued on the eventq by the +device every time an event the driver previously subscribed to using the +\textit{VIDIOC_SUBSCRIBE_EVENT} ioctl has been signaled. This is like an +implicit \textit{VIDIOC_DQEVENT} ioctl. + +\begin{lstlisting} +struct virtio_media_event_event { + struct virtio_media_event_header hdr; + struct v4l2_event event; +}; +\end{lstlisting} + +\begin{description} +\item[\field{event}] \textit{struct v4l2_event} describing the event that occurred. +\end{description}device-types/media/device-conformance.tex new file +10
-0@@ -0,0 +1,10 @@ +\conformance{\subsection}{Media Device Conformance}\label{sec:Conformance / Device Conformance / Media Device Conformance} + +A Media device MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Open device} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / V4L2 ioctls} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Unsupported ioctls} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Unmapping a MMAP buffer} +\end{itemize}device-types/media/driver-conformance.tex new file +8
-1@@ -0,0 +1,8 @@ +\conformance{\subsection}{Media Device Conformance}\label{sec:Conformance / Driver Conformance / Media Driver Conformance} + +A Media device MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{drivernormative:Device Types / Media Device / Device Operation / Command Virtqueue} +\item \ref{drivernormative:Device Types / Media Device / Device Operation / Close device} +\end{itemize} --2.47.0
Re: [PATCH v4 1/1] virtio-media: Add virtio media device specification
Quoted 409 lines · Matias Ezequiel Vara Larsen
Quoted 390 lines · Albert Esteve
Hi Matias,
Quoted 386 lines · Matias Ezequiel Vara Larsen
Hello Albert,
I added some minor comments below. I may need to read it again since there are some points that I did not understand completely.
Quoted 138 lines · Albert Esteve
Virtio-media is an encapsulation of the V4L2 UAPI into virtio, able to virtualize any video device supported by V4L2.
Note that virtio-media does not require the use of a V4L2 device driver on the host or guest side - V4L2 is only used as a host-guest protocol, and both sides are free to convert it from/to any model that they wish to use.
Signed-off-by Albert Esteve <[email protected]>--- conformance.tex | 13 +- content.tex | 1 + device-types/media/description.tex | 575 ++++++++++++++++++++++ device-types/media/device-conformance.tex | 10 + device-types/media/driver-conformance.tex | 8 + 5 files changed, 603 insertions(+), 4 deletions(-) create mode 100644 device-types/media/description.tex create mode 100644 device-types/media/device-conformance.tex create mode 100644 device-types/media/driver-conformance.texconformance.tex +9
-4@@ -32,8 +32,10 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Driver Conformance / Memory Driver Conformance}, \ref{sec:Conformance / Driver Conformance / I2C Adapter Driver Conformance}, \ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance}, -\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance} or -\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}. +\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance}, +\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance} or +\ref{sec:Conformance / Driver Conformance / Media Driver Conformance}. + \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -59,8 +61,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Device Conformance / Memory Device Conformance}, \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance}, \ref{sec:Conformance / Device Conformance / SCMI Device Conformance}, -\ref{sec:Conformance / Device Conformance / GPIO Device Conformance} or -\ref{sec:Conformance / Device Conformance / PMEM Device Conformance}. +\ref{sec:Conformance / Device Conformance / GPIO Device Conformance}, +\ref{sec:Conformance / Device Conformance / PMEM Device Conformance} or +\ref{sec:Conformance / Device Conformance / Media Device Conformance}. \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -152,6 +155,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/driver-conformance.tex} \input{device-types/gpio/driver-conformance.tex} \input{device-types/pmem/driver-conformance.tex} +\input{device-types/media/driver-conformance.tex} \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}@@ -238,6 +242,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/device-conformance.tex} \input{device-types/gpio/device-conformance.tex} \input{device-types/pmem/device-conformance.tex} +\input{device-types/media/device-conformance.tex} \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance} A conformant implementation MUST be either transitional orcontent.tex +1
-0@@ -767,6 +767,7 @@ \chapter{Device Types}\label{sec:Device Types} \input{device-types/scmi/description.tex} \input{device-types/gpio/description.tex} \input{device-types/pmem/description.tex} +\input{device-types/media/description.tex} \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}device-types/media/description.tex new file +53
-0@@ -0,0 +1,575 @@ +\section{Media Device}\label{sec:Device Types / Media Device} + +The virtio media device follows the same model (and structures) as V4L2. It +can be used to virtualize cameras, codec devices, or any other device +supported by V4L2. The complete definition of V4L2 structures and ioctls can +be found under the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/index.html}{V4L2 UAPI documentation}. + +V4L2 is a UAPI that allows a less privileged entity (user-space) to use video +hardware exposed by a more privileged entity (the kernel). Virtio-media is an +encapsulation of this API into virtio, turning it into a virtualization API +for all classes of video devices supported by V4L2, where the device plays the +role of the kernel and the driver the role of user-space. + +The device is therefore responsible for presenting a virtual device that behaves +like an actual V4L2 device, which the driver can control. + +Note that virtio-media does not require the use of a V4L2 device driver or of +Linux on any side - V4L2 is only used as a transport protocol, +and both sides are free to convert it from/to any model that they wish to use. + +\subsection{Device ID}\label{sec:Device Types / Media Device / Device ID} + +48 + +\subsection{Virtqueues}\label{sec:Device Types / Media Device / Virtqueues} + +\begin{description} +\item[0] commandq - used for driver commands and device responses to these +commands. +\item[1] eventq - used for events sent by the device to the driver. +\end{description} + +\subsection{Feature Bits}\label{sec:Device Types / Media Device / Feature Bits} + +None + +\subsection{Device Configuration Layout}\label{sec:Device Types / Media Device / Device Configuration Layout} + +The video device configuration space uses the following layout: + +\begin{lstlisting} +struct virtio_media_config { + le32 device_caps; + le32 device_type; + le8 card[32]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{device_caps}] (driver-read-only) flags representing the device +capabilities as used in +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/vidioc-querycap.html#c.V4L.v4l2_capability}{struct v4l2_capability}.device-types/media/description.tex line 63
Quoted 10 lines
device-types/media/description.tex new file +10
-0+It corresponds with the \field{device_caps} field in the \textit{struct video_device}. +\item[\field{device_type}] (driver-read-only) informs the driver of the type +of the video device. It corresponds with the \field{vfl_devnode_type} field of the device. +\item[\field{card}] (driver-read-only) name of the device, a NUL-terminated +UTF-8 string. It corresponds with the \field{card} field of the \textit{struct v4l2_capability}. +If all the characters of the field are used, it does not need to be NUL-terminated. +\end{description} + +\subsection{Device Initialization} +Just to clarify, I would add a sentence here like:
A driver executes the following sequence to initialize a device:device-types/media/description.tex line 67
Quoted 4 lines
device-types/media/description.tex new file +4
-0+\begin{enumerate} +\item Read the \field{device_caps} and \field{device_type} fields +from the configuration layout to identify the device. +\item Set up the \field{commandq} and \field{eventq}.Maybe add
... \field{commandq} and \field{eventq} virtqueues.device-types/media/description.tex line 76
Quoted 9 lines
device-types/media/description.tex new file +9
-0+\item May open a session (see Section \ref{sec:Device Operation: Command Virtqueue: Sessions}) +to use the device and send V4L2 ioctls in order to receive more information +about the device, such as supported formats or controls. +\end{enumerate} + +\subsection{Device Operation}\label{sec:Device Types / Media Device / Device Operation} + +Commands are queued on the command queue by the driver for the device to +process. The errors returned by each command are standardI would rewrite it as below but I do not have a strong opinion:
The driver enqueues commands in the command queue for the device to process them.device-types/media/description.tex line 79
Quoted 3 lines
device-types/media/description.tex new file +3
-0+\href{https://www.kernel.org/doc/html/latest/userspace-api/media/gen-errors.html}{Linux kernel error codes}. +For instance, a command that contains invalid options will return \textit{EINVAL}. +I think here is not the command that returns EINVAL but the device when trying to process a command that has an invalid option.
device-types/media/description.tex line 80
Quoted 1 line
device-types/media/description.tex new file +1
-0+Events are sent on the event queue by the device for the driver to handle.I would rewrite it as below but I do not have a strong opinion:
The device enqueues events in the event queue for the driver to process them.Quoted 31 lines
device-types/media/description.tex new file +31
-0+ +\subsubsection{Command Virtqueue} + +\paragraph{Device Operation: Command headers} + +\begin{lstlisting} +#define VIRTIO_MEDIA_CMD_OPEN 1 +#define VIRTIO_MEDIA_CMD_CLOSE 2 +#define VIRTIO_MEDIA_CMD_IOCTL 3 +#define VIRTIO_MEDIA_CMD_MMAP 4 +#define VIRTIO_MEDIA_CMD_MUNMAP 5 + +/* Header for all virtio commands from the driver to the device on the commandq. */ +struct virtio_media_cmd_header { + le32 cmd; + le32 __reserved; +}; + +/* Header for all virtio responses from the device to the driver on the commandq. */ +struct virtio_media_resp_header { + le32 status; + le32 __reserved; +}; +\end{lstlisting} + +A command consists of a command header \textit{virtio_media_cmd_header} +containing the following device-readable field: + +\begin{description} +\item[\field{cmd}] specifies a device request type (VIRTIO_MEDIA_CMD_*). +\end{description}device-types/media/description.tex line 121
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +A response consists of a response header \textit{virtio_media_resp_header} +containing the following device-writable field: + +\begin{description} +\item[\field{status}] indicates a device request status. +\end{description} + +The status field can take 0 if the command was successful, or one of the +standard Linux error codes if it was not.I would rewrite it as but I do not have a strong opinion about it:
`When the device executes the command successfully, the value of the status field is 0. When the device fails to execute the command, the value of the status field is one of the standard Linux error codes.`
Quoted 51 lines
device-types/media/description.tex new file +51
-0+ +\drivernormative{\paragraph}{Device Operation: Command Virtqueue: Sessions}{Device Types / Media Device / Device Operation / Command Virtqueue} + +Sessions are how the device is multiplexed, allowing several distinct works to +take place simultaneously. Before start operating, the driver needs to open a +session. This is equivalent to opening the \textit{/dev/videoX} file of the +V4L2 device. Each session gets a unique ID assigned, which can be then used +to perform actions on it. + +\paragraph{Device Operation: Open device} + +\textbf{VIRTIO_MEDIA_CMD_OPEN} Command for creating a new session. + +This is the equivalent of calling \textit{open} on a V4L2 device node. +The driver uses \textit{virtio_media_cmd_open} to send an open request. + +\begin{lstlisting} +struct virtio_media_cmd_open { + struct virtio_media_cmd_header hdr; +}; +\end{lstlisting} + +The device responds to \textit{VIRTIO_MEDIA_CMD_OPEN} with \textit{virtio_media_resp_open}. + +\begin{lstlisting} +struct virtio_media_resp_open { + struct virtio_media_resp_header hdr; + le32 session_id; + le32 __reserved; +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the current session, which is used for +other commands, predominantly ioctls. +\end{description} + +\devicenormative{\subparagraph}{Device Operation: Open device}{Device Types / Media Device / Device Operation / Open device} + +Upon success, the device MUST set a \field{session_id} in \textit{virtio_media_resp_open} +to an integer that is NOT used by any other open session. + +\paragraph{Device Operation: Close device} + +\textbf{VIRTIO_MEDIA_CMD_CLOSE} Command for closing an active session. + +This is the equivalent of calling \textit{close} on a previously opened V4L2 +device node. All resources associated with this session will be freed. + +This command does not require a response from the device. +device-types/media/description.tex line 182
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\begin{lstlisting} +struct virtio_media_cmd_close { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 __reserved; +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] specifies an identifier for the session to close.I would rewrite it shorter as:
session_id identifies the session to close.Quoted 77 lines
device-types/media/description.tex new file +77
-0+\end{description} + +\drivernormative{\subparagraph}{Device Operation: Close device}{Device Types / Media Device / Device Operation / Close device} + +The session ID SHALL NOT be used again after queueing this command. + +\paragraph{Device Operation: V4L2 ioctls} + +\textbf{VIRTIO_MEDIA_CMD_IOCTL} Command for executing an ioctl on an open +session. + +This command tells the device to run one of the `VIDIOC_*` ioctls on the +session identified by \textit{session_id}. + +\begin{lstlisting} +struct virtio_media_cmd_ioctl { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 code; + /* Followed by the relevant ioctl payload as defined in the macro */ +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the session to run the ioctl on. +\item[\field{code}] specifies the code of the \field{VIDIOC_*} ioctl to run. +\end{description} + +The code is extracted from the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/videodev.html}{videodev2.h}, +header file. The file defines the ioctl's codes, type of payload, and +direction. The code consists of the second argument of the \field{_IO*} macro. + +For example, the \textit{VIDIOC_G_FMT} is defined as follows: + +\begin{lstlisting} +#define VIDIOC_G_FMT _IOWR('V', 4, struct v4l2_format) +\end{lstlisting} + +This means that its ioctl code is \textit{4}, its payload is a +\textit{struct v4l2_format}, and its direction is \textit{WR} (i.e., the +payload is written by both the driver and the device). +See Section \ref{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} +for more information about the direction of ioctls. +Note that although most architectures use this format, there +are some architecture-specific encoding differences. +See \textit{include/ARCH/ioctl.h} for more details. + +The payload struct layout always matches the 64-bit, little-endian +representation of the corresponding V4L2 structure. For most structs, the +size is identical for both 32 and 64 bits versions. Otherwise, the driver +must translate them to the aforementioned size and endianess. + +The device responds to \textit{VIRTIO_MEDIA_CMD_IOCTL} with \textit{virtio_media_resp_ioctl}. + +\begin{lstlisting} +struct virtio_media_resp_ioctl { + struct virtio_media_resp_header hdr; + /* Followed by the ioctl payload as defined in the macro */ +}; +\end{lstlisting} + +\subparagraph{Ioctls payload}\label{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} + +Each ioctl has a payload, which is defined by the third argument of the +\field{_IO*} macro. + +The payload of an ioctl in the descriptor chain follows the command structure, +the response structure, or both depending on the direction: + +\begin{itemize} +\item \textbf{_IOR} is read-only for the driver, meaning the payload +follows the response in the device-writable section of the descriptor chain. +\item \textbf{_IOW} is read-only for the device, meaning the payload +follows the command in the driver-writable section of the descriptor chain. +\item \textbf{_IOWR} is writable by both the device and driver, +meaning the payload must follow both the command in the driver-writable sectionQuoted 8 lines
Quoted 8 lines
device-types/media/description.tex line 267
Quoted 8 lines
device-types/media/description.tex new file +8
-0+of the descriptor chain, and the response in the device-writable section. +\end{itemize} + +A common optimization for \textit{WR} ioctls is to provide the payload using +descriptors that both point to the same buffer. This mimics the behavior of +V4L2 ioctls where the data is only passed once and used as both input and +output by the kernel. +I think you're suggesting that two descriptors in the descriptor ring point to the same memory address. For instance, if the chain consists of only two descriptors, the first one would be read-only and the second one would be write-only, both pointing to the same memory location. According to my understanding, this behavior is allowed by the specification as long as two separate descriptors are used. However, I think it's worth noting that the specification does not explicitly define a descriptor for simultaneous reading and writing.
Quoted 4 lines
It may be helpful to add a note in a new patch to inform other developers about this potential optimization. Nevertheless, it's crucial for both the driver and the device to understand it when accessing the data to avoid any issues.
Except please talk about buffers not descriptors.
Definitely not descriptor chains.
descriptors are an implementation detail.
Quoted 396 lines
Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
device-types/media/description.tex new file +13
-0+\devicenormative{\subparagraph}{Device Operation: V4L2 ioctls}{Device Types / Media Device / Device Operation / V4L2 ioctls} + +In case of success of a device-writable ioctl, the device MUST always write the +payload in the device-writable part of the descriptor chain. + +In case of failure of a device-writable ioctl, the device is free to write the +payload in the device-writable part of the descriptor chain or not. Some errors +may still result in the payload being updated, and in this case the device is +expected to write the updated payload. If the device has not written the +payload after an error (i.e., only the header is returned), the driver MUST +assume that the payload has not been modified. + +\subparagraph{Handling of pointers in ioctl payload}Quoted 8 lines
device-types/media/description.tex line 286
Quoted 6 lines
device-types/media/description.tex new file +6
-0+ +A few structures used as ioctl payloads contain pointers to further +data needed for the ioctl. There are notably: + +\begin{itemize} +\item The \field{planes} pointer ofDo you mean
points to?Quoted 5 lines
Here we specify the type ""of"" the data the pointer is pointing to. I feel this construction is correct as is, although 'points to' would mean the same, and be equally valid. I will keep it as is unless there is a strong preference.
Oh, I see. I missunderstood the sentence.
Quoted 366 lines
Quoted 167 lines
Quoted 110 lines
device-types/media/description.tex new file +108
-0+\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-buffer}{struct v4l2_buffer}, +which size is determined by the length member. +\item The \field{controls} pointer of \textit{struct v4l2_ext_controls}, which +size is determined by the count member. +\end{itemize} + +If the size of the pointed area is determined to be non-zero, then the main +payload is immediately followed by the pointed data in their order of +appearance in the structure, and the pointer value itself is ignored by the +device, which must also return the value initially passed by the driver. + +\subparagraph{Handling of pointers to userspace memory} +\label{sec:Device Types / Media Device / V4L2 ioctls / Userspace memory} + +A few pointers are special in that they point to userspace memory in the +original V4L2 specification. They are: + +\begin{itemize} +\item The \field{m.userptr} member of \textit{struct v4l2_buffer} and +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-plane}{struct v4l2_plane} +(technically an unsigned long, but designated a userspace address). +\item The \field{ptr} member of \textit{struct v4l2_ext_ctrl}. +\end{itemize} + +These pointers can cover large areas of scattered memory, which has the +potential to require more descriptors than the virtio queue can provide. For +these particular pointers only, a list of \textit{struct virtio_media_sg_entry} +that covers the needed amount of memory for the pointer is used instead of +using descriptors to map the pointed memory directly. + +\begin{lstlisting} +struct virtio_media_sg_entry { + le64 start; + le32 len; + le32 __reserved; +}; +\end{lstlisting} + +For each such pointer to read, the device reads as many SG entries as needed +to cover the length of the pointed buffer, as described by its parent +structure (\field{length} member of \textit{struct v4l2_buffer} or +\textit{struct v4l2_plane} for buffer memory, and \field{size} member of +\textit{struct v4l2_ext_control} for control data). + +Since the device never needs to modify the list of SG entries, it is only +provided by the driver in the device-readable section of the descriptor chain, +and not repeated in the device-writable section, even for WR ioctls. + +\subparagraph{Unsupported ioctls} + +A few ioctls are replaced by other, more suitable mechanisms. + +\begin{itemize} +\item \textit{VIDIOC_QUERYCAP} is replaced by reading the configuration area +(see \ref{sec:Device Types / Media Device / Device Configuration Layout}). +\item \textit{VIDIOC_DQBUF} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Dequeue buffer}). +\item \textit{VIDIOC_DQEVENT} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Emit an event}). +\item \textit{VIDIOC_G_JPEGCOMP} and \textit{VIDIOC_S_JPEGCOMP} are deprecated +and replaced by the controls of the JPEG class. +\item \textit{VIDIOC_LOG_STATUS} is a driver-only operation and shall not be +implemented by the device. +\end{itemize} + +\devicenormative{\subparagraph}{Device Operation: Unsupported ioctls}{Device Types / Media Device / Device Operation / Unsupported ioctls} + +When a request is not supported, the device MUST return \textit{ENOTTY}, +which corresponds with the response for unknown ioctls. + +\paragraph{Device Operation: Mapping a MMAP buffer} + +\textbf{VIRTIO_MEDIA_CMD_MMAP} Command for mapping a MMAP buffer into the +driver's address space. + +Shared memory region ID 0 is used to map MMAP buffers with +the \textit{VIRTIO_MEDIA_CMD_MMAP} command. + +\begin{lstlisting} +#define VIRTIO_MEDIA_MMAP_FLAG_RW (1 << 0) + +struct virtio_media_cmd_mmap { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 flags; + le32 offset; +}; +\end{lstlisting} + +\begin{description} +\item[\field{flags}] is the set of flags for the mapping. \field{VIRTIO_MEDIA_MMAP_FLAG_RW} +can be set if a read-write mapping is desired. Without this flag the mapping +will be read-only. +\item[\field{offset}] corresponds to the \field{mem_offset} field of the +\textit{union v4l2_plane} for the plane to map. This field can be obtained +using the \textit{VIDIOC_QUERYBUF} ioctl. +\end{description} + +The device responds to \textit{VIRTIO_MEDIA_CMD_MMAP} with \textit{virtio_media_resp_mmap}. + +\begin{lstlisting} +struct virtio_media_resp_mmap { + struct virtio_media_resp_header hdr; + le64 driver_addr; + le64 len; +}; +\end{lstlisting} +device-types/media/description.tex line 404
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\begin{description} +\item[\field{driver_addr}] offset into SHM region ID 0 of the start of the mapping. +\item[\field{len}] length of the mapping as indicated by the \textit{struct v4l2_plane} +the buffer belongs to. +\end{description} + +\paragraph{Device Operation: Unmapping a MMAP buffer} + +\textbf{VIRTIO_MEDIA_CMD_MUNMAP} Unmap a MMAP buffer previously mapped using \field{VIRTIO_MEDIA_CMD_MMAP}. +s/Unmap/unmaps
Quoted 44 lines
device-types/media/description.tex new file +44
-0+\begin{lstlisting} +struct virtio_media_cmd_munmap { + struct virtio_media_cmd_header hdr; + le64 driver_addr; +}; +\end{lstlisting} + +\begin{description} +\item[\field{driver_addr}] offset into SHM region ID 0 previously returned by +\textit{VIRTIO_MEDIA_CMD_MMAP} at which the buffer has been previously mapped. +\end{description} + +The device responds to \textit{VIRTIO_MEDIA_CMD_MUNMAP} with \textit{virtio_media_resp_munmap}. + +\begin{lstlisting} +struct virtio_media_resp_munmap { + struct virtio_media_resp_header hdr; +}; +\end{lstlisting} + +\devicenormative{\subparagraph}{Device Operation: Unmapping a MMAP buffer}{Device Types / Media Device / Device Operation / Unmapping a MMAP buffer} + +The device MUST keep mappings performed using \textit{VIRTIO_MEDIA_CMD_MMAP} +valid until \textit{VIRTIO_MEDIA_CMD_MUNMAP} is called, even if the buffers or +session they belong to are released or closed by the driver. + +\paragraph{Device Operation: Memory Types} + +The semantics of the three V4L2 memory types (\textit{MMAP}, \textit{USERPTR} +and \textit{DMABUF}) can easily be mapped to both driver and device context. + +However, the driver shall only advertise support for \textit{MMAP} and +\textit{DMABUF} to the guest userspace. \textit{USERPTR} is merely employed to +discern \textit{SHARED_PAGES} buffers, similar to how \textit{DMABUF} is used +to signify \textit{VIRTIO_OBJECT} buffers in the driver and device context. + +\subparagraph{MMAP} + +In virtio-media, \textit{MMAP} buffers are provisioned by the device, just like +they are by the kernel in regular V4L2. Similarly to how userspace can map a +\textit{MMAP} buffer into its address space using mmap and munmap, the +virtio-media driver can map device buffers into the driver space by queueing the +\textit{struct virtio_media_cmd_mmap} and \textit{struct virtio_media_cmd_munmap} +commands to the commandq.Quoted 8 lines
device-types/media/description.tex line 454
Quoted 6 lines
device-types/media/description.tex new file +6
-0+ +\subparagraph{SHARED_PAGES} + +In virtio-media, \textit{SHARED_PAGES} buffers are provisioned by the driver, +and use guest physical addresses. Instances of \textit{struct v4l2_buffer} +and \textit{struct v4l2_plane} of this type are followed by a list ofI think here is
these types.It refers to the memory type. I will clarify in the text.
Quoted 84 lines
Quoted 10 lines
device-types/media/description.tex new file +9
-0+\textit{struct virtio_media_sg_entry}. For more information, see +\ref{sec:Device Types / Media Device / V4L2 ioctls / Userspace memory} + +The device must not alter the pointer values provided by the driver, i.e. +\field{the m.userptr} member of \textit{struct v4l2_buffer} and +\textit{struct v4l2_plane} must be returned to the driver with the same value +as it was provided. + +\subparagraph{VIRTIO_OBJECT}device-types/media/description.tex line 473
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +In virtio-media, \textit{VIRTIO_OBJECT} buffers are provisioned by a virtio +object, just like they are by a \textit{DMABUF} in regular V4L2. Virtio objects +are 16-bytes UUIDs and do not fit in the placeholders for file descriptors, so +they follow their embedding data structure as needed and the device must +leave the V4L2 structure placeholder unchanged. + +Contrary to \textit{USERPTR} buffers, virtio objects UUIDs need to be added in +both the device-readable and device-writable section of the descriptor chain. +I think you should use
Conversely to.device-types/media/description.tex line 482
Quoted 9 lines
device-types/media/description.tex new file +9
-0+Device-allocated buffers with the \textit{V4L2_MEMORY_MMAP} memory type can also +be exported as virtio objects for use with another virtio device using the +\textit{VIDIOC_EXPBUF} ioctl. The fd placefolder of \textit{v4l2_exportbuffer} +means that space for the UUID needs to be reserved right after that structure + +\subsubsection{Event Virtqueue} + +Events are a way for the device to inform the driver about asynchronous events +that it should know about. In virtio-media, they are used as a replacement forI think you can rewrite the first sentence as (I do not have a strong opinion though):
Events are asynchronous notifications to the driver.Quoted 34 lines
device-types/media/description.tex new file +34
-0+the \textit{VIDIOC_DQBUF} and \textit{VIDIOC_DQEVENT} ioctls and the polling +mechanism, which would be impractical to implement on top of virtio. + +\paragraph{Device Operation: Event header} + +\begin{lstlisting} +#define VIRTIO_MEDIA_EVT_ERROR 0 +#define VIRTIO_MEDIA_EVT_DQBUF 1 +#define VIRTIO_MEDIA_EVT_EVENT 2 + +/* Header for events queued by the device for the driver on the eventq. */ +struct virtio_media_event_header { + le32 event; + le32 session_id; +}; +\end{lstlisting} + +\begin{description} +\item[\field{event}] one of \field{VIRTIO_MEDIA_EVT_*}. +\item[\field{session_id}] ID of the session the event applies to. +\end{description} + +\paragraph{Device Operation: Device-side error} + +\textbf{VIRTIO_MEDIA_EVT_ERROR} Upon receiving this event, the session +mentioned in the header is considered corrupted and automatically closed by +the device. + +\begin{lstlisting} +struct virtio_media_event_error { + struct virtio_media_event_header hdr; + le32 errno; + le32 __reserved; +};device-types/media/description.tex line 526
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\end{lstlisting} + +\begin{description} +\item[\field{errno}] error code describing the kind of error that occurred. +\end{description} + +\paragraph{Device Operation: Dequeue buffer} +\label{sec:Device Types / Media Device / Device Operation / Dequeue buffer} + +\textbf{VIRTIO_MEDIA_EVT_DQBUF} Signals that a buffer is not being used anymores/Signals/signals
Quoted 12 lines
device-types/media/description.tex line 533
Quoted 7 lines
device-types/media/description.tex new file +7
-0+by the device and is returned to the driver. + +A \textit{struct virtio_media_event_dqbuf} event is queued on the eventq by the +device every time a buffer previously queued using the \textit{VIDIOC_QBUF} +ioctl is done being processed and can be used by the driver again. This is like +an implicit \textit{VIDIOC_DQBUF} ioctl. +I think this paragraph could be rewritten. If I understand correctly, every time a buffer is queued using the
VIDIOC_QBUFioctl, the device queues avirtio_media_event_dqbufevent in the eventq to indicate the driver that the buffer can be used again.Yes, that is more or less the idea. Once the buffer is filled (or displayed), and
is ready to be reused, the device sends this event to the driver. I will rewrite this.Also complied with all the other changes in your comment.
Thanks for taking the time to review!
Quoted 80 lines
Quoted 79 lines
device-types/media/description.tex new file +42
-0+\begin{lstlisting} +struct virtio_media_event_dqbuf { + struct virtio_media_event_header hdr; + struct v4l2_buffer buffer; + struct v4l2_plane planes[VIDEO_MAX_PLANES]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{buffer}] \textit{struct v4l2_buffer} describing the buffer that has been dequeued. +\item[\field{planes}] array of \textit{struct v4l2_plane} containing the plane information for multi-planar buffers. +\end{description} + +Pointer values in the \textit{struct v4l2_buffer} and \textit{struct v4l2_plane} +are meaningless and must be ignored by the driver. It is recommended that the +device sets them to NULL in order to avoid leaking potential device addresses. + +Note that in the case of a \field{USERPTR} buffer, the \textit{struct v4l2_buffer} +used as event payload is not followed by the buffer memory: since that memory +is the same that the driver submitted with the \textit{VIDIOC_QBUF}, it would +be redundant to have it here. + +\paragraph{Device Operation: Emit an event} +\label{sec:Device Types / Media Device / Device Operation / Emit an event} + +\textbf{VIRTIO_MEDIA_EVT_EVENT} Signals that a V4L2 event has been emitted for a session. + +A \textit{struct virtio_media_event_event} event is queued on the eventq by the +device every time an event the driver previously subscribed to using the +\textit{VIDIOC_SUBSCRIBE_EVENT} ioctl has been signaled. This is like an +implicit \textit{VIDIOC_DQEVENT} ioctl. + +\begin{lstlisting} +struct virtio_media_event_event { + struct virtio_media_event_header hdr; + struct v4l2_event event; +}; +\end{lstlisting} + +\begin{description} +\item[\field{event}] \textit{struct v4l2_event} describing the event that occurred. +\end{description}device-types/media/device-conformance.tex new file +10
-0@@ -0,0 +1,10 @@ +\conformance{\subsection}{Media Device Conformance}\label{sec:Conformance / Device Conformance / Media Device Conformance} + +A Media device MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Open device} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / V4L2 ioctls} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Unsupported ioctls} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Unmapping a MMAP buffer} +\end{itemize}device-types/media/driver-conformance.tex new file +8
-1@@ -0,0 +1,8 @@ +\conformance{\subsection}{Media Device Conformance}\label{sec:Conformance / Driver Conformance / Media Driver Conformance} + +A Media device MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{drivernormative:Device Types / Media Device / Device Operation / Command Virtqueue} +\item \ref{drivernormative:Device Types / Media Device / Device Operation / Close device} +\end{itemize} --2.47.0
Re: [PATCH v4 1/1] virtio-media: Add virtio media device specification
Quoted 417 lines · Michael S. Tsirkin
Quoted 409 lines · Matias Ezequiel Vara Larsen
Quoted 390 lines · Albert Esteve
Hi Matias,
Quoted 386 lines · Matias Ezequiel Vara Larsen
Hello Albert,
I added some minor comments below. I may need to read it again since there are some points that I did not understand completely.
Quoted 138 lines · Albert Esteve
Virtio-media is an encapsulation of the V4L2 UAPI into virtio, able to virtualize any video device supported by V4L2.
Note that virtio-media does not require the use of a V4L2 device driver on the host or guest side - V4L2 is only used as a host-guest protocol, and both sides are free to convert it from/to any model that they wish to use.
Signed-off-by Albert Esteve <[email protected]>--- conformance.tex | 13 +- content.tex | 1 + device-types/media/description.tex | 575 ++++++++++++++++++++++ device-types/media/device-conformance.tex | 10 + device-types/media/driver-conformance.tex | 8 + 5 files changed, 603 insertions(+), 4 deletions(-) create mode 100644 device-types/media/description.tex create mode 100644 device-types/media/device-conformance.tex create mode 100644 device-types/media/driver-conformance.texconformance.tex +9
-4@@ -32,8 +32,10 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Driver Conformance / Memory Driver Conformance}, \ref{sec:Conformance / Driver Conformance / I2C Adapter Driver Conformance}, \ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance}, -\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance} or -\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}. +\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance}, +\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance} or +\ref{sec:Conformance / Driver Conformance / Media Driver Conformance}. + \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -59,8 +61,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Device Conformance / Memory Device Conformance}, \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance}, \ref{sec:Conformance / Device Conformance / SCMI Device Conformance}, -\ref{sec:Conformance / Device Conformance / GPIO Device Conformance} or -\ref{sec:Conformance / Device Conformance / PMEM Device Conformance}. +\ref{sec:Conformance / Device Conformance / GPIO Device Conformance}, +\ref{sec:Conformance / Device Conformance / PMEM Device Conformance} or +\ref{sec:Conformance / Device Conformance / Media Device Conformance}. \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -152,6 +155,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/driver-conformance.tex} \input{device-types/gpio/driver-conformance.tex} \input{device-types/pmem/driver-conformance.tex} +\input{device-types/media/driver-conformance.tex} \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}@@ -238,6 +242,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/device-conformance.tex} \input{device-types/gpio/device-conformance.tex} \input{device-types/pmem/device-conformance.tex} +\input{device-types/media/device-conformance.tex} \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance} A conformant implementation MUST be either transitional orcontent.tex +1
-0@@ -767,6 +767,7 @@ \chapter{Device Types}\label{sec:Device Types} \input{device-types/scmi/description.tex} \input{device-types/gpio/description.tex} \input{device-types/pmem/description.tex} +\input{device-types/media/description.tex} \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}device-types/media/description.tex new file +53
-0@@ -0,0 +1,575 @@ +\section{Media Device}\label{sec:Device Types / Media Device} + +The virtio media device follows the same model (and structures) as V4L2. It +can be used to virtualize cameras, codec devices, or any other device +supported by V4L2. The complete definition of V4L2 structures and ioctls can +be found under the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/index.html}{V4L2 UAPI documentation}. + +V4L2 is a UAPI that allows a less privileged entity (user-space) to use video +hardware exposed by a more privileged entity (the kernel). Virtio-media is an +encapsulation of this API into virtio, turning it into a virtualization API +for all classes of video devices supported by V4L2, where the device plays the +role of the kernel and the driver the role of user-space. + +The device is therefore responsible for presenting a virtual device that behaves +like an actual V4L2 device, which the driver can control. + +Note that virtio-media does not require the use of a V4L2 device driver or of +Linux on any side - V4L2 is only used as a transport protocol, +and both sides are free to convert it from/to any model that they wish to use. + +\subsection{Device ID}\label{sec:Device Types / Media Device / Device ID} + +48 + +\subsection{Virtqueues}\label{sec:Device Types / Media Device / Virtqueues} + +\begin{description} +\item[0] commandq - used for driver commands and device responses to these +commands. +\item[1] eventq - used for events sent by the device to the driver. +\end{description} + +\subsection{Feature Bits}\label{sec:Device Types / Media Device / Feature Bits} + +None + +\subsection{Device Configuration Layout}\label{sec:Device Types / Media Device / Device Configuration Layout} + +The video device configuration space uses the following layout: + +\begin{lstlisting} +struct virtio_media_config { + le32 device_caps; + le32 device_type; + le8 card[32]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{device_caps}] (driver-read-only) flags representing the device +capabilities as used in +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/vidioc-querycap.html#c.V4L.v4l2_capability}{struct v4l2_capability}.device-types/media/description.tex line 63
Quoted 10 lines
device-types/media/description.tex new file +10
-0+It corresponds with the \field{device_caps} field in the \textit{struct video_device}. +\item[\field{device_type}] (driver-read-only) informs the driver of the type +of the video device. It corresponds with the \field{vfl_devnode_type} field of the device. +\item[\field{card}] (driver-read-only) name of the device, a NUL-terminated +UTF-8 string. It corresponds with the \field{card} field of the \textit{struct v4l2_capability}. +If all the characters of the field are used, it does not need to be NUL-terminated. +\end{description} + +\subsection{Device Initialization} +Just to clarify, I would add a sentence here like:
A driver executes the following sequence to initialize a device:device-types/media/description.tex line 67
Quoted 4 lines
device-types/media/description.tex new file +4
-0+\begin{enumerate} +\item Read the \field{device_caps} and \field{device_type} fields +from the configuration layout to identify the device. +\item Set up the \field{commandq} and \field{eventq}.Maybe add
... \field{commandq} and \field{eventq} virtqueues.device-types/media/description.tex line 76
Quoted 9 lines
device-types/media/description.tex new file +9
-0+\item May open a session (see Section \ref{sec:Device Operation: Command Virtqueue: Sessions}) +to use the device and send V4L2 ioctls in order to receive more information +about the device, such as supported formats or controls. +\end{enumerate} + +\subsection{Device Operation}\label{sec:Device Types / Media Device / Device Operation} + +Commands are queued on the command queue by the driver for the device to +process. The errors returned by each command are standardI would rewrite it as below but I do not have a strong opinion:
The driver enqueues commands in the command queue for the device to process them.device-types/media/description.tex line 79
Quoted 3 lines
device-types/media/description.tex new file +3
-0+\href{https://www.kernel.org/doc/html/latest/userspace-api/media/gen-errors.html}{Linux kernel error codes}. +For instance, a command that contains invalid options will return \textit{EINVAL}. +I think here is not the command that returns EINVAL but the device when trying to process a command that has an invalid option.
device-types/media/description.tex line 80
Quoted 1 line
device-types/media/description.tex new file +1
-0+Events are sent on the event queue by the device for the driver to handle.I would rewrite it as below but I do not have a strong opinion:
The device enqueues events in the event queue for the driver to process them.Quoted 31 lines
device-types/media/description.tex new file +31
-0+ +\subsubsection{Command Virtqueue} + +\paragraph{Device Operation: Command headers} + +\begin{lstlisting} +#define VIRTIO_MEDIA_CMD_OPEN 1 +#define VIRTIO_MEDIA_CMD_CLOSE 2 +#define VIRTIO_MEDIA_CMD_IOCTL 3 +#define VIRTIO_MEDIA_CMD_MMAP 4 +#define VIRTIO_MEDIA_CMD_MUNMAP 5 + +/* Header for all virtio commands from the driver to the device on the commandq. */ +struct virtio_media_cmd_header { + le32 cmd; + le32 __reserved; +}; + +/* Header for all virtio responses from the device to the driver on the commandq. */ +struct virtio_media_resp_header { + le32 status; + le32 __reserved; +}; +\end{lstlisting} + +A command consists of a command header \textit{virtio_media_cmd_header} +containing the following device-readable field: + +\begin{description} +\item[\field{cmd}] specifies a device request type (VIRTIO_MEDIA_CMD_*). +\end{description}device-types/media/description.tex line 121
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +A response consists of a response header \textit{virtio_media_resp_header} +containing the following device-writable field: + +\begin{description} +\item[\field{status}] indicates a device request status. +\end{description} + +The status field can take 0 if the command was successful, or one of the +standard Linux error codes if it was not.I would rewrite it as but I do not have a strong opinion about it:
`When the device executes the command successfully, the value of the status field is 0. When the device fails to execute the command, the value of the status field is one of the standard Linux error codes.`
Quoted 51 lines
device-types/media/description.tex new file +51
-0+ +\drivernormative{\paragraph}{Device Operation: Command Virtqueue: Sessions}{Device Types / Media Device / Device Operation / Command Virtqueue} + +Sessions are how the device is multiplexed, allowing several distinct works to +take place simultaneously. Before start operating, the driver needs to open a +session. This is equivalent to opening the \textit{/dev/videoX} file of the +V4L2 device. Each session gets a unique ID assigned, which can be then used +to perform actions on it. + +\paragraph{Device Operation: Open device} + +\textbf{VIRTIO_MEDIA_CMD_OPEN} Command for creating a new session. + +This is the equivalent of calling \textit{open} on a V4L2 device node. +The driver uses \textit{virtio_media_cmd_open} to send an open request. + +\begin{lstlisting} +struct virtio_media_cmd_open { + struct virtio_media_cmd_header hdr; +}; +\end{lstlisting} + +The device responds to \textit{VIRTIO_MEDIA_CMD_OPEN} with \textit{virtio_media_resp_open}. + +\begin{lstlisting} +struct virtio_media_resp_open { + struct virtio_media_resp_header hdr; + le32 session_id; + le32 __reserved; +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the current session, which is used for +other commands, predominantly ioctls. +\end{description} + +\devicenormative{\subparagraph}{Device Operation: Open device}{Device Types / Media Device / Device Operation / Open device} + +Upon success, the device MUST set a \field{session_id} in \textit{virtio_media_resp_open} +to an integer that is NOT used by any other open session. + +\paragraph{Device Operation: Close device} + +\textbf{VIRTIO_MEDIA_CMD_CLOSE} Command for closing an active session. + +This is the equivalent of calling \textit{close} on a previously opened V4L2 +device node. All resources associated with this session will be freed. + +This command does not require a response from the device. +device-types/media/description.tex line 182
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\begin{lstlisting} +struct virtio_media_cmd_close { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 __reserved; +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] specifies an identifier for the session to close.I would rewrite it shorter as:
session_id identifies the session to close.Quoted 77 lines
device-types/media/description.tex new file +77
-0+\end{description} + +\drivernormative{\subparagraph}{Device Operation: Close device}{Device Types / Media Device / Device Operation / Close device} + +The session ID SHALL NOT be used again after queueing this command. + +\paragraph{Device Operation: V4L2 ioctls} + +\textbf{VIRTIO_MEDIA_CMD_IOCTL} Command for executing an ioctl on an open +session. + +This command tells the device to run one of the `VIDIOC_*` ioctls on the +session identified by \textit{session_id}. + +\begin{lstlisting} +struct virtio_media_cmd_ioctl { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 code; + /* Followed by the relevant ioctl payload as defined in the macro */ +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the session to run the ioctl on. +\item[\field{code}] specifies the code of the \field{VIDIOC_*} ioctl to run. +\end{description} + +The code is extracted from the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/videodev.html}{videodev2.h}, +header file. The file defines the ioctl's codes, type of payload, and +direction. The code consists of the second argument of the \field{_IO*} macro. + +For example, the \textit{VIDIOC_G_FMT} is defined as follows: + +\begin{lstlisting} +#define VIDIOC_G_FMT _IOWR('V', 4, struct v4l2_format) +\end{lstlisting} + +This means that its ioctl code is \textit{4}, its payload is a +\textit{struct v4l2_format}, and its direction is \textit{WR} (i.e., the +payload is written by both the driver and the device). +See Section \ref{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} +for more information about the direction of ioctls. +Note that although most architectures use this format, there +are some architecture-specific encoding differences. +See \textit{include/ARCH/ioctl.h} for more details. + +The payload struct layout always matches the 64-bit, little-endian +representation of the corresponding V4L2 structure. For most structs, the +size is identical for both 32 and 64 bits versions. Otherwise, the driver +must translate them to the aforementioned size and endianess. + +The device responds to \textit{VIRTIO_MEDIA_CMD_IOCTL} with \textit{virtio_media_resp_ioctl}. + +\begin{lstlisting} +struct virtio_media_resp_ioctl { + struct virtio_media_resp_header hdr; + /* Followed by the ioctl payload as defined in the macro */ +}; +\end{lstlisting} + +\subparagraph{Ioctls payload}\label{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} + +Each ioctl has a payload, which is defined by the third argument of the +\field{_IO*} macro. + +The payload of an ioctl in the descriptor chain follows the command structure, +the response structure, or both depending on the direction: + +\begin{itemize} +\item \textbf{_IOR} is read-only for the driver, meaning the payload +follows the response in the device-writable section of the descriptor chain. +\item \textbf{_IOW} is read-only for the device, meaning the payload +follows the command in the driver-writable section of the descriptor chain. +\item \textbf{_IOWR} is writable by both the device and driver, +meaning the payload must follow both the command in the driver-writable sectionQuoted 8 lines
Quoted 8 lines
device-types/media/description.tex line 267
Quoted 8 lines
device-types/media/description.tex new file +8
-0+of the descriptor chain, and the response in the device-writable section. +\end{itemize} + +A common optimization for \textit{WR} ioctls is to provide the payload using +descriptors that both point to the same buffer. This mimics the behavior of +V4L2 ioctls where the data is only passed once and used as both input and +output by the kernel. +I think you're suggesting that two descriptors in the descriptor ring point to the same memory address. For instance, if the chain consists of only two descriptors, the first one would be read-only and the second one would be write-only, both pointing to the same memory location. According to my understanding, this behavior is allowed by the specification as long as two separate descriptors are used. However, I think it's worth noting that the specification does not explicitly define a descriptor for simultaneous reading and writing.
Quoted 4 lines
It may be helpful to add a note in a new patch to inform other developers about this potential optimization. Nevertheless, it's crucial for both the driver and the device to understand it when accessing the data to avoid any issues.
Quoted 3 lines
Except please talk about buffers not descriptors.
Definitely not descriptor chains.
descriptors are an implementation detail.
Hi Matias and Michael,
Thanks, I will change the text in this patch to avoid using descriptors.
Also, I'll consider posting a separate patch to make that clarification in the spec.
Quoted 399 lines
Quoted 398 lines
Quoted 24 lines
Quoted 15 lines
Quoted 15 lines
device-types/media/description.tex new file +13
-0+\devicenormative{\subparagraph}{Device Operation: V4L2 ioctls}{Device Types / Media Device / Device Operation / V4L2 ioctls} + +In case of success of a device-writable ioctl, the device MUST always write the +payload in the device-writable part of the descriptor chain. + +In case of failure of a device-writable ioctl, the device is free to write the +payload in the device-writable part of the descriptor chain or not. Some errors +may still result in the payload being updated, and in this case the device is +expected to write the updated payload. If the device has not written the +payload after an error (i.e., only the header is returned), the driver MUST +assume that the payload has not been modified. + +\subparagraph{Handling of pointers in ioctl payload}Quoted 8 lines
device-types/media/description.tex line 286
Quoted 6 lines
device-types/media/description.tex new file +6
-0+ +A few structures used as ioctl payloads contain pointers to further +data needed for the ioctl. There are notably: + +\begin{itemize} +\item The \field{planes} pointer ofDo you mean
points to?Quoted 5 lines
Here we specify the type ""of"" the data the pointer is pointing to. I feel this construction is correct as is, although 'points to' would mean the same, and be equally valid. I will keep it as is unless there is a strong preference.
Oh, I see. I missunderstood the sentence.
Quoted 366 lines
Quoted 167 lines
Quoted 110 lines
device-types/media/description.tex new file +108
-0+\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-buffer}{struct v4l2_buffer}, +which size is determined by the length member. +\item The \field{controls} pointer of \textit{struct v4l2_ext_controls}, which +size is determined by the count member. +\end{itemize} + +If the size of the pointed area is determined to be non-zero, then the main +payload is immediately followed by the pointed data in their order of +appearance in the structure, and the pointer value itself is ignored by the +device, which must also return the value initially passed by the driver. + +\subparagraph{Handling of pointers to userspace memory} +\label{sec:Device Types / Media Device / V4L2 ioctls / Userspace memory} + +A few pointers are special in that they point to userspace memory in the +original V4L2 specification. They are: + +\begin{itemize} +\item The \field{m.userptr} member of \textit{struct v4l2_buffer} and +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-plane}{struct v4l2_plane} +(technically an unsigned long, but designated a userspace address). +\item The \field{ptr} member of \textit{struct v4l2_ext_ctrl}. +\end{itemize} + +These pointers can cover large areas of scattered memory, which has the +potential to require more descriptors than the virtio queue can provide. For +these particular pointers only, a list of \textit{struct virtio_media_sg_entry} +that covers the needed amount of memory for the pointer is used instead of +using descriptors to map the pointed memory directly. + +\begin{lstlisting} +struct virtio_media_sg_entry { + le64 start; + le32 len; + le32 __reserved; +}; +\end{lstlisting} + +For each such pointer to read, the device reads as many SG entries as needed +to cover the length of the pointed buffer, as described by its parent +structure (\field{length} member of \textit{struct v4l2_buffer} or +\textit{struct v4l2_plane} for buffer memory, and \field{size} member of +\textit{struct v4l2_ext_control} for control data). + +Since the device never needs to modify the list of SG entries, it is only +provided by the driver in the device-readable section of the descriptor chain, +and not repeated in the device-writable section, even for WR ioctls. + +\subparagraph{Unsupported ioctls} + +A few ioctls are replaced by other, more suitable mechanisms. + +\begin{itemize} +\item \textit{VIDIOC_QUERYCAP} is replaced by reading the configuration area +(see \ref{sec:Device Types / Media Device / Device Configuration Layout}). +\item \textit{VIDIOC_DQBUF} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Dequeue buffer}). +\item \textit{VIDIOC_DQEVENT} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Emit an event}). +\item \textit{VIDIOC_G_JPEGCOMP} and \textit{VIDIOC_S_JPEGCOMP} are deprecated +and replaced by the controls of the JPEG class. +\item \textit{VIDIOC_LOG_STATUS} is a driver-only operation and shall not be +implemented by the device. +\end{itemize} + +\devicenormative{\subparagraph}{Device Operation: Unsupported ioctls}{Device Types / Media Device / Device Operation / Unsupported ioctls} + +When a request is not supported, the device MUST return \textit{ENOTTY}, +which corresponds with the response for unknown ioctls. + +\paragraph{Device Operation: Mapping a MMAP buffer} + +\textbf{VIRTIO_MEDIA_CMD_MMAP} Command for mapping a MMAP buffer into the +driver's address space. + +Shared memory region ID 0 is used to map MMAP buffers with +the \textit{VIRTIO_MEDIA_CMD_MMAP} command. + +\begin{lstlisting} +#define VIRTIO_MEDIA_MMAP_FLAG_RW (1 << 0) + +struct virtio_media_cmd_mmap { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 flags; + le32 offset; +}; +\end{lstlisting} + +\begin{description} +\item[\field{flags}] is the set of flags for the mapping. \field{VIRTIO_MEDIA_MMAP_FLAG_RW} +can be set if a read-write mapping is desired. Without this flag the mapping +will be read-only. +\item[\field{offset}] corresponds to the \field{mem_offset} field of the +\textit{union v4l2_plane} for the plane to map. This field can be obtained +using the \textit{VIDIOC_QUERYBUF} ioctl. +\end{description} + +The device responds to \textit{VIRTIO_MEDIA_CMD_MMAP} with \textit{virtio_media_resp_mmap}. + +\begin{lstlisting} +struct virtio_media_resp_mmap { + struct virtio_media_resp_header hdr; + le64 driver_addr; + le64 len; +}; +\end{lstlisting} +device-types/media/description.tex line 404
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\begin{description} +\item[\field{driver_addr}] offset into SHM region ID 0 of the start of the mapping. +\item[\field{len}] length of the mapping as indicated by the \textit{struct v4l2_plane} +the buffer belongs to. +\end{description} + +\paragraph{Device Operation: Unmapping a MMAP buffer} + +\textbf{VIRTIO_MEDIA_CMD_MUNMAP} Unmap a MMAP buffer previously mapped using \field{VIRTIO_MEDIA_CMD_MMAP}. +s/Unmap/unmaps
Quoted 44 lines
device-types/media/description.tex new file +44
-0+\begin{lstlisting} +struct virtio_media_cmd_munmap { + struct virtio_media_cmd_header hdr; + le64 driver_addr; +}; +\end{lstlisting} + +\begin{description} +\item[\field{driver_addr}] offset into SHM region ID 0 previously returned by +\textit{VIRTIO_MEDIA_CMD_MMAP} at which the buffer has been previously mapped. +\end{description} + +The device responds to \textit{VIRTIO_MEDIA_CMD_MUNMAP} with \textit{virtio_media_resp_munmap}. + +\begin{lstlisting} +struct virtio_media_resp_munmap { + struct virtio_media_resp_header hdr; +}; +\end{lstlisting} + +\devicenormative{\subparagraph}{Device Operation: Unmapping a MMAP buffer}{Device Types / Media Device / Device Operation / Unmapping a MMAP buffer} + +The device MUST keep mappings performed using \textit{VIRTIO_MEDIA_CMD_MMAP} +valid until \textit{VIRTIO_MEDIA_CMD_MUNMAP} is called, even if the buffers or +session they belong to are released or closed by the driver. + +\paragraph{Device Operation: Memory Types} + +The semantics of the three V4L2 memory types (\textit{MMAP}, \textit{USERPTR} +and \textit{DMABUF}) can easily be mapped to both driver and device context. + +However, the driver shall only advertise support for \textit{MMAP} and +\textit{DMABUF} to the guest userspace. \textit{USERPTR} is merely employed to +discern \textit{SHARED_PAGES} buffers, similar to how \textit{DMABUF} is used +to signify \textit{VIRTIO_OBJECT} buffers in the driver and device context. + +\subparagraph{MMAP} + +In virtio-media, \textit{MMAP} buffers are provisioned by the device, just like +they are by the kernel in regular V4L2. Similarly to how userspace can map a +\textit{MMAP} buffer into its address space using mmap and munmap, the +virtio-media driver can map device buffers into the driver space by queueing the +\textit{struct virtio_media_cmd_mmap} and \textit{struct virtio_media_cmd_munmap} +commands to the commandq.Quoted 8 lines
device-types/media/description.tex line 454
Quoted 6 lines
device-types/media/description.tex new file +6
-0+ +\subparagraph{SHARED_PAGES} + +In virtio-media, \textit{SHARED_PAGES} buffers are provisioned by the driver, +and use guest physical addresses. Instances of \textit{struct v4l2_buffer} +and \textit{struct v4l2_plane} of this type are followed by a list ofI think here is
these types.It refers to the memory type. I will clarify in the text.
Quoted 84 lines
Quoted 10 lines
device-types/media/description.tex new file +9
-0+\textit{struct virtio_media_sg_entry}. For more information, see +\ref{sec:Device Types / Media Device / V4L2 ioctls / Userspace memory} + +The device must not alter the pointer values provided by the driver, i.e. +\field{the m.userptr} member of \textit{struct v4l2_buffer} and +\textit{struct v4l2_plane} must be returned to the driver with the same value +as it was provided. + +\subparagraph{VIRTIO_OBJECT}device-types/media/description.tex line 473
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +In virtio-media, \textit{VIRTIO_OBJECT} buffers are provisioned by a virtio +object, just like they are by a \textit{DMABUF} in regular V4L2. Virtio objects +are 16-bytes UUIDs and do not fit in the placeholders for file descriptors, so +they follow their embedding data structure as needed and the device must +leave the V4L2 structure placeholder unchanged. + +Contrary to \textit{USERPTR} buffers, virtio objects UUIDs need to be added in +both the device-readable and device-writable section of the descriptor chain. +I think you should use
Conversely to.device-types/media/description.tex line 482
Quoted 9 lines
device-types/media/description.tex new file +9
-0+Device-allocated buffers with the \textit{V4L2_MEMORY_MMAP} memory type can also +be exported as virtio objects for use with another virtio device using the +\textit{VIDIOC_EXPBUF} ioctl. The fd placefolder of \textit{v4l2_exportbuffer} +means that space for the UUID needs to be reserved right after that structure + +\subsubsection{Event Virtqueue} + +Events are a way for the device to inform the driver about asynchronous events +that it should know about. In virtio-media, they are used as a replacement forI think you can rewrite the first sentence as (I do not have a strong opinion though):
Events are asynchronous notifications to the driver.Quoted 34 lines
device-types/media/description.tex new file +34
-0+the \textit{VIDIOC_DQBUF} and \textit{VIDIOC_DQEVENT} ioctls and the polling +mechanism, which would be impractical to implement on top of virtio. + +\paragraph{Device Operation: Event header} + +\begin{lstlisting} +#define VIRTIO_MEDIA_EVT_ERROR 0 +#define VIRTIO_MEDIA_EVT_DQBUF 1 +#define VIRTIO_MEDIA_EVT_EVENT 2 + +/* Header for events queued by the device for the driver on the eventq. */ +struct virtio_media_event_header { + le32 event; + le32 session_id; +}; +\end{lstlisting} + +\begin{description} +\item[\field{event}] one of \field{VIRTIO_MEDIA_EVT_*}. +\item[\field{session_id}] ID of the session the event applies to. +\end{description} + +\paragraph{Device Operation: Device-side error} + +\textbf{VIRTIO_MEDIA_EVT_ERROR} Upon receiving this event, the session +mentioned in the header is considered corrupted and automatically closed by +the device. + +\begin{lstlisting} +struct virtio_media_event_error { + struct virtio_media_event_header hdr; + le32 errno; + le32 __reserved; +};device-types/media/description.tex line 526
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\end{lstlisting} + +\begin{description} +\item[\field{errno}] error code describing the kind of error that occurred. +\end{description} + +\paragraph{Device Operation: Dequeue buffer} +\label{sec:Device Types / Media Device / Device Operation / Dequeue buffer} + +\textbf{VIRTIO_MEDIA_EVT_DQBUF} Signals that a buffer is not being used anymores/Signals/signals
Quoted 12 lines
device-types/media/description.tex line 533
Quoted 7 lines
device-types/media/description.tex new file +7
-0+by the device and is returned to the driver. + +A \textit{struct virtio_media_event_dqbuf} event is queued on the eventq by the +device every time a buffer previously queued using the \textit{VIDIOC_QBUF} +ioctl is done being processed and can be used by the driver again. This is like +an implicit \textit{VIDIOC_DQBUF} ioctl. +I think this paragraph could be rewritten. If I understand correctly, every time a buffer is queued using the
VIDIOC_QBUFioctl, the device queues avirtio_media_event_dqbufevent in the eventq to indicate the driver that the buffer can be used again.Yes, that is more or less the idea. Once the buffer is filled (or displayed), and
is ready to be reused, the device sends this event to the driver. I will rewrite this.Also complied with all the other changes in your comment.
Thanks for taking the time to review!
Quoted 80 lines
Quoted 79 lines
device-types/media/description.tex new file +42
-0+\begin{lstlisting} +struct virtio_media_event_dqbuf { + struct virtio_media_event_header hdr; + struct v4l2_buffer buffer; + struct v4l2_plane planes[VIDEO_MAX_PLANES]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{buffer}] \textit{struct v4l2_buffer} describing the buffer that has been dequeued. +\item[\field{planes}] array of \textit{struct v4l2_plane} containing the plane information for multi-planar buffers. +\end{description} + +Pointer values in the \textit{struct v4l2_buffer} and \textit{struct v4l2_plane} +are meaningless and must be ignored by the driver. It is recommended that the +device sets them to NULL in order to avoid leaking potential device addresses. + +Note that in the case of a \field{USERPTR} buffer, the \textit{struct v4l2_buffer} +used as event payload is not followed by the buffer memory: since that memory +is the same that the driver submitted with the \textit{VIDIOC_QBUF}, it would +be redundant to have it here. + +\paragraph{Device Operation: Emit an event} +\label{sec:Device Types / Media Device / Device Operation / Emit an event} + +\textbf{VIRTIO_MEDIA_EVT_EVENT} Signals that a V4L2 event has been emitted for a session. + +A \textit{struct virtio_media_event_event} event is queued on the eventq by the +device every time an event the driver previously subscribed to using the +\textit{VIDIOC_SUBSCRIBE_EVENT} ioctl has been signaled. This is like an +implicit \textit{VIDIOC_DQEVENT} ioctl. + +\begin{lstlisting} +struct virtio_media_event_event { + struct virtio_media_event_header hdr; + struct v4l2_event event; +}; +\end{lstlisting} + +\begin{description} +\item[\field{event}] \textit{struct v4l2_event} describing the event that occurred. +\end{description}device-types/media/device-conformance.tex new file +10
-0@@ -0,0 +1,10 @@ +\conformance{\subsection}{Media Device Conformance}\label{sec:Conformance / Device Conformance / Media Device Conformance} + +A Media device MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Open device} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / V4L2 ioctls} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Unsupported ioctls} +\item \ref{devicenormative:Device Types / Media Device / Device Operation / Unmapping a MMAP buffer} +\end{itemize}device-types/media/driver-conformance.tex new file +8
-1@@ -0,0 +1,8 @@ +\conformance{\subsection}{Media Device Conformance}\label{sec:Conformance / Driver Conformance / Media Driver Conformance} + +A Media device MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{drivernormative:Device Types / Media Device / Device Operation / Command Virtqueue} +\item \ref{drivernormative:Device Types / Media Device / Device Operation / Close device} +\end{itemize} --2.47.0
A message that is not in the archive (sent off-list or not yet received): CAPBb6MVDgdBiqV0J4DXKX2u-KKpBs+119X7obNFsDpKjH0iarg@mail.gmail.com
Re: [PATCH v4 1/1] virtio-media: Add virtio media device specification
Hi Alexandre,
Thanks for your comments! Partially covered here, responded inline below. I'll go through the rest tomorrow.
Quoted 331 lines · Alexandre Courbot
Hi Albert,
Thanks again for driving this. The current version looks good to me, but here are a few things I found while doing another pass.
Quoted 144 lines · Albert Esteve
Virtio-media is an encapsulation of the V4L2 UAPI into virtio, able to virtualize any video device supported by V4L2.
Note that virtio-media does not require the use of a V4L2 device driver on the host or guest side - V4L2 is only used as a host-guest protocol, and both sides are free to convert it from/to any model that they wish to use.
Signed-off-by Albert Esteve <[email protected]>--- conformance.tex | 13 +- content.tex | 1 + device-types/media/description.tex | 575 ++++++++++++++++++++++ device-types/media/device-conformance.tex | 10 + device-types/media/driver-conformance.tex | 8 + 5 files changed, 603 insertions(+), 4 deletions(-) create mode 100644 device-types/media/description.tex create mode 100644 device-types/media/device-conformance.tex create mode 100644 device-types/media/driver-conformance.texconformance.tex +9
-4@@ -32,8 +32,10 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Driver Conformance / Memory Driver Conformance}, \ref{sec:Conformance / Driver Conformance / I2C Adapter Driver Conformance}, \ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance}, -\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance} or -\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}. +\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance}, +\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance} or +\ref{sec:Conformance / Driver Conformance / Media Driver Conformance}. + \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -59,8 +61,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Device Conformance / Memory Device Conformance}, \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance}, \ref{sec:Conformance / Device Conformance / SCMI Device Conformance}, -\ref{sec:Conformance / Device Conformance / GPIO Device Conformance} or -\ref{sec:Conformance / Device Conformance / PMEM Device Conformance}. +\ref{sec:Conformance / Device Conformance / GPIO Device Conformance}, +\ref{sec:Conformance / Device Conformance / PMEM Device Conformance} or +\ref{sec:Conformance / Device Conformance / Media Device Conformance}. \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -152,6 +155,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/driver-conformance.tex} \input{device-types/gpio/driver-conformance.tex} \input{device-types/pmem/driver-conformance.tex} +\input{device-types/media/driver-conformance.tex} \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}@@ -238,6 +242,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/device-conformance.tex} \input{device-types/gpio/device-conformance.tex} \input{device-types/pmem/device-conformance.tex} +\input{device-types/media/device-conformance.tex} \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance} A conformant implementation MUST be either transitional orcontent.tex +1
-0@@ -767,6 +767,7 @@ \chapter{Device Types}\label{sec:Device Types} \input{device-types/scmi/description.tex} \input{device-types/gpio/description.tex} \input{device-types/pmem/description.tex} +\input{device-types/media/description.tex} \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}device-types/media/description.tex new file +58
-0@@ -0,0 +1,575 @@ +\section{Media Device}\label{sec:Device Types / Media Device} + +The virtio media device follows the same model (and structures) as V4L2. It +can be used to virtualize cameras, codec devices, or any other device +supported by V4L2. The complete definition of V4L2 structures and ioctls can +be found under the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/index.html}{V4L2 UAPI documentation}. + +V4L2 is a UAPI that allows a less privileged entity (user-space) to use video +hardware exposed by a more privileged entity (the kernel). Virtio-media is an +encapsulation of this API into virtio, turning it into a virtualization API +for all classes of video devices supported by V4L2, where the device plays the +role of the kernel and the driver the role of user-space. + +The device is therefore responsible for presenting a virtual device that behaves +like an actual V4L2 device, which the driver can control. + +Note that virtio-media does not require the use of a V4L2 device driver or of +Linux on any side - V4L2 is only used as a transport protocol, +and both sides are free to convert it from/to any model that they wish to use. + +\subsection{Device ID}\label{sec:Device Types / Media Device / Device ID} + +48 + +\subsection{Virtqueues}\label{sec:Device Types / Media Device / Virtqueues} + +\begin{description} +\item[0] commandq - used for driver commands and device responses to these +commands. +\item[1] eventq - used for events sent by the device to the driver. +\end{description} + +\subsection{Feature Bits}\label{sec:Device Types / Media Device / Feature Bits} + +None + +\subsection{Device Configuration Layout}\label{sec:Device Types / Media Device / Device Configuration Layout} + +The video device configuration space uses the following layout: + +\begin{lstlisting} +struct virtio_media_config { + le32 device_caps; + le32 device_type; + le8 card[32]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{device_caps}] (driver-read-only) flags representing the device +capabilities as used in +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/vidioc-querycap.html#c.V4L.v4l2_capability}{struct v4l2_capability}. +It corresponds with the \field{device_caps} field in the \textit{struct video_device}. +\item[\field{device_type}] (driver-read-only) informs the driver of the type +of the video device. It corresponds with the \field{vfl_devnode_type} field of the device. +\item[\field{card}] (driver-read-only) name of the device, a NUL-terminated +UTF-8 string. It corresponds with the \field{card} field of the \textit{struct v4l2_capability}.device-types/media/description.tex line 68
Quoted 10 lines
device-types/media/description.tex new file +10
-0+If all the characters of the field are used, it does not need to be NUL-terminated. +\end{description} + +\subsection{Device Initialization} + +\begin{enumerate} +\item Read the \field{device_caps} and \field{device_type} fields +from the configuration layout to identify the device. +\item Set up the \field{commandq} and \field{eventq}. +\item May open a session (see Section \ref{sec:Device Operation: Command Virtqueue: Sessions})This reference does not resolve.
Quoted 48 lines
device-types/media/description.tex new file +48
-0+to use the device and send V4L2 ioctls in order to receive more information +about the device, such as supported formats or controls. +\end{enumerate} + +\subsection{Device Operation}\label{sec:Device Types / Media Device / Device Operation} + +Commands are queued on the command queue by the driver for the device to +process. The errors returned by each command are standard +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/gen-errors.html}{Linux kernel error codes}. +For instance, a command that contains invalid options will return \textit{EINVAL}. + +Events are sent on the event queue by the device for the driver to handle. + +\subsubsection{Command Virtqueue} + +\paragraph{Device Operation: Command headers} + +\begin{lstlisting} +#define VIRTIO_MEDIA_CMD_OPEN 1 +#define VIRTIO_MEDIA_CMD_CLOSE 2 +#define VIRTIO_MEDIA_CMD_IOCTL 3 +#define VIRTIO_MEDIA_CMD_MMAP 4 +#define VIRTIO_MEDIA_CMD_MUNMAP 5 + +/* Header for all virtio commands from the driver to the device on the commandq. */ +struct virtio_media_cmd_header { + le32 cmd; + le32 __reserved; +}; + +/* Header for all virtio responses from the device to the driver on the commandq. */ +struct virtio_media_resp_header { + le32 status; + le32 __reserved; +}; +\end{lstlisting} + +A command consists of a command header \textit{virtio_media_cmd_header} +containing the following device-readable field: + +\begin{description} +\item[\field{cmd}] specifies a device request type (VIRTIO_MEDIA_CMD_*). +\end{description} + +A response consists of a response header \textit{virtio_media_resp_header} +containing the following device-writable field: + +\begin{description}device-types/media/description.tex line 126
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\item[\field{status}] indicates a device request status. +\end{description} + +The status field can take 0 if the command was successful, or one of the +standard Linux error codes if it was not. + +\drivernormative{\paragraph}{Device Operation: Command Virtqueue: Sessions}{Device Types / Media Device / Device Operation / Command Virtqueue} + +Sessions are how the device is multiplexed, allowing several distinct works to +take place simultaneously. Before start operating, the driver needs to open anit: "Before starting operation" maybe?
Quoted 51 lines
device-types/media/description.tex new file +51
-0+session. This is equivalent to opening the \textit{/dev/videoX} file of the +V4L2 device. Each session gets a unique ID assigned, which can be then used +to perform actions on it. + +\paragraph{Device Operation: Open device} + +\textbf{VIRTIO_MEDIA_CMD_OPEN} Command for creating a new session. + +This is the equivalent of calling \textit{open} on a V4L2 device node. +The driver uses \textit{virtio_media_cmd_open} to send an open request. + +\begin{lstlisting} +struct virtio_media_cmd_open { + struct virtio_media_cmd_header hdr; +}; +\end{lstlisting} + +The device responds to \textit{VIRTIO_MEDIA_CMD_OPEN} with \textit{virtio_media_resp_open}. + +\begin{lstlisting} +struct virtio_media_resp_open { + struct virtio_media_resp_header hdr; + le32 session_id; + le32 __reserved; +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the current session, which is used for +other commands, predominantly ioctls. +\end{description} + +\devicenormative{\subparagraph}{Device Operation: Open device}{Device Types / Media Device / Device Operation / Open device} + +Upon success, the device MUST set a \field{session_id} in \textit{virtio_media_resp_open} +to an integer that is NOT used by any other open session. + +\paragraph{Device Operation: Close device} + +\textbf{VIRTIO_MEDIA_CMD_CLOSE} Command for closing an active session. + +This is the equivalent of calling \textit{close} on a previously opened V4L2 +device node. All resources associated with this session will be freed. + +This command does not require a response from the device. + +\begin{lstlisting} +struct virtio_media_cmd_close { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 __reserved;device-types/media/description.tex line 187
Quoted 10 lines
device-types/media/description.tex new file +10
-0+}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] specifies an identifier for the session to close. +\end{description} + +\drivernormative{\subparagraph}{Device Operation: Close device}{Device Types / Media Device / Device Operation / Close device} + +The session ID SHALL NOT be used again after queueing this command.... until it has been obtained again through a subsequent
VIRTIO_MEDIA_CMD_OPENcall. (session IDs can be recycled ; this should go without saying but it doesn't hurt to mention it)Quoted 5 lines
device-types/media/description.tex new file +5
-0+ +\paragraph{Device Operation: V4L2 ioctls} + +\textbf{VIRTIO_MEDIA_CMD_IOCTL} Command for executing an ioctl on an open +session.device-types/media/description.tex line 202
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +This command tells the device to run one of the `VIDIOC_*` ioctls on the +session identified by \textit{session_id}. + +\begin{lstlisting} +struct virtio_media_cmd_ioctl { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 code; + /* Followed by the relevant ioctl payload as defined in the macro */"ioctl command payload" maybe.
Quoted 22 lines
device-types/media/description.tex new file +22
-0+}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the session to run the ioctl on. +\item[\field{code}] specifies the code of the \field{VIDIOC_*} ioctl to run. +\end{description} + +The code is extracted from the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/videodev.html}{videodev2.h}, +header file. The file defines the ioctl's codes, type of payload, and +direction. The code consists of the second argument of the \field{_IO*} macro. + +For example, the \textit{VIDIOC_G_FMT} is defined as follows: + +\begin{lstlisting} +#define VIDIOC_G_FMT _IOWR('V', 4, struct v4l2_format) +\end{lstlisting} + +This means that its ioctl code is \textit{4}, its payload is a +\textit{struct v4l2_format}, and its direction is \textit{WR} (i.e., the +payload is written by both the driver and the device).
Quoted 8 lines
device-types/media/description.tex line 229
Quoted 5 lines
device-types/media/description.tex new file +5
-0+See Section \ref{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} +for more information about the direction of ioctls. +Note that although most architectures use this format, there +are some architecture-specific encoding differences. +See \textit{include/ARCH/ioctl.h} for more details.Is the last sentence needed?
videodev2.his not architecture-specific, so I don't think this adds anything for our purpose?
Probably not. The idea was to highlight that in some cases the payload may vary depending on the underlying architecture for some ioctls, due to these architecture differences.
But I agree that it is a detail that should not concern to those wanting to implement the specs. So I will remove the lines.
Quoted 88 lines
device-types/media/description.tex line 241
Quoted 13 lines
device-types/media/description.tex new file +12
-0+ +The payload struct layout always matches the 64-bit, little-endian +representation of the corresponding V4L2 structure. For most structs, the +size is identical for both 32 and 64 bits versions. Otherwise, the driver +must translate them to the aforementioned size and endianess. + +The device responds to \textit{VIRTIO_MEDIA_CMD_IOCTL} with \textit{virtio_media_resp_ioctl}. + +\begin{lstlisting} +struct virtio_media_resp_ioctl { + struct virtio_media_resp_header hdr; + /* Followed by the ioctl payload as defined in the macro */Similarly, "ioctl response payload"?
Quoted 12 lines
device-types/media/description.tex new file +12
-0+}; +\end{lstlisting} + +\subparagraph{Ioctls payload}\label{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} + +Each ioctl has a payload, which is defined by the third argument of the +\field{_IO*} macro. + +The payload of an ioctl in the descriptor chain follows the command structure, +the response structure, or both depending on the direction: + +\begin{itemize}device-types/media/description.tex line 263
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\item \textbf{_IOR} is read-only for the driver, meaning the payload +follows the response in the device-writable section of the descriptor chain. +\item \textbf{_IOW} is read-only for the device, meaning the payload +follows the command in the driver-writable section of the descriptor chain. +\item \textbf{_IOWR} is writable by both the device and driver, +meaning the payload must follow both the command in the driver-writable section +of the descriptor chain, and the response in the device-writable section. +\end{itemize} + +A common optimization for \textit{WR} ioctls is to provide the payload usings/common/possible? Unless this is indeed commonly used by other devices, but I doubt it.
device-types/media/description.tex line 266
Quoted 3 lines
device-types/media/description.tex new file +3
-0+descriptors that both point to the same buffer. This mimics the behavior of +V4L2 ioctls where the data is only passed once and used as both input and +output by the kernel.That's actually something I'd like to confirm - is it ok to have the device-readable and device-writable descriptors point to the same memory? I don't see why it wouldn't, but in case this is a problem we may want to change this and update the driver accordingly.
device-types/media/description.tex line 278
Quoted 12 lines
device-types/media/description.tex new file +12
-0+ +\devicenormative{\subparagraph}{Device Operation: V4L2 ioctls}{Device Types / Media Device / Device Operation / V4L2 ioctls} + +In case of success of a device-writable ioctl, the device MUST always write the +payload in the device-writable part of the descriptor chain. + +In case of failure of a device-writable ioctl, the device is free to write the +payload in the device-writable part of the descriptor chain or not. Some errors +may still result in the payload being updated, and in this case the device is +expected to write the updated payload. If the device has not written the +payload after an error (i.e., only the header is returned), the driver MUST +assume that the payload has not been modified.This last sentence sounds like it is driver-normative, not device-normative.
device-types/media/description.tex line 289
Quoted 11 lines
device-types/media/description.tex new file +11
-0+ +\subparagraph{Handling of pointers in ioctl payload} + +A few structures used as ioctl payloads contain pointers to further +data needed for the ioctl. There are notably: + +\begin{itemize} +\item The \field{planes} pointer of +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-buffer}{struct v4l2_buffer}, +which size is determined by the length member. +\item The \field{controls} pointer of \textit{struct v4l2_ext_controls}, whichSince
struct v4l2_bufferabove is a link, do we want to one forstruct v4l2_ext_controlsas well?device-types/media/description.tex line 293
Quoted 4 lines
device-types/media/description.tex new file +4
-0+size is determined by the count member. +\end{itemize} + +If the size of the pointed area is determined to be non-zero, then the mainnit: "If the size of the pointed area is non-zero, ..."
Quoted 7 lines
device-types/media/description.tex line 298
Quoted 5 lines
device-types/media/description.tex new file +5
-0+payload is immediately followed by the pointed data in their order of +appearance in the structure, and the pointer value itself is ignored by the +device, which must also return the value initially passed by the driver. + +\subparagraph{Handling of pointers to userspace memory}s/userspace/guest ?
To me this refers to the fact that the original fields in the related structures refer to userspace memory (outside of the host/guest context). I think I'd keep it as is.
Maybe changing the titles of this and the previous sections to:
\subparagraph{Handling of pointers to data in ioctl payload} and
\subparagraph{Handling of pointers to userspace memory in ioctl payload}
wdyt?
Quoted 4 lines
I'd also suggest adding a "(
SHARED_PAGESmemory type)" at the end of the title to make it explicit that this is specific to this particular memory type.
Maybe a clarification in the first paragraph (and a link to the related section) would be enough? I feel having a memory type in the title when this concept has not been yet introduced may be confusing. Having that in the text allows to link the section and jump ahead to relate both concepts.
Quoted 16 lines
device-types/media/description.tex line 309
Quoted 12 lines
device-types/media/description.tex new file +11
-0+\label{sec:Device Types / Media Device / V4L2 ioctls / Userspace memory} + +A few pointers are special in that they point to userspace memory in the +original V4L2 specification. They are: + +\begin{itemize} +\item The \field{m.userptr} member of \textit{struct v4l2_buffer} and +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-plane}{struct v4l2_plane} +(technically an unsigned long, but designated a userspace address). +\item The \field{ptr} member of \textit{struct v4l2_ext_ctrl}. +\end{itemize}Same as above, let's either add links to all V4L2 structs, or just remove them altogether since we point to the documentation in the introduction anyway.
I added the link for your comment above, but I think with that change, this is ok as it is. I only add the link to those structs that have not been linked before.
Quoted 93 lines
Quoted 37 lines
device-types/media/description.tex new file +36
-0+ +These pointers can cover large areas of scattered memory, which has the +potential to require more descriptors than the virtio queue can provide. For +these particular pointers only, a list of \textit{struct virtio_media_sg_entry} +that covers the needed amount of memory for the pointer is used instead of +using descriptors to map the pointed memory directly. + +\begin{lstlisting} +struct virtio_media_sg_entry { + le64 start; + le32 len; + le32 __reserved; +}; +\end{lstlisting} + +For each such pointer to read, the device reads as many SG entries as needed +to cover the length of the pointed buffer, as described by its parent +structure (\field{length} member of \textit{struct v4l2_buffer} or +\textit{struct v4l2_plane} for buffer memory, and \field{size} member of +\textit{struct v4l2_ext_control} for control data). + +Since the device never needs to modify the list of SG entries, it is only +provided by the driver in the device-readable section of the descriptor chain, +and not repeated in the device-writable section, even for WR ioctls. + +\subparagraph{Unsupported ioctls} + +A few ioctls are replaced by other, more suitable mechanisms. + +\begin{itemize} +\item \textit{VIDIOC_QUERYCAP} is replaced by reading the configuration area +(see \ref{sec:Device Types / Media Device / Device Configuration Layout}). +\item \textit{VIDIOC_DQBUF} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Dequeue buffer}). +\item \textit{VIDIOC_DQEVENT} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Emit an event}).device-types/media/description.tex line 355
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\item \textit{VIDIOC_G_JPEGCOMP} and \textit{VIDIOC_S_JPEGCOMP} are deprecated +and replaced by the controls of the JPEG class. +\item \textit{VIDIOC_LOG_STATUS} is a driver-only operation and shall not be +implemented by the device. +\end{itemize} + +\devicenormative{\subparagraph}{Device Operation: Unsupported ioctls}{Device Types / Media Device / Device Operation / Unsupported ioctls} + +When a request is not supported, the device MUST return \textit{ENOTTY}, +which corresponds with the response for unknown ioctls.s/with/to
Quoted 5 lines
device-types/media/description.tex new file +5
-0+ +\paragraph{Device Operation: Mapping a MMAP buffer} + +\textbf{VIRTIO_MEDIA_CMD_MMAP} Command for mapping a MMAP buffer into the +driver's address space.device-types/media/description.tex line 370
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +Shared memory region ID 0 is used to map MMAP buffers with +the \textit{VIRTIO_MEDIA_CMD_MMAP} command. + +\begin{lstlisting} +#define VIRTIO_MEDIA_MMAP_FLAG_RW (1 << 0) + +struct virtio_media_cmd_mmap { + struct virtio_media_cmd_header hdr; + le32 session_id;
session_idis not mentioned in the member description below.Quoted 25 lines
device-types/media/description.tex new file +25
-0+ le32 flags; + le32 offset; +}; +\end{lstlisting} + +\begin{description} +\item[\field{flags}] is the set of flags for the mapping. \field{VIRTIO_MEDIA_MMAP_FLAG_RW} +can be set if a read-write mapping is desired. Without this flag the mapping +will be read-only. +\item[\field{offset}] corresponds to the \field{mem_offset} field of the +\textit{union v4l2_plane} for the plane to map. This field can be obtained +using the \textit{VIDIOC_QUERYBUF} ioctl. +\end{description} + +The device responds to \textit{VIRTIO_MEDIA_CMD_MMAP} with \textit{virtio_media_resp_mmap}. + +\begin{lstlisting} +struct virtio_media_resp_mmap { + struct virtio_media_resp_header hdr; + le64 driver_addr; + le64 len; +}; +\end{lstlisting} + +\begin{description}
Quoted 8 lines
device-types/media/description.tex line 398
Quoted 3 lines
device-types/media/description.tex new file +3
-0+\item[\field{driver_addr}] offset into SHM region ID 0 of the start of the mapping. +\item[\field{len}] length of the mapping as indicated by the \textit{struct v4l2_plane} +the buffer belongs to.This makes me think: do we need to return the length of the mapping when we already know the length of the
v4l2_planeand we only map whole buffers? It probably doesn't hurt to repeat that information here but maybe that's something we can optimize out?
That's a good point. Indeed, no partial mapping should be allowed, so the length field does sound a bit redundant. Unless it needs error checking, which probably does not, I think it can be skipped. I'll do for the next version. Thanks!
Quoted 39 lines
Quoted 39 lines
device-types/media/description.tex new file +38
-0+\end{description} + +\paragraph{Device Operation: Unmapping a MMAP buffer} + +\textbf{VIRTIO_MEDIA_CMD_MUNMAP} Unmap a MMAP buffer previously mapped using \field{VIRTIO_MEDIA_CMD_MMAP}. + +\begin{lstlisting} +struct virtio_media_cmd_munmap { + struct virtio_media_cmd_header hdr; + le64 driver_addr; +}; +\end{lstlisting} + +\begin{description} +\item[\field{driver_addr}] offset into SHM region ID 0 previously returned by +\textit{VIRTIO_MEDIA_CMD_MMAP} at which the buffer has been previously mapped. +\end{description} + +The device responds to \textit{VIRTIO_MEDIA_CMD_MUNMAP} with \textit{virtio_media_resp_munmap}. + +\begin{lstlisting} +struct virtio_media_resp_munmap { + struct virtio_media_resp_header hdr; +}; +\end{lstlisting} + +\devicenormative{\subparagraph}{Device Operation: Unmapping a MMAP buffer}{Device Types / Media Device / Device Operation / Unmapping a MMAP buffer} + +The device MUST keep mappings performed using \textit{VIRTIO_MEDIA_CMD_MMAP} +valid until \textit{VIRTIO_MEDIA_CMD_MUNMAP} is called, even if the buffers or +session they belong to are released or closed by the driver. + +\paragraph{Device Operation: Memory Types} + +The semantics of the three V4L2 memory types (\textit{MMAP}, \textit{USERPTR} +and \textit{DMABUF}) can easily be mapped to both driver and device context. + +However, the driver shall only advertise support for \textit{MMAP} and
Quoted 8 lines
device-types/media/description.tex line 439
Quoted 3 lines
device-types/media/description.tex new file +3
-0+\textit{DMABUF} to the guest userspace. \textit{USERPTR} is merely employed to +discern \textit{SHARED_PAGES} buffers, similar to how \textit{DMABUF} is used +to signify \textit{VIRTIO_OBJECT} buffers in the driver and device context.This paragraph sounds specific to the Linux guest driver, as upstream signaled they prefer if drivers don't expose USERPTR to userspace. But it's probably not related for non-V4L2 guest drivers, so I'd suggest we remove it.
Uhm, good point. I think I am biased and failed to consider that. I agree in that the spec should be kept generic and avoid guest-specific details like this. I'll remove the paragraph!
Quoted 53 lines
We probably also want to define the values, I am using this in the driver:
enum
virtio_media_memory{VIRTIO_MEDIA_MMAP=V4L2_MEMORY_MMAP,VIRTIO_MEDIA_SHARED_PAGES=V4L2_MEMORY_USERPTR,VIRTIO_MEDIA_OBJECT=V4L2_MEMORY_DMABUF,
};device-types/media/description.tex line 450
Quoted 11 lines
device-types/media/description.tex new file +11
-0+ +\subparagraph{MMAP} + +In virtio-media, \textit{MMAP} buffers are provisioned by the device, just like +they are by the kernel in regular V4L2. Similarly to how userspace can map a +\textit{MMAP} buffer into its address space using mmap and munmap, the +virtio-media driver can map device buffers into the driver space by queueing the +\textit{struct virtio_media_cmd_mmap} and \textit{struct virtio_media_cmd_munmap} +commands to the commandq. + +\subparagraph{SHARED_PAGES}Maybe add something to say this is the semantic equivalent of
V4L2_MEMORY_USERPTR.device-types/media/description.tex line 463
Quoted 13 lines
device-types/media/description.tex new file +13
-0+ +In virtio-media, \textit{SHARED_PAGES} buffers are provisioned by the driver, +and use guest physical addresses. Instances of \textit{struct v4l2_buffer} +and \textit{struct v4l2_plane} of this type are followed by a list of +\textit{struct virtio_media_sg_entry}. For more information, see +\ref{sec:Device Types / Media Device / V4L2 ioctls / Userspace memory} + +The device must not alter the pointer values provided by the driver, i.e. +\field{the m.userptr} member of \textit{struct v4l2_buffer} and +\textit{struct v4l2_plane} must be returned to the driver with the same value +as it was provided. + +\subparagraph{VIRTIO_OBJECT}Similarly we probably want to say this is the semantic equivalent of
V4L2_MEMORY_DMABUF.device-types/media/description.tex line 472
Quoted 9 lines
device-types/media/description.tex new file +9
-0+ +In virtio-media, \textit{VIRTIO_OBJECT} buffers are provisioned by a virtio +object, just like they are by a \textit{DMABUF} in regular V4L2. Virtio objects +are 16-bytes UUIDs and do not fit in the placeholders for file descriptors, so +they follow their embedding data structure as needed and the device must +leave the V4L2 structure placeholder unchanged. + +Contrary to \textit{USERPTR} buffers, virtio objects UUIDs need to be added in +both the device-readable and device-writable section of the descriptor chain.s/USERPTR/SHARED_PAGES.
Quoted 3 lines
I'm also not quite sure why we concluded the objects UUIDs need to be on both the device-readable and device-writable sections. Do you happen to remember?
I took this directly from the virtio-media project at
https://github.com/chromeos/virtio-media/blob/main/README.md?plain=1#L444
I do not think I got involved in any discussion regarding this detail? Maybe it has to do with the fact that mmaped buffers can be exported by the device, and uses the device-writable section to communicate the new ID back to the driver? Just a guess though.
Quoted 123 lines
device-types/media/description.tex line 474
Quoted 3 lines
device-types/media/description.tex new file +2
-0+ +Device-allocated buffers with the \textit{V4L2_MEMORY_MMAP} memory type can alsos/V4L2_MEMORY_MMAP/VIRTIO_MEDIA_MMAP
device-types/media/description.tex line 477
Quoted 3 lines
device-types/media/description.tex new file +3
-0+be exported as virtio objects for use with another virtio device using the +\textit{VIDIOC_EXPBUF} ioctl. The fd placefolder of \textit{v4l2_exportbuffer} +means that space for the UUID needs to be reserved right after that structurenit: missing "." at the end of the sentence.
Quoted 34 lines
device-types/media/description.tex new file +34
-0+ +\subsubsection{Event Virtqueue} + +Events are a way for the device to inform the driver about asynchronous events +that it should know about. In virtio-media, they are used as a replacement for +the \textit{VIDIOC_DQBUF} and \textit{VIDIOC_DQEVENT} ioctls and the polling +mechanism, which would be impractical to implement on top of virtio. + +\paragraph{Device Operation: Event header} + +\begin{lstlisting} +#define VIRTIO_MEDIA_EVT_ERROR 0 +#define VIRTIO_MEDIA_EVT_DQBUF 1 +#define VIRTIO_MEDIA_EVT_EVENT 2 + +/* Header for events queued by the device for the driver on the eventq. */ +struct virtio_media_event_header { + le32 event; + le32 session_id; +}; +\end{lstlisting} + +\begin{description} +\item[\field{event}] one of \field{VIRTIO_MEDIA_EVT_*}. +\item[\field{session_id}] ID of the session the event applies to. +\end{description} + +\paragraph{Device Operation: Device-side error} + +\textbf{VIRTIO_MEDIA_EVT_ERROR} Upon receiving this event, the session +mentioned in the header is considered corrupted and automatically closed by +the device. + +\begin{lstlisting}device-types/media/description.tex line 521
Quoted 10 lines
device-types/media/description.tex new file +10
-0+struct virtio_media_event_error { + struct virtio_media_event_header hdr; + le32 errno; + le32 __reserved; +}; +\end{lstlisting} + +\begin{description} +\item[\field{errno}] error code describing the kind of error that occurred. +\end{description}I think there is a risk for a race condition if the session ID is recycled by the device and the driver opens a new session before acknowledging the error event. We probably need to make sure that the device acknowledges that the session is invalid before that happens. I'd suggest adding the following as a device-normative section:
"While the session is considered non-existing by the device after the error is signaled, the device MUST NOT recycle the session ID until the driver has explicitly closed it."
And a driver-normative section:
"Upon receiving an error event for a session, the driver MUST explicitly close the session using a
VIRTIO_MEDIA_CMD_CLOSEcommand."Quoted 7 lines
device-types/media/description.tex new file +7
-0+ +\paragraph{Device Operation: Dequeue buffer} +\label{sec:Device Types / Media Device / Device Operation / Dequeue buffer} + +\textbf{VIRTIO_MEDIA_EVT_DQBUF} Signals that a buffer is not being used anymore +by the device and is returned to the driver. +device-types/media/description.tex line 538
Quoted 10 lines
device-types/media/description.tex new file +10
-0+A \textit{struct virtio_media_event_dqbuf} event is queued on the eventq by the +device every time a buffer previously queued using the \textit{VIDIOC_QBUF} +ioctl is done being processed and can be used by the driver again. This is like +an implicit \textit{VIDIOC_DQBUF} ioctl. + +\begin{lstlisting} +struct virtio_media_event_dqbuf { + struct virtio_media_event_header hdr; + struct v4l2_buffer buffer; + struct v4l2_plane planes[VIDEO_MAX_PLANES];Maybe use
VIRTIO_MEDIA_MAX_PLANESand define it as#define
VIRTIO_MEDIA_MAX_PLANESVIDEO_MAX_PLANESor
#define
VIRTIO_MEDIA_MAX_PLANES8if we don't want to depend on videodev2.h's definitions here.
device-types/media/description.tex line 551
Quoted 13 lines
device-types/media/description.tex new file +13
-0+}; +\end{lstlisting} + +\begin{description} +\item[\field{buffer}] \textit{struct v4l2_buffer} describing the buffer that has been dequeued. +\item[\field{planes}] array of \textit{struct v4l2_plane} containing the plane information for multi-planar buffers. +\end{description} + +Pointer values in the \textit{struct v4l2_buffer} and \textit{struct v4l2_plane} +are meaningless and must be ignored by the driver. It is recommended that the +device sets them to NULL in order to avoid leaking potential device addresses. + +Note that in the case of a \field{USERPTR} buffer, the \textit{struct v4l2_buffer}s/USERPTR/SHARED_PAGES.
device-types/media/description.tex line 552
Quoted 1 line
device-types/media/description.tex new file +1
-0+used as event payload is not followed by the buffer memory:"... not followed by the buffer's SG entries" is probably more precise.
Cheers,
Alex.
Re: [PATCH v4 1/1] virtio-media: Add virtio media device specification
Quoted 5 lines · Albert Esteve
Hi Alexandre,
Thanks for your comments! Partially covered here, responded inline below. I'll go through the rest tomorrow.
Hi,
I have already checked and agreed on the rest. I will soon post a new version of the spec. Waiting for some inputs for now.
BR,
Albert.
Quoted 833 lines · Alexandre Courbot
Quoted 331 lines
Hi Albert,
Thanks again for driving this. The current version looks good to me, but here are a few things I found while doing another pass.
Quoted 144 lines · Albert Esteve
Virtio-media is an encapsulation of the V4L2 UAPI into virtio, able to virtualize any video device supported by V4L2.
Note that virtio-media does not require the use of a V4L2 device driver on the host or guest side - V4L2 is only used as a host-guest protocol, and both sides are free to convert it from/to any model that they wish to use.
Signed-off-by Albert Esteve <[email protected]>--- conformance.tex | 13 +- content.tex | 1 + device-types/media/description.tex | 575 ++++++++++++++++++++++ device-types/media/device-conformance.tex | 10 + device-types/media/driver-conformance.tex | 8 + 5 files changed, 603 insertions(+), 4 deletions(-) create mode 100644 device-types/media/description.tex create mode 100644 device-types/media/device-conformance.tex create mode 100644 device-types/media/driver-conformance.texconformance.tex +9
-4@@ -32,8 +32,10 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Driver Conformance / Memory Driver Conformance}, \ref{sec:Conformance / Driver Conformance / I2C Adapter Driver Conformance}, \ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance}, -\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance} or -\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}. +\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance}, +\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance} or +\ref{sec:Conformance / Driver Conformance / Media Driver Conformance}. + \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -59,8 +61,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Device Conformance / Memory Device Conformance}, \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance}, \ref{sec:Conformance / Device Conformance / SCMI Device Conformance}, -\ref{sec:Conformance / Device Conformance / GPIO Device Conformance} or -\ref{sec:Conformance / Device Conformance / PMEM Device Conformance}. +\ref{sec:Conformance / Device Conformance / GPIO Device Conformance}, +\ref{sec:Conformance / Device Conformance / PMEM Device Conformance} or +\ref{sec:Conformance / Device Conformance / Media Device Conformance}. \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -152,6 +155,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/driver-conformance.tex} \input{device-types/gpio/driver-conformance.tex} \input{device-types/pmem/driver-conformance.tex} +\input{device-types/media/driver-conformance.tex} \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}@@ -238,6 +242,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/device-conformance.tex} \input{device-types/gpio/device-conformance.tex} \input{device-types/pmem/device-conformance.tex} +\input{device-types/media/device-conformance.tex} \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance} A conformant implementation MUST be either transitional orcontent.tex +1
-0@@ -767,6 +767,7 @@ \chapter{Device Types}\label{sec:Device Types} \input{device-types/scmi/description.tex} \input{device-types/gpio/description.tex} \input{device-types/pmem/description.tex} +\input{device-types/media/description.tex} \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}device-types/media/description.tex new file +58
-0@@ -0,0 +1,575 @@ +\section{Media Device}\label{sec:Device Types / Media Device} + +The virtio media device follows the same model (and structures) as V4L2. It +can be used to virtualize cameras, codec devices, or any other device +supported by V4L2. The complete definition of V4L2 structures and ioctls can +be found under the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/index.html}{V4L2 UAPI documentation}. + +V4L2 is a UAPI that allows a less privileged entity (user-space) to use video +hardware exposed by a more privileged entity (the kernel). Virtio-media is an +encapsulation of this API into virtio, turning it into a virtualization API +for all classes of video devices supported by V4L2, where the device plays the +role of the kernel and the driver the role of user-space. + +The device is therefore responsible for presenting a virtual device that behaves +like an actual V4L2 device, which the driver can control. + +Note that virtio-media does not require the use of a V4L2 device driver or of +Linux on any side - V4L2 is only used as a transport protocol, +and both sides are free to convert it from/to any model that they wish to use. + +\subsection{Device ID}\label{sec:Device Types / Media Device / Device ID} + +48 + +\subsection{Virtqueues}\label{sec:Device Types / Media Device / Virtqueues} + +\begin{description} +\item[0] commandq - used for driver commands and device responses to these +commands. +\item[1] eventq - used for events sent by the device to the driver. +\end{description} + +\subsection{Feature Bits}\label{sec:Device Types / Media Device / Feature Bits} + +None + +\subsection{Device Configuration Layout}\label{sec:Device Types / Media Device / Device Configuration Layout} + +The video device configuration space uses the following layout: + +\begin{lstlisting} +struct virtio_media_config { + le32 device_caps; + le32 device_type; + le8 card[32]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{device_caps}] (driver-read-only) flags representing the device +capabilities as used in +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/vidioc-querycap.html#c.V4L.v4l2_capability}{struct v4l2_capability}. +It corresponds with the \field{device_caps} field in the \textit{struct video_device}. +\item[\field{device_type}] (driver-read-only) informs the driver of the type +of the video device. It corresponds with the \field{vfl_devnode_type} field of the device. +\item[\field{card}] (driver-read-only) name of the device, a NUL-terminated +UTF-8 string. It corresponds with the \field{card} field of the \textit{struct v4l2_capability}.device-types/media/description.tex line 68
Quoted 10 lines
device-types/media/description.tex new file +10
-0+If all the characters of the field are used, it does not need to be NUL-terminated. +\end{description} + +\subsection{Device Initialization} + +\begin{enumerate} +\item Read the \field{device_caps} and \field{device_type} fields +from the configuration layout to identify the device. +\item Set up the \field{commandq} and \field{eventq}. +\item May open a session (see Section \ref{sec:Device Operation: Command Virtqueue: Sessions})This reference does not resolve.
Quoted 48 lines
device-types/media/description.tex new file +48
-0+to use the device and send V4L2 ioctls in order to receive more information +about the device, such as supported formats or controls. +\end{enumerate} + +\subsection{Device Operation}\label{sec:Device Types / Media Device / Device Operation} + +Commands are queued on the command queue by the driver for the device to +process. The errors returned by each command are standard +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/gen-errors.html}{Linux kernel error codes}. +For instance, a command that contains invalid options will return \textit{EINVAL}. + +Events are sent on the event queue by the device for the driver to handle. + +\subsubsection{Command Virtqueue} + +\paragraph{Device Operation: Command headers} + +\begin{lstlisting} +#define VIRTIO_MEDIA_CMD_OPEN 1 +#define VIRTIO_MEDIA_CMD_CLOSE 2 +#define VIRTIO_MEDIA_CMD_IOCTL 3 +#define VIRTIO_MEDIA_CMD_MMAP 4 +#define VIRTIO_MEDIA_CMD_MUNMAP 5 + +/* Header for all virtio commands from the driver to the device on the commandq. */ +struct virtio_media_cmd_header { + le32 cmd; + le32 __reserved; +}; + +/* Header for all virtio responses from the device to the driver on the commandq. */ +struct virtio_media_resp_header { + le32 status; + le32 __reserved; +}; +\end{lstlisting} + +A command consists of a command header \textit{virtio_media_cmd_header} +containing the following device-readable field: + +\begin{description} +\item[\field{cmd}] specifies a device request type (VIRTIO_MEDIA_CMD_*). +\end{description} + +A response consists of a response header \textit{virtio_media_resp_header} +containing the following device-writable field: + +\begin{description}device-types/media/description.tex line 126
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\item[\field{status}] indicates a device request status. +\end{description} + +The status field can take 0 if the command was successful, or one of the +standard Linux error codes if it was not. + +\drivernormative{\paragraph}{Device Operation: Command Virtqueue: Sessions}{Device Types / Media Device / Device Operation / Command Virtqueue} + +Sessions are how the device is multiplexed, allowing several distinct works to +take place simultaneously. Before start operating, the driver needs to open anit: "Before starting operation" maybe?
Quoted 51 lines
device-types/media/description.tex new file +51
-0+session. This is equivalent to opening the \textit{/dev/videoX} file of the +V4L2 device. Each session gets a unique ID assigned, which can be then used +to perform actions on it. + +\paragraph{Device Operation: Open device} + +\textbf{VIRTIO_MEDIA_CMD_OPEN} Command for creating a new session. + +This is the equivalent of calling \textit{open} on a V4L2 device node. +The driver uses \textit{virtio_media_cmd_open} to send an open request. + +\begin{lstlisting} +struct virtio_media_cmd_open { + struct virtio_media_cmd_header hdr; +}; +\end{lstlisting} + +The device responds to \textit{VIRTIO_MEDIA_CMD_OPEN} with \textit{virtio_media_resp_open}. + +\begin{lstlisting} +struct virtio_media_resp_open { + struct virtio_media_resp_header hdr; + le32 session_id; + le32 __reserved; +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the current session, which is used for +other commands, predominantly ioctls. +\end{description} + +\devicenormative{\subparagraph}{Device Operation: Open device}{Device Types / Media Device / Device Operation / Open device} + +Upon success, the device MUST set a \field{session_id} in \textit{virtio_media_resp_open} +to an integer that is NOT used by any other open session. + +\paragraph{Device Operation: Close device} + +\textbf{VIRTIO_MEDIA_CMD_CLOSE} Command for closing an active session. + +This is the equivalent of calling \textit{close} on a previously opened V4L2 +device node. All resources associated with this session will be freed. + +This command does not require a response from the device. + +\begin{lstlisting} +struct virtio_media_cmd_close { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 __reserved;device-types/media/description.tex line 187
Quoted 10 lines
device-types/media/description.tex new file +10
-0+}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] specifies an identifier for the session to close. +\end{description} + +\drivernormative{\subparagraph}{Device Operation: Close device}{Device Types / Media Device / Device Operation / Close device} + +The session ID SHALL NOT be used again after queueing this command.... until it has been obtained again through a subsequent
VIRTIO_MEDIA_CMD_OPENcall. (session IDs can be recycled ; this should go without saying but it doesn't hurt to mention it)Quoted 5 lines
device-types/media/description.tex new file +5
-0+ +\paragraph{Device Operation: V4L2 ioctls} + +\textbf{VIRTIO_MEDIA_CMD_IOCTL} Command for executing an ioctl on an open +session.device-types/media/description.tex line 202
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +This command tells the device to run one of the `VIDIOC_*` ioctls on the +session identified by \textit{session_id}. + +\begin{lstlisting} +struct virtio_media_cmd_ioctl { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 code; + /* Followed by the relevant ioctl payload as defined in the macro */"ioctl command payload" maybe.
Quoted 22 lines
device-types/media/description.tex new file +22
-0+}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the session to run the ioctl on. +\item[\field{code}] specifies the code of the \field{VIDIOC_*} ioctl to run. +\end{description} + +The code is extracted from the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/videodev.html}{videodev2.h}, +header file. The file defines the ioctl's codes, type of payload, and +direction. The code consists of the second argument of the \field{_IO*} macro. + +For example, the \textit{VIDIOC_G_FMT} is defined as follows: + +\begin{lstlisting} +#define VIDIOC_G_FMT _IOWR('V', 4, struct v4l2_format) +\end{lstlisting} + +This means that its ioctl code is \textit{4}, its payload is a +\textit{struct v4l2_format}, and its direction is \textit{WR} (i.e., the +payload is written by both the driver and the device).Quoted 8 lines
device-types/media/description.tex line 229
Quoted 5 lines
device-types/media/description.tex new file +5
-0+See Section \ref{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} +for more information about the direction of ioctls. +Note that although most architectures use this format, there +are some architecture-specific encoding differences. +See \textit{include/ARCH/ioctl.h} for more details.Is the last sentence needed?
videodev2.his not architecture-specific, so I don't think this adds anything for our purpose?Probably not. The idea was to highlight that in some cases the payload may vary depending on the underlying architecture for some ioctls, due to these architecture differences.
But I agree that it is a detail that should not concern to those wanting to implement the specs. So I will remove the lines.Quoted 88 lines
device-types/media/description.tex line 241
Quoted 13 lines
device-types/media/description.tex new file +12
-0+ +The payload struct layout always matches the 64-bit, little-endian +representation of the corresponding V4L2 structure. For most structs, the +size is identical for both 32 and 64 bits versions. Otherwise, the driver +must translate them to the aforementioned size and endianess. + +The device responds to \textit{VIRTIO_MEDIA_CMD_IOCTL} with \textit{virtio_media_resp_ioctl}. + +\begin{lstlisting} +struct virtio_media_resp_ioctl { + struct virtio_media_resp_header hdr; + /* Followed by the ioctl payload as defined in the macro */Similarly, "ioctl response payload"?
Quoted 12 lines
device-types/media/description.tex new file +12
-0+}; +\end{lstlisting} + +\subparagraph{Ioctls payload}\label{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} + +Each ioctl has a payload, which is defined by the third argument of the +\field{_IO*} macro. + +The payload of an ioctl in the descriptor chain follows the command structure, +the response structure, or both depending on the direction: + +\begin{itemize}device-types/media/description.tex line 263
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\item \textbf{_IOR} is read-only for the driver, meaning the payload +follows the response in the device-writable section of the descriptor chain. +\item \textbf{_IOW} is read-only for the device, meaning the payload +follows the command in the driver-writable section of the descriptor chain. +\item \textbf{_IOWR} is writable by both the device and driver, +meaning the payload must follow both the command in the driver-writable section +of the descriptor chain, and the response in the device-writable section. +\end{itemize} + +A common optimization for \textit{WR} ioctls is to provide the payload usings/common/possible? Unless this is indeed commonly used by other devices, but I doubt it.
device-types/media/description.tex line 266
Quoted 3 lines
device-types/media/description.tex new file +3
-0+descriptors that both point to the same buffer. This mimics the behavior of +V4L2 ioctls where the data is only passed once and used as both input and +output by the kernel.That's actually something I'd like to confirm - is it ok to have the device-readable and device-writable descriptors point to the same memory? I don't see why it wouldn't, but in case this is a problem we may want to change this and update the driver accordingly.
device-types/media/description.tex line 278
Quoted 12 lines
device-types/media/description.tex new file +12
-0+ +\devicenormative{\subparagraph}{Device Operation: V4L2 ioctls}{Device Types / Media Device / Device Operation / V4L2 ioctls} + +In case of success of a device-writable ioctl, the device MUST always write the +payload in the device-writable part of the descriptor chain. + +In case of failure of a device-writable ioctl, the device is free to write the +payload in the device-writable part of the descriptor chain or not. Some errors +may still result in the payload being updated, and in this case the device is +expected to write the updated payload. If the device has not written the +payload after an error (i.e., only the header is returned), the driver MUST +assume that the payload has not been modified.This last sentence sounds like it is driver-normative, not device-normative.
device-types/media/description.tex line 289
Quoted 11 lines
device-types/media/description.tex new file +11
-0+ +\subparagraph{Handling of pointers in ioctl payload} + +A few structures used as ioctl payloads contain pointers to further +data needed for the ioctl. There are notably: + +\begin{itemize} +\item The \field{planes} pointer of +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-buffer}{struct v4l2_buffer}, +which size is determined by the length member. +\item The \field{controls} pointer of \textit{struct v4l2_ext_controls}, whichSince
struct v4l2_bufferabove is a link, do we want to one forstruct v4l2_ext_controlsas well?device-types/media/description.tex line 293
Quoted 4 lines
device-types/media/description.tex new file +4
-0+size is determined by the count member. +\end{itemize} + +If the size of the pointed area is determined to be non-zero, then the mainnit: "If the size of the pointed area is non-zero, ..."
Quoted 7 lines
device-types/media/description.tex line 298
Quoted 5 lines
device-types/media/description.tex new file +5
-0+payload is immediately followed by the pointed data in their order of +appearance in the structure, and the pointer value itself is ignored by the +device, which must also return the value initially passed by the driver. + +\subparagraph{Handling of pointers to userspace memory}s/userspace/guest ?
To me this refers to the fact that the original fields in the related structures refer to userspace memory (outside of the host/guest context). I think I'd keep it as is.
Maybe changing the titles of this and the previous sections to:
\subparagraph{Handling of pointers to data in ioctl payload} and
\subparagraph{Handling of pointers to userspace memory in ioctl payload}wdyt?
Quoted 4 lines
I'd also suggest adding a "(
SHARED_PAGESmemory type)" at the end of the title to make it explicit that this is specific to this particular memory type.Maybe a clarification in the first paragraph (and a link to the related section) would be enough? I feel having a memory type in the title when this concept has not been yet introduced may be confusing. Having that in the text allows to link the section and jump ahead to relate both concepts.
Quoted 16 lines
device-types/media/description.tex line 309
Quoted 12 lines
device-types/media/description.tex new file +11
-0+\label{sec:Device Types / Media Device / V4L2 ioctls / Userspace memory} + +A few pointers are special in that they point to userspace memory in the +original V4L2 specification. They are: + +\begin{itemize} +\item The \field{m.userptr} member of \textit{struct v4l2_buffer} and +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-plane}{struct v4l2_plane} +(technically an unsigned long, but designated a userspace address). +\item The \field{ptr} member of \textit{struct v4l2_ext_ctrl}. +\end{itemize}Same as above, let's either add links to all V4L2 structs, or just remove them altogether since we point to the documentation in the introduction anyway.
I added the link for your comment above, but I think with that change, this is ok as it is. I only add the link to those structs that have not been linked before.
Quoted 93 lines
Quoted 37 lines
device-types/media/description.tex new file +36
-0+ +These pointers can cover large areas of scattered memory, which has the +potential to require more descriptors than the virtio queue can provide. For +these particular pointers only, a list of \textit{struct virtio_media_sg_entry} +that covers the needed amount of memory for the pointer is used instead of +using descriptors to map the pointed memory directly. + +\begin{lstlisting} +struct virtio_media_sg_entry { + le64 start; + le32 len; + le32 __reserved; +}; +\end{lstlisting} + +For each such pointer to read, the device reads as many SG entries as needed +to cover the length of the pointed buffer, as described by its parent +structure (\field{length} member of \textit{struct v4l2_buffer} or +\textit{struct v4l2_plane} for buffer memory, and \field{size} member of +\textit{struct v4l2_ext_control} for control data). + +Since the device never needs to modify the list of SG entries, it is only +provided by the driver in the device-readable section of the descriptor chain, +and not repeated in the device-writable section, even for WR ioctls. + +\subparagraph{Unsupported ioctls} + +A few ioctls are replaced by other, more suitable mechanisms. + +\begin{itemize} +\item \textit{VIDIOC_QUERYCAP} is replaced by reading the configuration area +(see \ref{sec:Device Types / Media Device / Device Configuration Layout}). +\item \textit{VIDIOC_DQBUF} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Dequeue buffer}). +\item \textit{VIDIOC_DQEVENT} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Emit an event}).device-types/media/description.tex line 355
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\item \textit{VIDIOC_G_JPEGCOMP} and \textit{VIDIOC_S_JPEGCOMP} are deprecated +and replaced by the controls of the JPEG class. +\item \textit{VIDIOC_LOG_STATUS} is a driver-only operation and shall not be +implemented by the device. +\end{itemize} + +\devicenormative{\subparagraph}{Device Operation: Unsupported ioctls}{Device Types / Media Device / Device Operation / Unsupported ioctls} + +When a request is not supported, the device MUST return \textit{ENOTTY}, +which corresponds with the response for unknown ioctls.s/with/to
Quoted 5 lines
device-types/media/description.tex new file +5
-0+ +\paragraph{Device Operation: Mapping a MMAP buffer} + +\textbf{VIRTIO_MEDIA_CMD_MMAP} Command for mapping a MMAP buffer into the +driver's address space.device-types/media/description.tex line 370
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +Shared memory region ID 0 is used to map MMAP buffers with +the \textit{VIRTIO_MEDIA_CMD_MMAP} command. + +\begin{lstlisting} +#define VIRTIO_MEDIA_MMAP_FLAG_RW (1 << 0) + +struct virtio_media_cmd_mmap { + struct virtio_media_cmd_header hdr; + le32 session_id;
session_idis not mentioned in the member description below.Quoted 25 lines
device-types/media/description.tex new file +25
-0+ le32 flags; + le32 offset; +}; +\end{lstlisting} + +\begin{description} +\item[\field{flags}] is the set of flags for the mapping. \field{VIRTIO_MEDIA_MMAP_FLAG_RW} +can be set if a read-write mapping is desired. Without this flag the mapping +will be read-only. +\item[\field{offset}] corresponds to the \field{mem_offset} field of the +\textit{union v4l2_plane} for the plane to map. This field can be obtained +using the \textit{VIDIOC_QUERYBUF} ioctl. +\end{description} + +The device responds to \textit{VIRTIO_MEDIA_CMD_MMAP} with \textit{virtio_media_resp_mmap}. + +\begin{lstlisting} +struct virtio_media_resp_mmap { + struct virtio_media_resp_header hdr; + le64 driver_addr; + le64 len; +}; +\end{lstlisting} + +\begin{description}Quoted 8 lines
device-types/media/description.tex line 398
Quoted 3 lines
device-types/media/description.tex new file +3
-0+\item[\field{driver_addr}] offset into SHM region ID 0 of the start of the mapping. +\item[\field{len}] length of the mapping as indicated by the \textit{struct v4l2_plane} +the buffer belongs to.This makes me think: do we need to return the length of the mapping when we already know the length of the
v4l2_planeand we only map whole buffers? It probably doesn't hurt to repeat that information here but maybe that's something we can optimize out?That's a good point. Indeed, no partial mapping should be allowed, so the length field does sound a bit redundant. Unless it needs error checking, which probably does not, I think it can be skipped. I'll do for the next version. Thanks!
Quoted 39 lines
Quoted 39 lines
device-types/media/description.tex new file +38
-0+\end{description} + +\paragraph{Device Operation: Unmapping a MMAP buffer} + +\textbf{VIRTIO_MEDIA_CMD_MUNMAP} Unmap a MMAP buffer previously mapped using \field{VIRTIO_MEDIA_CMD_MMAP}. + +\begin{lstlisting} +struct virtio_media_cmd_munmap { + struct virtio_media_cmd_header hdr; + le64 driver_addr; +}; +\end{lstlisting} + +\begin{description} +\item[\field{driver_addr}] offset into SHM region ID 0 previously returned by +\textit{VIRTIO_MEDIA_CMD_MMAP} at which the buffer has been previously mapped. +\end{description} + +The device responds to \textit{VIRTIO_MEDIA_CMD_MUNMAP} with \textit{virtio_media_resp_munmap}. + +\begin{lstlisting} +struct virtio_media_resp_munmap { + struct virtio_media_resp_header hdr; +}; +\end{lstlisting} + +\devicenormative{\subparagraph}{Device Operation: Unmapping a MMAP buffer}{Device Types / Media Device / Device Operation / Unmapping a MMAP buffer} + +The device MUST keep mappings performed using \textit{VIRTIO_MEDIA_CMD_MMAP} +valid until \textit{VIRTIO_MEDIA_CMD_MUNMAP} is called, even if the buffers or +session they belong to are released or closed by the driver. + +\paragraph{Device Operation: Memory Types} + +The semantics of the three V4L2 memory types (\textit{MMAP}, \textit{USERPTR} +and \textit{DMABUF}) can easily be mapped to both driver and device context. + +However, the driver shall only advertise support for \textit{MMAP} andQuoted 8 lines
device-types/media/description.tex line 439
Quoted 3 lines
device-types/media/description.tex new file +3
-0+\textit{DMABUF} to the guest userspace. \textit{USERPTR} is merely employed to +discern \textit{SHARED_PAGES} buffers, similar to how \textit{DMABUF} is used +to signify \textit{VIRTIO_OBJECT} buffers in the driver and device context.This paragraph sounds specific to the Linux guest driver, as upstream signaled they prefer if drivers don't expose USERPTR to userspace. But it's probably not related for non-V4L2 guest drivers, so I'd suggest we remove it.
Uhm, good point. I think I am biased and failed to consider that. I agree in that the spec should be kept generic and avoid guest-specific details like this. I'll remove the paragraph!
Quoted 53 lines
We probably also want to define the values, I am using this in the driver:
enum
virtio_media_memory{VIRTIO_MEDIA_MMAP=V4L2_MEMORY_MMAP,VIRTIO_MEDIA_SHARED_PAGES=V4L2_MEMORY_USERPTR,VIRTIO_MEDIA_OBJECT=V4L2_MEMORY_DMABUF,
};device-types/media/description.tex line 450
Quoted 11 lines
device-types/media/description.tex new file +11
-0+ +\subparagraph{MMAP} + +In virtio-media, \textit{MMAP} buffers are provisioned by the device, just like +they are by the kernel in regular V4L2. Similarly to how userspace can map a +\textit{MMAP} buffer into its address space using mmap and munmap, the +virtio-media driver can map device buffers into the driver space by queueing the +\textit{struct virtio_media_cmd_mmap} and \textit{struct virtio_media_cmd_munmap} +commands to the commandq. + +\subparagraph{SHARED_PAGES}Maybe add something to say this is the semantic equivalent of
V4L2_MEMORY_USERPTR.device-types/media/description.tex line 463
Quoted 13 lines
device-types/media/description.tex new file +13
-0+ +In virtio-media, \textit{SHARED_PAGES} buffers are provisioned by the driver, +and use guest physical addresses. Instances of \textit{struct v4l2_buffer} +and \textit{struct v4l2_plane} of this type are followed by a list of +\textit{struct virtio_media_sg_entry}. For more information, see +\ref{sec:Device Types / Media Device / V4L2 ioctls / Userspace memory} + +The device must not alter the pointer values provided by the driver, i.e. +\field{the m.userptr} member of \textit{struct v4l2_buffer} and +\textit{struct v4l2_plane} must be returned to the driver with the same value +as it was provided. + +\subparagraph{VIRTIO_OBJECT}Similarly we probably want to say this is the semantic equivalent of
V4L2_MEMORY_DMABUF.device-types/media/description.tex line 472
Quoted 9 lines
device-types/media/description.tex new file +9
-0+ +In virtio-media, \textit{VIRTIO_OBJECT} buffers are provisioned by a virtio +object, just like they are by a \textit{DMABUF} in regular V4L2. Virtio objects +are 16-bytes UUIDs and do not fit in the placeholders for file descriptors, so +they follow their embedding data structure as needed and the device must +leave the V4L2 structure placeholder unchanged. + +Contrary to \textit{USERPTR} buffers, virtio objects UUIDs need to be added in +both the device-readable and device-writable section of the descriptor chain.s/USERPTR/SHARED_PAGES.
Quoted 3 lines
I'm also not quite sure why we concluded the objects UUIDs need to be on both the device-readable and device-writable sections. Do you happen to remember?
I took this directly from the virtio-media project at
https://github.com/chromeos/virtio-media/blob/main/README.md?plain=1#L444I do not think I got involved in any discussion regarding this detail? Maybe it has to do with the fact that mmaped buffers can be exported by the device, and uses the device-writable section to communicate the new ID back to the driver? Just a guess though.
Quoted 123 lines
device-types/media/description.tex line 474
Quoted 3 lines
device-types/media/description.tex new file +2
-0+ +Device-allocated buffers with the \textit{V4L2_MEMORY_MMAP} memory type can alsos/V4L2_MEMORY_MMAP/VIRTIO_MEDIA_MMAP
device-types/media/description.tex line 477
Quoted 3 lines
device-types/media/description.tex new file +3
-0+be exported as virtio objects for use with another virtio device using the +\textit{VIDIOC_EXPBUF} ioctl. The fd placefolder of \textit{v4l2_exportbuffer} +means that space for the UUID needs to be reserved right after that structurenit: missing "." at the end of the sentence.
Quoted 34 lines
device-types/media/description.tex new file +34
-0+ +\subsubsection{Event Virtqueue} + +Events are a way for the device to inform the driver about asynchronous events +that it should know about. In virtio-media, they are used as a replacement for +the \textit{VIDIOC_DQBUF} and \textit{VIDIOC_DQEVENT} ioctls and the polling +mechanism, which would be impractical to implement on top of virtio. + +\paragraph{Device Operation: Event header} + +\begin{lstlisting} +#define VIRTIO_MEDIA_EVT_ERROR 0 +#define VIRTIO_MEDIA_EVT_DQBUF 1 +#define VIRTIO_MEDIA_EVT_EVENT 2 + +/* Header for events queued by the device for the driver on the eventq. */ +struct virtio_media_event_header { + le32 event; + le32 session_id; +}; +\end{lstlisting} + +\begin{description} +\item[\field{event}] one of \field{VIRTIO_MEDIA_EVT_*}. +\item[\field{session_id}] ID of the session the event applies to. +\end{description} + +\paragraph{Device Operation: Device-side error} + +\textbf{VIRTIO_MEDIA_EVT_ERROR} Upon receiving this event, the session +mentioned in the header is considered corrupted and automatically closed by +the device. + +\begin{lstlisting}device-types/media/description.tex line 521
Quoted 10 lines
device-types/media/description.tex new file +10
-0+struct virtio_media_event_error { + struct virtio_media_event_header hdr; + le32 errno; + le32 __reserved; +}; +\end{lstlisting} + +\begin{description} +\item[\field{errno}] error code describing the kind of error that occurred. +\end{description}I think there is a risk for a race condition if the session ID is recycled by the device and the driver opens a new session before acknowledging the error event. We probably need to make sure that the device acknowledges that the session is invalid before that happens. I'd suggest adding the following as a device-normative section:
"While the session is considered non-existing by the device after the error is signaled, the device MUST NOT recycle the session ID until the driver has explicitly closed it."
And a driver-normative section:
"Upon receiving an error event for a session, the driver MUST explicitly close the session using a
VIRTIO_MEDIA_CMD_CLOSEcommand."Quoted 7 lines
device-types/media/description.tex new file +7
-0+ +\paragraph{Device Operation: Dequeue buffer} +\label{sec:Device Types / Media Device / Device Operation / Dequeue buffer} + +\textbf{VIRTIO_MEDIA_EVT_DQBUF} Signals that a buffer is not being used anymore +by the device and is returned to the driver. +device-types/media/description.tex line 538
Quoted 10 lines
device-types/media/description.tex new file +10
-0+A \textit{struct virtio_media_event_dqbuf} event is queued on the eventq by the +device every time a buffer previously queued using the \textit{VIDIOC_QBUF} +ioctl is done being processed and can be used by the driver again. This is like +an implicit \textit{VIDIOC_DQBUF} ioctl. + +\begin{lstlisting} +struct virtio_media_event_dqbuf { + struct virtio_media_event_header hdr; + struct v4l2_buffer buffer; + struct v4l2_plane planes[VIDEO_MAX_PLANES];Maybe use
VIRTIO_MEDIA_MAX_PLANESand define it as#define
VIRTIO_MEDIA_MAX_PLANESVIDEO_MAX_PLANESor
#define
VIRTIO_MEDIA_MAX_PLANES8if we don't want to depend on videodev2.h's definitions here.
device-types/media/description.tex line 551
Quoted 13 lines
device-types/media/description.tex new file +13
-0+}; +\end{lstlisting} + +\begin{description} +\item[\field{buffer}] \textit{struct v4l2_buffer} describing the buffer that has been dequeued. +\item[\field{planes}] array of \textit{struct v4l2_plane} containing the plane information for multi-planar buffers. +\end{description} + +Pointer values in the \textit{struct v4l2_buffer} and \textit{struct v4l2_plane} +are meaningless and must be ignored by the driver. It is recommended that the +device sets them to NULL in order to avoid leaking potential device addresses. + +Note that in the case of a \field{USERPTR} buffer, the \textit{struct v4l2_buffer}s/USERPTR/SHARED_PAGES.
device-types/media/description.tex line 552
Quoted 1 line
device-types/media/description.tex new file +1
-0+used as event payload is not followed by the buffer memory:"... not followed by the buffer's SG entries" is probably more precise.
Cheers,
Alex.
Re: [PATCH v4 1/1] virtio-media: Add virtio media device specification
Hi Albert,
Quoted 346 lines · Albert Esteve
Hi Alexandre,
Thanks for your comments! Partially covered here, responded inline below. I'll go through the rest tomorrow.
Quoted 331 lines · Alexandre Courbot
Hi Albert,
Thanks again for driving this. The current version looks good to me, but here are a few things I found while doing another pass.
Quoted 144 lines · Albert Esteve
Virtio-media is an encapsulation of the V4L2 UAPI into virtio, able to virtualize any video device supported by V4L2.
Note that virtio-media does not require the use of a V4L2 device driver on the host or guest side - V4L2 is only used as a host-guest protocol, and both sides are free to convert it from/to any model that they wish to use.
Signed-off-by Albert Esteve <[email protected]>--- conformance.tex | 13 +- content.tex | 1 + device-types/media/description.tex | 575 ++++++++++++++++++++++ device-types/media/device-conformance.tex | 10 + device-types/media/driver-conformance.tex | 8 + 5 files changed, 603 insertions(+), 4 deletions(-) create mode 100644 device-types/media/description.tex create mode 100644 device-types/media/device-conformance.tex create mode 100644 device-types/media/driver-conformance.texconformance.tex +9
-4@@ -32,8 +32,10 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Driver Conformance / Memory Driver Conformance}, \ref{sec:Conformance / Driver Conformance / I2C Adapter Driver Conformance}, \ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance}, -\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance} or -\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}. +\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance}, +\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance} or +\ref{sec:Conformance / Driver Conformance / Media Driver Conformance}. + \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -59,8 +61,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Device Conformance / Memory Device Conformance}, \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance}, \ref{sec:Conformance / Device Conformance / SCMI Device Conformance}, -\ref{sec:Conformance / Device Conformance / GPIO Device Conformance} or -\ref{sec:Conformance / Device Conformance / PMEM Device Conformance}. +\ref{sec:Conformance / Device Conformance / GPIO Device Conformance}, +\ref{sec:Conformance / Device Conformance / PMEM Device Conformance} or +\ref{sec:Conformance / Device Conformance / Media Device Conformance}. \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -152,6 +155,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/driver-conformance.tex} \input{device-types/gpio/driver-conformance.tex} \input{device-types/pmem/driver-conformance.tex} +\input{device-types/media/driver-conformance.tex} \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}@@ -238,6 +242,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/device-conformance.tex} \input{device-types/gpio/device-conformance.tex} \input{device-types/pmem/device-conformance.tex} +\input{device-types/media/device-conformance.tex} \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance} A conformant implementation MUST be either transitional orcontent.tex +1
-0@@ -767,6 +767,7 @@ \chapter{Device Types}\label{sec:Device Types} \input{device-types/scmi/description.tex} \input{device-types/gpio/description.tex} \input{device-types/pmem/description.tex} +\input{device-types/media/description.tex} \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}device-types/media/description.tex new file +58
-0@@ -0,0 +1,575 @@ +\section{Media Device}\label{sec:Device Types / Media Device} + +The virtio media device follows the same model (and structures) as V4L2. It +can be used to virtualize cameras, codec devices, or any other device +supported by V4L2. The complete definition of V4L2 structures and ioctls can +be found under the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/index.html}{V4L2 UAPI documentation}. + +V4L2 is a UAPI that allows a less privileged entity (user-space) to use video +hardware exposed by a more privileged entity (the kernel). Virtio-media is an +encapsulation of this API into virtio, turning it into a virtualization API +for all classes of video devices supported by V4L2, where the device plays the +role of the kernel and the driver the role of user-space. + +The device is therefore responsible for presenting a virtual device that behaves +like an actual V4L2 device, which the driver can control. + +Note that virtio-media does not require the use of a V4L2 device driver or of +Linux on any side - V4L2 is only used as a transport protocol, +and both sides are free to convert it from/to any model that they wish to use. + +\subsection{Device ID}\label{sec:Device Types / Media Device / Device ID} + +48 + +\subsection{Virtqueues}\label{sec:Device Types / Media Device / Virtqueues} + +\begin{description} +\item[0] commandq - used for driver commands and device responses to these +commands. +\item[1] eventq - used for events sent by the device to the driver. +\end{description} + +\subsection{Feature Bits}\label{sec:Device Types / Media Device / Feature Bits} + +None + +\subsection{Device Configuration Layout}\label{sec:Device Types / Media Device / Device Configuration Layout} + +The video device configuration space uses the following layout: + +\begin{lstlisting} +struct virtio_media_config { + le32 device_caps; + le32 device_type; + le8 card[32]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{device_caps}] (driver-read-only) flags representing the device +capabilities as used in +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/vidioc-querycap.html#c.V4L.v4l2_capability}{struct v4l2_capability}. +It corresponds with the \field{device_caps} field in the \textit{struct video_device}. +\item[\field{device_type}] (driver-read-only) informs the driver of the type +of the video device. It corresponds with the \field{vfl_devnode_type} field of the device. +\item[\field{card}] (driver-read-only) name of the device, a NUL-terminated +UTF-8 string. It corresponds with the \field{card} field of the \textit{struct v4l2_capability}.device-types/media/description.tex line 68
Quoted 10 lines
device-types/media/description.tex new file +10
-0+If all the characters of the field are used, it does not need to be NUL-terminated. +\end{description} + +\subsection{Device Initialization} + +\begin{enumerate} +\item Read the \field{device_caps} and \field{device_type} fields +from the configuration layout to identify the device. +\item Set up the \field{commandq} and \field{eventq}. +\item May open a session (see Section \ref{sec:Device Operation: Command Virtqueue: Sessions})This reference does not resolve.
Quoted 48 lines
device-types/media/description.tex new file +48
-0+to use the device and send V4L2 ioctls in order to receive more information +about the device, such as supported formats or controls. +\end{enumerate} + +\subsection{Device Operation}\label{sec:Device Types / Media Device / Device Operation} + +Commands are queued on the command queue by the driver for the device to +process. The errors returned by each command are standard +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/gen-errors.html}{Linux kernel error codes}. +For instance, a command that contains invalid options will return \textit{EINVAL}. + +Events are sent on the event queue by the device for the driver to handle. + +\subsubsection{Command Virtqueue} + +\paragraph{Device Operation: Command headers} + +\begin{lstlisting} +#define VIRTIO_MEDIA_CMD_OPEN 1 +#define VIRTIO_MEDIA_CMD_CLOSE 2 +#define VIRTIO_MEDIA_CMD_IOCTL 3 +#define VIRTIO_MEDIA_CMD_MMAP 4 +#define VIRTIO_MEDIA_CMD_MUNMAP 5 + +/* Header for all virtio commands from the driver to the device on the commandq. */ +struct virtio_media_cmd_header { + le32 cmd; + le32 __reserved; +}; + +/* Header for all virtio responses from the device to the driver on the commandq. */ +struct virtio_media_resp_header { + le32 status; + le32 __reserved; +}; +\end{lstlisting} + +A command consists of a command header \textit{virtio_media_cmd_header} +containing the following device-readable field: + +\begin{description} +\item[\field{cmd}] specifies a device request type (VIRTIO_MEDIA_CMD_*). +\end{description} + +A response consists of a response header \textit{virtio_media_resp_header} +containing the following device-writable field: + +\begin{description}device-types/media/description.tex line 126
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\item[\field{status}] indicates a device request status. +\end{description} + +The status field can take 0 if the command was successful, or one of the +standard Linux error codes if it was not. + +\drivernormative{\paragraph}{Device Operation: Command Virtqueue: Sessions}{Device Types / Media Device / Device Operation / Command Virtqueue} + +Sessions are how the device is multiplexed, allowing several distinct works to +take place simultaneously. Before start operating, the driver needs to open anit: "Before starting operation" maybe?
Quoted 51 lines
device-types/media/description.tex new file +51
-0+session. This is equivalent to opening the \textit{/dev/videoX} file of the +V4L2 device. Each session gets a unique ID assigned, which can be then used +to perform actions on it. + +\paragraph{Device Operation: Open device} + +\textbf{VIRTIO_MEDIA_CMD_OPEN} Command for creating a new session. + +This is the equivalent of calling \textit{open} on a V4L2 device node. +The driver uses \textit{virtio_media_cmd_open} to send an open request. + +\begin{lstlisting} +struct virtio_media_cmd_open { + struct virtio_media_cmd_header hdr; +}; +\end{lstlisting} + +The device responds to \textit{VIRTIO_MEDIA_CMD_OPEN} with \textit{virtio_media_resp_open}. + +\begin{lstlisting} +struct virtio_media_resp_open { + struct virtio_media_resp_header hdr; + le32 session_id; + le32 __reserved; +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the current session, which is used for +other commands, predominantly ioctls. +\end{description} + +\devicenormative{\subparagraph}{Device Operation: Open device}{Device Types / Media Device / Device Operation / Open device} + +Upon success, the device MUST set a \field{session_id} in \textit{virtio_media_resp_open} +to an integer that is NOT used by any other open session. + +\paragraph{Device Operation: Close device} + +\textbf{VIRTIO_MEDIA_CMD_CLOSE} Command for closing an active session. + +This is the equivalent of calling \textit{close} on a previously opened V4L2 +device node. All resources associated with this session will be freed. + +This command does not require a response from the device. + +\begin{lstlisting} +struct virtio_media_cmd_close { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 __reserved;device-types/media/description.tex line 187
Quoted 10 lines
device-types/media/description.tex new file +10
-0+}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] specifies an identifier for the session to close. +\end{description} + +\drivernormative{\subparagraph}{Device Operation: Close device}{Device Types / Media Device / Device Operation / Close device} + +The session ID SHALL NOT be used again after queueing this command.... until it has been obtained again through a subsequent
VIRTIO_MEDIA_CMD_OPENcall. (session IDs can be recycled ; this should go without saying but it doesn't hurt to mention it)Quoted 5 lines
device-types/media/description.tex new file +5
-0+ +\paragraph{Device Operation: V4L2 ioctls} + +\textbf{VIRTIO_MEDIA_CMD_IOCTL} Command for executing an ioctl on an open +session.device-types/media/description.tex line 202
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +This command tells the device to run one of the `VIDIOC_*` ioctls on the +session identified by \textit{session_id}. + +\begin{lstlisting} +struct virtio_media_cmd_ioctl { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 code; + /* Followed by the relevant ioctl payload as defined in the macro */"ioctl command payload" maybe.
Quoted 22 lines
device-types/media/description.tex new file +22
-0+}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the session to run the ioctl on. +\item[\field{code}] specifies the code of the \field{VIDIOC_*} ioctl to run. +\end{description} + +The code is extracted from the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/videodev.html}{videodev2.h}, +header file. The file defines the ioctl's codes, type of payload, and +direction. The code consists of the second argument of the \field{_IO*} macro. + +For example, the \textit{VIDIOC_G_FMT} is defined as follows: + +\begin{lstlisting} +#define VIDIOC_G_FMT _IOWR('V', 4, struct v4l2_format) +\end{lstlisting} + +This means that its ioctl code is \textit{4}, its payload is a +\textit{struct v4l2_format}, and its direction is \textit{WR} (i.e., the +payload is written by both the driver and the device).Quoted 8 lines
device-types/media/description.tex line 229
Quoted 5 lines
device-types/media/description.tex new file +5
-0+See Section \ref{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} +for more information about the direction of ioctls. +Note that although most architectures use this format, there +are some architecture-specific encoding differences. +See \textit{include/ARCH/ioctl.h} for more details.Is the last sentence needed?
videodev2.his not architecture-specific, so I don't think this adds anything for our purpose?
Quoted 5 lines
Probably not. The idea was to highlight that in some cases the payload may vary depending on the underlying architecture for some ioctls, due to these architecture differences.
But I agree that it is a detail that should not concern to those wanting to implement the specs. So I will remove the lines.
Thanks. My concern was that reader would try to find reasons to care about architecture when all they need to do is look at the ioctls declarations from videodev2.h and copy the second field.
Quoted 43 lines
Quoted 43 lines
Quoted 4 lines
device-types/media/description.tex new file +2
-0+ +The payload struct layout always matches the 64-bit, little-endiandevice-types/media/description.tex line 241
Quoted 10 lines
device-types/media/description.tex new file +10
-0+representation of the corresponding V4L2 structure. For most structs, the +size is identical for both 32 and 64 bits versions. Otherwise, the driver +must translate them to the aforementioned size and endianess. + +The device responds to \textit{VIRTIO_MEDIA_CMD_IOCTL} with \textit{virtio_media_resp_ioctl}. + +\begin{lstlisting} +struct virtio_media_resp_ioctl { + struct virtio_media_resp_header hdr; + /* Followed by the ioctl payload as defined in the macro */Similarly, "ioctl response payload"?
Quoted 12 lines
device-types/media/description.tex new file +12
-0+}; +\end{lstlisting} + +\subparagraph{Ioctls payload}\label{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} + +Each ioctl has a payload, which is defined by the third argument of the +\field{_IO*} macro. + +The payload of an ioctl in the descriptor chain follows the command structure, +the response structure, or both depending on the direction: + +\begin{itemize}device-types/media/description.tex line 263
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\item \textbf{_IOR} is read-only for the driver, meaning the payload +follows the response in the device-writable section of the descriptor chain. +\item \textbf{_IOW} is read-only for the device, meaning the payload +follows the command in the driver-writable section of the descriptor chain. +\item \textbf{_IOWR} is writable by both the device and driver, +meaning the payload must follow both the command in the driver-writable section +of the descriptor chain, and the response in the device-writable section. +\end{itemize} + +A common optimization for \textit{WR} ioctls is to provide the payload usings/common/possible? Unless this is indeed commonly used by other devices, but I doubt it.
Quoted 8 lines
Quoted 8 lines
device-types/media/description.tex line 266
Quoted 3 lines
device-types/media/description.tex new file +3
-0+descriptors that both point to the same buffer. This mimics the behavior of +V4L2 ioctls where the data is only passed once and used as both input and +output by the kernel.That's actually something I'd like to confirm - is it ok to have the device-readable and device-writable descriptors point to the same memory? I don't see why it wouldn't, but in case this is a problem we may want to change this and update the driver accordingly.
Thanks to Matias and Michael's confirmation it seems there is no problem doing this.
Quoted 50 lines
Quoted 38 lines
device-types/media/description.tex line 278
Quoted 13 lines
device-types/media/description.tex new file +12
-0+ +\devicenormative{\subparagraph}{Device Operation: V4L2 ioctls}{Device Types / Media Device / Device Operation / V4L2 ioctls} + +In case of success of a device-writable ioctl, the device MUST always write the +payload in the device-writable part of the descriptor chain. + +In case of failure of a device-writable ioctl, the device is free to write the +payload in the device-writable part of the descriptor chain or not. Some errors +may still result in the payload being updated, and in this case the device is +expected to write the updated payload. If the device has not written the +payload after an error (i.e., only the header is returned), the driver MUST +assume that the payload has not been modified.This last sentence sounds like it is driver-normative, not device-normative.
device-types/media/description.tex line 289
Quoted 11 lines
device-types/media/description.tex new file +11
-0+ +\subparagraph{Handling of pointers in ioctl payload} + +A few structures used as ioctl payloads contain pointers to further +data needed for the ioctl. There are notably: + +\begin{itemize} +\item The \field{planes} pointer of +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-buffer}{struct v4l2_buffer}, +which size is determined by the length member. +\item The \field{controls} pointer of \textit{struct v4l2_ext_controls}, whichSince
struct v4l2_bufferabove is a link, do we want to one forstruct v4l2_ext_controlsas well?device-types/media/description.tex line 293
Quoted 4 lines
device-types/media/description.tex new file +4
-0+size is determined by the count member. +\end{itemize} + +If the size of the pointed area is determined to be non-zero, then the mainnit: "If the size of the pointed area is non-zero, ..."
Quoted 7 lines
device-types/media/description.tex line 298
Quoted 5 lines
device-types/media/description.tex new file +5
-0+payload is immediately followed by the pointed data in their order of +appearance in the structure, and the pointer value itself is ignored by the +device, which must also return the value initially passed by the driver. + +\subparagraph{Handling of pointers to userspace memory}s/userspace/guest ?
To me this refers to the fact that the original fields in the related structures refer to userspace memory (outside of the host/guest context). I think I'd keep it as is.
Quoted 6 lines
Maybe changing the titles of this and the previous sections to:
\subparagraph{Handling of pointers to data in ioctl payload} and
\subparagraph{Handling of pointers to userspace memory in ioctl payload}wdyt?
Sounds good to me.
Quoted 10 lines
Quoted 5 lines
I'd also suggest adding a "(
SHARED_PAGESmemory type)" at the end of the title to make it explicit that this is specific to this particular memory type.Maybe a clarification in the first paragraph (and a link to the related section) would be enough? I feel having a memory type in the title when this concept has not been yet introduced may be confusing. Having that in the text allows to link the section and jump ahead to relate both concepts.
Agreed, a clarification in the text is probably better for providing context.
Quoted 124 lines
Quoted 17 lines
device-types/media/description.tex line 309
Quoted 13 lines
device-types/media/description.tex new file +11
-0+\label{sec:Device Types / Media Device / V4L2 ioctls / Userspace memory} + +A few pointers are special in that they point to userspace memory in the +original V4L2 specification. They are: + +\begin{itemize} +\item The \field{m.userptr} member of \textit{struct v4l2_buffer} and +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-plane}{struct v4l2_plane} +(technically an unsigned long, but designated a userspace address). +\item The \field{ptr} member of \textit{struct v4l2_ext_ctrl}. +\end{itemize}Same as above, let's either add links to all V4L2 structs, or just remove them altogether since we point to the documentation in the introduction anyway.
I added the link for your comment above, but I think with that change, this is ok as it is. I only add the link to those structs that have not been linked before.
Quoted 93 lines
Quoted 37 lines
device-types/media/description.tex new file +36
-0+ +These pointers can cover large areas of scattered memory, which has the +potential to require more descriptors than the virtio queue can provide. For +these particular pointers only, a list of \textit{struct virtio_media_sg_entry} +that covers the needed amount of memory for the pointer is used instead of +using descriptors to map the pointed memory directly. + +\begin{lstlisting} +struct virtio_media_sg_entry { + le64 start; + le32 len; + le32 __reserved; +}; +\end{lstlisting} + +For each such pointer to read, the device reads as many SG entries as needed +to cover the length of the pointed buffer, as described by its parent +structure (\field{length} member of \textit{struct v4l2_buffer} or +\textit{struct v4l2_plane} for buffer memory, and \field{size} member of +\textit{struct v4l2_ext_control} for control data). + +Since the device never needs to modify the list of SG entries, it is only +provided by the driver in the device-readable section of the descriptor chain, +and not repeated in the device-writable section, even for WR ioctls. + +\subparagraph{Unsupported ioctls} + +A few ioctls are replaced by other, more suitable mechanisms. + +\begin{itemize} +\item \textit{VIDIOC_QUERYCAP} is replaced by reading the configuration area +(see \ref{sec:Device Types / Media Device / Device Configuration Layout}). +\item \textit{VIDIOC_DQBUF} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Dequeue buffer}). +\item \textit{VIDIOC_DQEVENT} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Emit an event}).device-types/media/description.tex line 355
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\item \textit{VIDIOC_G_JPEGCOMP} and \textit{VIDIOC_S_JPEGCOMP} are deprecated +and replaced by the controls of the JPEG class. +\item \textit{VIDIOC_LOG_STATUS} is a driver-only operation and shall not be +implemented by the device. +\end{itemize} + +\devicenormative{\subparagraph}{Device Operation: Unsupported ioctls}{Device Types / Media Device / Device Operation / Unsupported ioctls} + +When a request is not supported, the device MUST return \textit{ENOTTY}, +which corresponds with the response for unknown ioctls.s/with/to
Quoted 5 lines
device-types/media/description.tex new file +5
-0+ +\paragraph{Device Operation: Mapping a MMAP buffer} + +\textbf{VIRTIO_MEDIA_CMD_MMAP} Command for mapping a MMAP buffer into the +driver's address space.device-types/media/description.tex line 370
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +Shared memory region ID 0 is used to map MMAP buffers with +the \textit{VIRTIO_MEDIA_CMD_MMAP} command. + +\begin{lstlisting} +#define VIRTIO_MEDIA_MMAP_FLAG_RW (1 << 0) + +struct virtio_media_cmd_mmap { + struct virtio_media_cmd_header hdr; + le32 session_id;
session_idis not mentioned in the member description below.Quoted 25 lines
device-types/media/description.tex new file +25
-0+ le32 flags; + le32 offset; +}; +\end{lstlisting} + +\begin{description} +\item[\field{flags}] is the set of flags for the mapping. \field{VIRTIO_MEDIA_MMAP_FLAG_RW} +can be set if a read-write mapping is desired. Without this flag the mapping +will be read-only. +\item[\field{offset}] corresponds to the \field{mem_offset} field of the +\textit{union v4l2_plane} for the plane to map. This field can be obtained +using the \textit{VIDIOC_QUERYBUF} ioctl. +\end{description} + +The device responds to \textit{VIRTIO_MEDIA_CMD_MMAP} with \textit{virtio_media_resp_mmap}. + +\begin{lstlisting} +struct virtio_media_resp_mmap { + struct virtio_media_resp_header hdr; + le64 driver_addr; + le64 len; +}; +\end{lstlisting} + +\begin{description}Quoted 8 lines
device-types/media/description.tex line 398
Quoted 3 lines
device-types/media/description.tex new file +3
-0+\item[\field{driver_addr}] offset into SHM region ID 0 of the start of the mapping. +\item[\field{len}] length of the mapping as indicated by the \textit{struct v4l2_plane} +the buffer belongs to.This makes me think: do we need to return the length of the mapping when we already know the length of the
v4l2_planeand we only map whole buffers? It probably doesn't hurt to repeat that information here but maybe that's something we can optimize out?
Quoted 4 lines
That's a good point. Indeed, no partial mapping should be allowed, so the length field does sound a bit redundant. Unless it needs error checking, which probably does not, I think it can be skipped. I'll do for the next version. Thanks!
Thanks - I will make sure to update the driver and devices as well to confirm this doesn't introduce any issue in practice.
Quoted 110 lines
Quoted 40 lines
Quoted 40 lines
device-types/media/description.tex new file +38
-0+\end{description} + +\paragraph{Device Operation: Unmapping a MMAP buffer} + +\textbf{VIRTIO_MEDIA_CMD_MUNMAP} Unmap a MMAP buffer previously mapped using \field{VIRTIO_MEDIA_CMD_MMAP}. + +\begin{lstlisting} +struct virtio_media_cmd_munmap { + struct virtio_media_cmd_header hdr; + le64 driver_addr; +}; +\end{lstlisting} + +\begin{description} +\item[\field{driver_addr}] offset into SHM region ID 0 previously returned by +\textit{VIRTIO_MEDIA_CMD_MMAP} at which the buffer has been previously mapped. +\end{description} + +The device responds to \textit{VIRTIO_MEDIA_CMD_MUNMAP} with \textit{virtio_media_resp_munmap}. + +\begin{lstlisting} +struct virtio_media_resp_munmap { + struct virtio_media_resp_header hdr; +}; +\end{lstlisting} + +\devicenormative{\subparagraph}{Device Operation: Unmapping a MMAP buffer}{Device Types / Media Device / Device Operation / Unmapping a MMAP buffer} + +The device MUST keep mappings performed using \textit{VIRTIO_MEDIA_CMD_MMAP} +valid until \textit{VIRTIO_MEDIA_CMD_MUNMAP} is called, even if the buffers or +session they belong to are released or closed by the driver. + +\paragraph{Device Operation: Memory Types} + +The semantics of the three V4L2 memory types (\textit{MMAP}, \textit{USERPTR} +and \textit{DMABUF}) can easily be mapped to both driver and device context. + +However, the driver shall only advertise support for \textit{MMAP} andQuoted 8 lines
device-types/media/description.tex line 439
Quoted 3 lines
device-types/media/description.tex new file +3
-0+\textit{DMABUF} to the guest userspace. \textit{USERPTR} is merely employed to +discern \textit{SHARED_PAGES} buffers, similar to how \textit{DMABUF} is used +to signify \textit{VIRTIO_OBJECT} buffers in the driver and device context.This paragraph sounds specific to the Linux guest driver, as upstream signaled they prefer if drivers don't expose USERPTR to userspace. But it's probably not related for non-V4L2 guest drivers, so I'd suggest we remove it.
Uhm, good point. I think I am biased and failed to consider that. I agree in that the spec should be kept generic and avoid guest-specific details like this. I'll remove the paragraph!
Quoted 53 lines
We probably also want to define the values, I am using this in the driver:
enum
virtio_media_memory{VIRTIO_MEDIA_MMAP=V4L2_MEMORY_MMAP,VIRTIO_MEDIA_SHARED_PAGES=V4L2_MEMORY_USERPTR,VIRTIO_MEDIA_OBJECT=V4L2_MEMORY_DMABUF,
};device-types/media/description.tex line 450
Quoted 11 lines
device-types/media/description.tex new file +11
-0+ +\subparagraph{MMAP} + +In virtio-media, \textit{MMAP} buffers are provisioned by the device, just like +they are by the kernel in regular V4L2. Similarly to how userspace can map a +\textit{MMAP} buffer into its address space using mmap and munmap, the +virtio-media driver can map device buffers into the driver space by queueing the +\textit{struct virtio_media_cmd_mmap} and \textit{struct virtio_media_cmd_munmap} +commands to the commandq. + +\subparagraph{SHARED_PAGES}Maybe add something to say this is the semantic equivalent of
V4L2_MEMORY_USERPTR.device-types/media/description.tex line 463
Quoted 13 lines
device-types/media/description.tex new file +13
-0+ +In virtio-media, \textit{SHARED_PAGES} buffers are provisioned by the driver, +and use guest physical addresses. Instances of \textit{struct v4l2_buffer} +and \textit{struct v4l2_plane} of this type are followed by a list of +\textit{struct virtio_media_sg_entry}. For more information, see +\ref{sec:Device Types / Media Device / V4L2 ioctls / Userspace memory} + +The device must not alter the pointer values provided by the driver, i.e. +\field{the m.userptr} member of \textit{struct v4l2_buffer} and +\textit{struct v4l2_plane} must be returned to the driver with the same value +as it was provided. + +\subparagraph{VIRTIO_OBJECT}Similarly we probably want to say this is the semantic equivalent of
V4L2_MEMORY_DMABUF.device-types/media/description.tex line 472
Quoted 9 lines
device-types/media/description.tex new file +9
-0+ +In virtio-media, \textit{VIRTIO_OBJECT} buffers are provisioned by a virtio +object, just like they are by a \textit{DMABUF} in regular V4L2. Virtio objects +are 16-bytes UUIDs and do not fit in the placeholders for file descriptors, so +they follow their embedding data structure as needed and the device must +leave the V4L2 structure placeholder unchanged. + +Contrary to \textit{USERPTR} buffers, virtio objects UUIDs need to be added in +both the device-readable and device-writable section of the descriptor chain.s/USERPTR/SHARED_PAGES.
Quoted 3 lines
I'm also not quite sure why we concluded the objects UUIDs need to be on both the device-readable and device-writable sections. Do you happen to remember?
Quoted 7 lines
I took this directly from the virtio-media project at
https://github.com/chromeos/virtio-media/blob/main/README.md?plain=1#L444I do not think I got involved in any discussion regarding this detail? Maybe it has to do with the fact that mmaped buffers can be exported by the device, and uses the device-writable section to communicate the new ID back to the driver? Just a guess though.
Right, that was the reason! And indeed I don't think we discussed this thoroughly, and it shows. >_<
The problem with the way things are currently specified is that it changes the behavior of e.g. VIDIOC_QBUF depending on the memory type - buffers backed by a virtio object need to have some space reserved in the device-writable part of the descriptor while SHARED_PAGES buffers don't. That's inconsistent.
It is probably clearer and simpler to add a paragraph about the VIDIOC_EXPBUF ioctl and tell that space for the object ID need to be reserved in the device-writable part of the descriptor, as the device writes there. This can normally be inferred from the documentation of VIDIOC_EXPBUF, but I'd also explicitly state it here for clarity. WDYT?
Cheers,
Alex.
Re: [PATCH v4 1/1] virtio-media: Add virtio media device specification
Quoted 611 lines · Alexandre Courbot
Hi Albert,
Quoted 346 lines · Albert Esteve
Hi Alexandre,
Thanks for your comments! Partially covered here, responded inline below. I'll go through the rest tomorrow.
Quoted 331 lines · Alexandre Courbot
Hi Albert,
Thanks again for driving this. The current version looks good to me, but here are a few things I found while doing another pass.
Quoted 144 lines · Albert Esteve
Virtio-media is an encapsulation of the V4L2 UAPI into virtio, able to virtualize any video device supported by V4L2.
Note that virtio-media does not require the use of a V4L2 device driver on the host or guest side - V4L2 is only used as a host-guest protocol, and both sides are free to convert it from/to any model that they wish to use.
Signed-off-by Albert Esteve <[email protected]>--- conformance.tex | 13 +- content.tex | 1 + device-types/media/description.tex | 575 ++++++++++++++++++++++ device-types/media/device-conformance.tex | 10 + device-types/media/driver-conformance.tex | 8 + 5 files changed, 603 insertions(+), 4 deletions(-) create mode 100644 device-types/media/description.tex create mode 100644 device-types/media/device-conformance.tex create mode 100644 device-types/media/driver-conformance.texconformance.tex +9
-4@@ -32,8 +32,10 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Driver Conformance / Memory Driver Conformance}, \ref{sec:Conformance / Driver Conformance / I2C Adapter Driver Conformance}, \ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance}, -\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance} or -\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}. +\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance}, +\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance} or +\ref{sec:Conformance / Driver Conformance / Media Driver Conformance}. + \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -59,8 +61,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Device Conformance / Memory Device Conformance}, \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance}, \ref{sec:Conformance / Device Conformance / SCMI Device Conformance}, -\ref{sec:Conformance / Device Conformance / GPIO Device Conformance} or -\ref{sec:Conformance / Device Conformance / PMEM Device Conformance}. +\ref{sec:Conformance / Device Conformance / GPIO Device Conformance}, +\ref{sec:Conformance / Device Conformance / PMEM Device Conformance} or +\ref{sec:Conformance / Device Conformance / Media Device Conformance}. \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -152,6 +155,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/driver-conformance.tex} \input{device-types/gpio/driver-conformance.tex} \input{device-types/pmem/driver-conformance.tex} +\input{device-types/media/driver-conformance.tex} \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}@@ -238,6 +242,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/device-conformance.tex} \input{device-types/gpio/device-conformance.tex} \input{device-types/pmem/device-conformance.tex} +\input{device-types/media/device-conformance.tex} \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance} A conformant implementation MUST be either transitional orcontent.tex +1
-0@@ -767,6 +767,7 @@ \chapter{Device Types}\label{sec:Device Types} \input{device-types/scmi/description.tex} \input{device-types/gpio/description.tex} \input{device-types/pmem/description.tex} +\input{device-types/media/description.tex} \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}device-types/media/description.tex new file +58
-0@@ -0,0 +1,575 @@ +\section{Media Device}\label{sec:Device Types / Media Device} + +The virtio media device follows the same model (and structures) as V4L2. It +can be used to virtualize cameras, codec devices, or any other device +supported by V4L2. The complete definition of V4L2 structures and ioctls can +be found under the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/index.html}{V4L2 UAPI documentation}. + +V4L2 is a UAPI that allows a less privileged entity (user-space) to use video +hardware exposed by a more privileged entity (the kernel). Virtio-media is an +encapsulation of this API into virtio, turning it into a virtualization API +for all classes of video devices supported by V4L2, where the device plays the +role of the kernel and the driver the role of user-space. + +The device is therefore responsible for presenting a virtual device that behaves +like an actual V4L2 device, which the driver can control. + +Note that virtio-media does not require the use of a V4L2 device driver or of +Linux on any side - V4L2 is only used as a transport protocol, +and both sides are free to convert it from/to any model that they wish to use. + +\subsection{Device ID}\label{sec:Device Types / Media Device / Device ID} + +48 + +\subsection{Virtqueues}\label{sec:Device Types / Media Device / Virtqueues} + +\begin{description} +\item[0] commandq - used for driver commands and device responses to these +commands. +\item[1] eventq - used for events sent by the device to the driver. +\end{description} + +\subsection{Feature Bits}\label{sec:Device Types / Media Device / Feature Bits} + +None + +\subsection{Device Configuration Layout}\label{sec:Device Types / Media Device / Device Configuration Layout} + +The video device configuration space uses the following layout: + +\begin{lstlisting} +struct virtio_media_config { + le32 device_caps; + le32 device_type; + le8 card[32]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{device_caps}] (driver-read-only) flags representing the device +capabilities as used in +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/vidioc-querycap.html#c.V4L.v4l2_capability}{struct v4l2_capability}. +It corresponds with the \field{device_caps} field in the \textit{struct video_device}. +\item[\field{device_type}] (driver-read-only) informs the driver of the type +of the video device. It corresponds with the \field{vfl_devnode_type} field of the device. +\item[\field{card}] (driver-read-only) name of the device, a NUL-terminated +UTF-8 string. It corresponds with the \field{card} field of the \textit{struct v4l2_capability}.device-types/media/description.tex line 68
Quoted 10 lines
device-types/media/description.tex new file +10
-0+If all the characters of the field are used, it does not need to be NUL-terminated. +\end{description} + +\subsection{Device Initialization} + +\begin{enumerate} +\item Read the \field{device_caps} and \field{device_type} fields +from the configuration layout to identify the device. +\item Set up the \field{commandq} and \field{eventq}. +\item May open a session (see Section \ref{sec:Device Operation: Command Virtqueue: Sessions})This reference does not resolve.
Quoted 48 lines
device-types/media/description.tex new file +48
-0+to use the device and send V4L2 ioctls in order to receive more information +about the device, such as supported formats or controls. +\end{enumerate} + +\subsection{Device Operation}\label{sec:Device Types / Media Device / Device Operation} + +Commands are queued on the command queue by the driver for the device to +process. The errors returned by each command are standard +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/gen-errors.html}{Linux kernel error codes}. +For instance, a command that contains invalid options will return \textit{EINVAL}. + +Events are sent on the event queue by the device for the driver to handle. + +\subsubsection{Command Virtqueue} + +\paragraph{Device Operation: Command headers} + +\begin{lstlisting} +#define VIRTIO_MEDIA_CMD_OPEN 1 +#define VIRTIO_MEDIA_CMD_CLOSE 2 +#define VIRTIO_MEDIA_CMD_IOCTL 3 +#define VIRTIO_MEDIA_CMD_MMAP 4 +#define VIRTIO_MEDIA_CMD_MUNMAP 5 + +/* Header for all virtio commands from the driver to the device on the commandq. */ +struct virtio_media_cmd_header { + le32 cmd; + le32 __reserved; +}; + +/* Header for all virtio responses from the device to the driver on the commandq. */ +struct virtio_media_resp_header { + le32 status; + le32 __reserved; +}; +\end{lstlisting} + +A command consists of a command header \textit{virtio_media_cmd_header} +containing the following device-readable field: + +\begin{description} +\item[\field{cmd}] specifies a device request type (VIRTIO_MEDIA_CMD_*). +\end{description} + +A response consists of a response header \textit{virtio_media_resp_header} +containing the following device-writable field: + +\begin{description}device-types/media/description.tex line 126
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\item[\field{status}] indicates a device request status. +\end{description} + +The status field can take 0 if the command was successful, or one of the +standard Linux error codes if it was not. + +\drivernormative{\paragraph}{Device Operation: Command Virtqueue: Sessions}{Device Types / Media Device / Device Operation / Command Virtqueue} + +Sessions are how the device is multiplexed, allowing several distinct works to +take place simultaneously. Before start operating, the driver needs to open anit: "Before starting operation" maybe?
Quoted 51 lines
device-types/media/description.tex new file +51
-0+session. This is equivalent to opening the \textit{/dev/videoX} file of the +V4L2 device. Each session gets a unique ID assigned, which can be then used +to perform actions on it. + +\paragraph{Device Operation: Open device} + +\textbf{VIRTIO_MEDIA_CMD_OPEN} Command for creating a new session. + +This is the equivalent of calling \textit{open} on a V4L2 device node. +The driver uses \textit{virtio_media_cmd_open} to send an open request. + +\begin{lstlisting} +struct virtio_media_cmd_open { + struct virtio_media_cmd_header hdr; +}; +\end{lstlisting} + +The device responds to \textit{VIRTIO_MEDIA_CMD_OPEN} with \textit{virtio_media_resp_open}. + +\begin{lstlisting} +struct virtio_media_resp_open { + struct virtio_media_resp_header hdr; + le32 session_id; + le32 __reserved; +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the current session, which is used for +other commands, predominantly ioctls. +\end{description} + +\devicenormative{\subparagraph}{Device Operation: Open device}{Device Types / Media Device / Device Operation / Open device} + +Upon success, the device MUST set a \field{session_id} in \textit{virtio_media_resp_open} +to an integer that is NOT used by any other open session. + +\paragraph{Device Operation: Close device} + +\textbf{VIRTIO_MEDIA_CMD_CLOSE} Command for closing an active session. + +This is the equivalent of calling \textit{close} on a previously opened V4L2 +device node. All resources associated with this session will be freed. + +This command does not require a response from the device. + +\begin{lstlisting} +struct virtio_media_cmd_close { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 __reserved;device-types/media/description.tex line 187
Quoted 10 lines
device-types/media/description.tex new file +10
-0+}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] specifies an identifier for the session to close. +\end{description} + +\drivernormative{\subparagraph}{Device Operation: Close device}{Device Types / Media Device / Device Operation / Close device} + +The session ID SHALL NOT be used again after queueing this command.... until it has been obtained again through a subsequent
VIRTIO_MEDIA_CMD_OPENcall. (session IDs can be recycled ; this should go without saying but it doesn't hurt to mention it)Quoted 5 lines
device-types/media/description.tex new file +5
-0+ +\paragraph{Device Operation: V4L2 ioctls} + +\textbf{VIRTIO_MEDIA_CMD_IOCTL} Command for executing an ioctl on an open +session.device-types/media/description.tex line 202
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +This command tells the device to run one of the `VIDIOC_*` ioctls on the +session identified by \textit{session_id}. + +\begin{lstlisting} +struct virtio_media_cmd_ioctl { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 code; + /* Followed by the relevant ioctl payload as defined in the macro */"ioctl command payload" maybe.
Quoted 22 lines
device-types/media/description.tex new file +22
-0+}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the session to run the ioctl on. +\item[\field{code}] specifies the code of the \field{VIDIOC_*} ioctl to run. +\end{description} + +The code is extracted from the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/videodev.html}{videodev2.h}, +header file. The file defines the ioctl's codes, type of payload, and +direction. The code consists of the second argument of the \field{_IO*} macro. + +For example, the \textit{VIDIOC_G_FMT} is defined as follows: + +\begin{lstlisting} +#define VIDIOC_G_FMT _IOWR('V', 4, struct v4l2_format) +\end{lstlisting} + +This means that its ioctl code is \textit{4}, its payload is a +\textit{struct v4l2_format}, and its direction is \textit{WR} (i.e., the +payload is written by both the driver and the device).Quoted 8 lines
device-types/media/description.tex line 229
Quoted 5 lines
device-types/media/description.tex new file +5
-0+See Section \ref{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} +for more information about the direction of ioctls. +Note that although most architectures use this format, there +are some architecture-specific encoding differences. +See \textit{include/ARCH/ioctl.h} for more details.Is the last sentence needed?
videodev2.his not architecture-specific, so I don't think this adds anything for our purpose?Quoted 5 lines
Probably not. The idea was to highlight that in some cases the payload may vary depending on the underlying architecture for some ioctls, due to these architecture differences.
But I agree that it is a detail that should not concern to those wanting to implement the specs. So I will remove the lines.Thanks. My concern was that reader would try to find reasons to care about architecture when all they need to do is look at the ioctls declarations from
videodev2.hand copy the second field.Quoted 43 lines
Quoted 43 lines
Quoted 4 lines
device-types/media/description.tex new file +2
-0+ +The payload struct layout always matches the 64-bit, little-endiandevice-types/media/description.tex line 241
Quoted 10 lines
device-types/media/description.tex new file +10
-0+representation of the corresponding V4L2 structure. For most structs, the +size is identical for both 32 and 64 bits versions. Otherwise, the driver +must translate them to the aforementioned size and endianess. + +The device responds to \textit{VIRTIO_MEDIA_CMD_IOCTL} with \textit{virtio_media_resp_ioctl}. + +\begin{lstlisting} +struct virtio_media_resp_ioctl { + struct virtio_media_resp_header hdr; + /* Followed by the ioctl payload as defined in the macro */Similarly, "ioctl response payload"?
Quoted 12 lines
device-types/media/description.tex new file +12
-0+}; +\end{lstlisting} + +\subparagraph{Ioctls payload}\label{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} + +Each ioctl has a payload, which is defined by the third argument of the +\field{_IO*} macro. + +The payload of an ioctl in the descriptor chain follows the command structure, +the response structure, or both depending on the direction: + +\begin{itemize}device-types/media/description.tex line 263
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\item \textbf{_IOR} is read-only for the driver, meaning the payload +follows the response in the device-writable section of the descriptor chain. +\item \textbf{_IOW} is read-only for the device, meaning the payload +follows the command in the driver-writable section of the descriptor chain. +\item \textbf{_IOWR} is writable by both the device and driver, +meaning the payload must follow both the command in the driver-writable section +of the descriptor chain, and the response in the device-writable section. +\end{itemize} + +A common optimization for \textit{WR} ioctls is to provide the payload usings/common/possible? Unless this is indeed commonly used by other devices, but I doubt it.
Quoted 8 lines
Quoted 8 lines
device-types/media/description.tex line 266
Quoted 3 lines
device-types/media/description.tex new file +3
-0+descriptors that both point to the same buffer. This mimics the behavior of +V4L2 ioctls where the data is only passed once and used as both input and +output by the kernel.That's actually something I'd like to confirm - is it ok to have the device-readable and device-writable descriptors point to the same memory? I don't see why it wouldn't, but in case this is a problem we may want to change this and update the driver accordingly.
Thanks to Matias and Michael's confirmation it seems there is no problem doing this.
Quoted 50 lines
Quoted 38 lines
device-types/media/description.tex line 278
Quoted 13 lines
device-types/media/description.tex new file +12
-0+ +\devicenormative{\subparagraph}{Device Operation: V4L2 ioctls}{Device Types / Media Device / Device Operation / V4L2 ioctls} + +In case of success of a device-writable ioctl, the device MUST always write the +payload in the device-writable part of the descriptor chain. + +In case of failure of a device-writable ioctl, the device is free to write the +payload in the device-writable part of the descriptor chain or not. Some errors +may still result in the payload being updated, and in this case the device is +expected to write the updated payload. If the device has not written the +payload after an error (i.e., only the header is returned), the driver MUST +assume that the payload has not been modified.This last sentence sounds like it is driver-normative, not device-normative.
device-types/media/description.tex line 289
Quoted 11 lines
device-types/media/description.tex new file +11
-0+ +\subparagraph{Handling of pointers in ioctl payload} + +A few structures used as ioctl payloads contain pointers to further +data needed for the ioctl. There are notably: + +\begin{itemize} +\item The \field{planes} pointer of +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-buffer}{struct v4l2_buffer}, +which size is determined by the length member. +\item The \field{controls} pointer of \textit{struct v4l2_ext_controls}, whichSince
struct v4l2_bufferabove is a link, do we want to one forstruct v4l2_ext_controlsas well?device-types/media/description.tex line 293
Quoted 4 lines
device-types/media/description.tex new file +4
-0+size is determined by the count member. +\end{itemize} + +If the size of the pointed area is determined to be non-zero, then the mainnit: "If the size of the pointed area is non-zero, ..."
Quoted 7 lines
device-types/media/description.tex line 298
Quoted 5 lines
device-types/media/description.tex new file +5
-0+payload is immediately followed by the pointed data in their order of +appearance in the structure, and the pointer value itself is ignored by the +device, which must also return the value initially passed by the driver. + +\subparagraph{Handling of pointers to userspace memory}s/userspace/guest ?
To me this refers to the fact that the original fields in the related structures refer to userspace memory (outside of the host/guest context). I think I'd keep it as is.
Quoted 6 lines
Maybe changing the titles of this and the previous sections to:
\subparagraph{Handling of pointers to data in ioctl payload} and
\subparagraph{Handling of pointers to userspace memory in ioctl payload}wdyt?
Sounds good to me.
Quoted 10 lines
Quoted 5 lines
I'd also suggest adding a "(
SHARED_PAGESmemory type)" at the end of the title to make it explicit that this is specific to this particular memory type.Maybe a clarification in the first paragraph (and a link to the related section) would be enough? I feel having a memory type in the title when this concept has not been yet introduced may be confusing. Having that in the text allows to link the section and jump ahead to relate both concepts.
Agreed, a clarification in the text is probably better for providing context.
Quoted 124 lines
Quoted 17 lines
device-types/media/description.tex line 309
Quoted 13 lines
device-types/media/description.tex new file +11
-0+\label{sec:Device Types / Media Device / V4L2 ioctls / Userspace memory} + +A few pointers are special in that they point to userspace memory in the +original V4L2 specification. They are: + +\begin{itemize} +\item The \field{m.userptr} member of \textit{struct v4l2_buffer} and +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-plane}{struct v4l2_plane} +(technically an unsigned long, but designated a userspace address). +\item The \field{ptr} member of \textit{struct v4l2_ext_ctrl}. +\end{itemize}Same as above, let's either add links to all V4L2 structs, or just remove them altogether since we point to the documentation in the introduction anyway.
I added the link for your comment above, but I think with that change, this is ok as it is. I only add the link to those structs that have not been linked before.
Quoted 101 lines
Quoted 37 lines
device-types/media/description.tex new file +36
-0+ +These pointers can cover large areas of scattered memory, which has the +potential to require more descriptors than the virtio queue can provide. For +these particular pointers only, a list of \textit{struct virtio_media_sg_entry} +that covers the needed amount of memory for the pointer is used instead of +using descriptors to map the pointed memory directly. + +\begin{lstlisting} +struct virtio_media_sg_entry { + le64 start; + le32 len; + le32 __reserved; +}; +\end{lstlisting} + +For each such pointer to read, the device reads as many SG entries as needed +to cover the length of the pointed buffer, as described by its parent +structure (\field{length} member of \textit{struct v4l2_buffer} or +\textit{struct v4l2_plane} for buffer memory, and \field{size} member of +\textit{struct v4l2_ext_control} for control data). + +Since the device never needs to modify the list of SG entries, it is only +provided by the driver in the device-readable section of the descriptor chain, +and not repeated in the device-writable section, even for WR ioctls. + +\subparagraph{Unsupported ioctls} + +A few ioctls are replaced by other, more suitable mechanisms. + +\begin{itemize} +\item \textit{VIDIOC_QUERYCAP} is replaced by reading the configuration area +(see \ref{sec:Device Types / Media Device / Device Configuration Layout}). +\item \textit{VIDIOC_DQBUF} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Dequeue buffer}). +\item \textit{VIDIOC_DQEVENT} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Emit an event}).device-types/media/description.tex line 355
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\item \textit{VIDIOC_G_JPEGCOMP} and \textit{VIDIOC_S_JPEGCOMP} are deprecated +and replaced by the controls of the JPEG class. +\item \textit{VIDIOC_LOG_STATUS} is a driver-only operation and shall not be +implemented by the device. +\end{itemize} + +\devicenormative{\subparagraph}{Device Operation: Unsupported ioctls}{Device Types / Media Device / Device Operation / Unsupported ioctls} + +When a request is not supported, the device MUST return \textit{ENOTTY}, +which corresponds with the response for unknown ioctls.s/with/to
Quoted 5 lines
device-types/media/description.tex new file +5
-0+ +\paragraph{Device Operation: Mapping a MMAP buffer} + +\textbf{VIRTIO_MEDIA_CMD_MMAP} Command for mapping a MMAP buffer into the +driver's address space.device-types/media/description.tex line 370
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +Shared memory region ID 0 is used to map MMAP buffers with +the \textit{VIRTIO_MEDIA_CMD_MMAP} command. + +\begin{lstlisting} +#define VIRTIO_MEDIA_MMAP_FLAG_RW (1 << 0) + +struct virtio_media_cmd_mmap { + struct virtio_media_cmd_header hdr; + le32 session_id;
session_idis not mentioned in the member description below.Quoted 18 lines
device-types/media/description.tex new file +18
-0+ le32 flags; + le32 offset; +}; +\end{lstlisting} + +\begin{description} +\item[\field{flags}] is the set of flags for the mapping. \field{VIRTIO_MEDIA_MMAP_FLAG_RW} +can be set if a read-write mapping is desired. Without this flag the mapping +will be read-only. +\item[\field{offset}] corresponds to the \field{mem_offset} field of the +\textit{union v4l2_plane} for the plane to map. This field can be obtained +using the \textit{VIDIOC_QUERYBUF} ioctl. +\end{description} + +The device responds to \textit{VIRTIO_MEDIA_CMD_MMAP} with \textit{virtio_media_resp_mmap}. + +\begin{lstlisting} +struct virtio_media_resp_mmap {device-types/media/description.tex line 398
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ struct virtio_media_resp_header hdr; + le64 driver_addr; + le64 len; +}; +\end{lstlisting} + +\begin{description} +\item[\field{driver_addr}] offset into SHM region ID 0 of the start of the mapping. +\item[\field{len}] length of the mapping as indicated by the \textit{struct v4l2_plane} +the buffer belongs to.This makes me think: do we need to return the length of the mapping when we already know the length of the
v4l2_planeand we only map whole buffers? It probably doesn't hurt to repeat that information here but maybe that's something we can optimize out?
Quoted 7 lines
Quoted 4 lines
That's a good point. Indeed, no partial mapping should be allowed, so the length field does sound a bit redundant. Unless it needs error checking, which probably does not, I think it can be skipped. I'll do for the next version. Thanks!
Thanks - I will make sure to update the driver and devices as well to confirm this doesn't introduce any issue in practice.
So after trying to update the code to remove the length field, I have come to the conclusion that it is useful after all.
While it is true that the returned length of the mapping will always be equal to the length of the buffer, the semantics of mmap require that we check that the mapped area fits inside the requested VMA, and in order to do so we need the length of the mapping.
Why can't we use the length of the buffer, since it is the same value? The driver doesn't know that length until a QUERYBUF ioctl is performed, and although it is almost certain that the user will do a QUERYBUF -> mmap in practice (if only to obtain the offset to use as a mmap parameter), there is no formal guarantee of this, and doing so would require some caching mechanism that is a bit more complex than what I am comfortable putting in a kernel driver if it is not absolutely necessary.
So we should keep this len parameter, and maybe just add a device-normative paragraph stating that it must always be equal to the length of the buffer.
Thanks,
Alex.
Re: [PATCH v4 1/1] virtio-media: Add virtio media device specification
Quoted 638 lines · Alexandre Courbot
Quoted 611 lines · Alexandre Courbot
Hi Albert,
Quoted 346 lines · Albert Esteve
Hi Alexandre,
Thanks for your comments! Partially covered here, responded inline below. I'll go through the rest tomorrow.
Quoted 331 lines · Alexandre Courbot
Hi Albert,
Thanks again for driving this. The current version looks good to me, but here are a few things I found while doing another pass.
Quoted 144 lines · Albert Esteve
Virtio-media is an encapsulation of the V4L2 UAPI into virtio, able to virtualize any video device supported by V4L2.
Note that virtio-media does not require the use of a V4L2 device driver on the host or guest side - V4L2 is only used as a host-guest protocol, and both sides are free to convert it from/to any model that they wish to use.
Signed-off-by Albert Esteve <[email protected]>--- conformance.tex | 13 +- content.tex | 1 + device-types/media/description.tex | 575 ++++++++++++++++++++++ device-types/media/device-conformance.tex | 10 + device-types/media/driver-conformance.tex | 8 + 5 files changed, 603 insertions(+), 4 deletions(-) create mode 100644 device-types/media/description.tex create mode 100644 device-types/media/device-conformance.tex create mode 100644 device-types/media/driver-conformance.texconformance.tex +9
-4@@ -32,8 +32,10 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Driver Conformance / Memory Driver Conformance}, \ref{sec:Conformance / Driver Conformance / I2C Adapter Driver Conformance}, \ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance}, -\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance} or -\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}. +\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance}, +\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance} or +\ref{sec:Conformance / Driver Conformance / Media Driver Conformance}. + \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -59,8 +61,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Device Conformance / Memory Device Conformance}, \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance}, \ref{sec:Conformance / Device Conformance / SCMI Device Conformance}, -\ref{sec:Conformance / Device Conformance / GPIO Device Conformance} or -\ref{sec:Conformance / Device Conformance / PMEM Device Conformance}. +\ref{sec:Conformance / Device Conformance / GPIO Device Conformance}, +\ref{sec:Conformance / Device Conformance / PMEM Device Conformance} or +\ref{sec:Conformance / Device Conformance / Media Device Conformance}. \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize}@@ -152,6 +155,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/driver-conformance.tex} \input{device-types/gpio/driver-conformance.tex} \input{device-types/pmem/driver-conformance.tex} +\input{device-types/media/driver-conformance.tex} \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}@@ -238,6 +242,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/scmi/device-conformance.tex} \input{device-types/gpio/device-conformance.tex} \input{device-types/pmem/device-conformance.tex} +\input{device-types/media/device-conformance.tex} \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance} A conformant implementation MUST be either transitional orcontent.tex +1
-0@@ -767,6 +767,7 @@ \chapter{Device Types}\label{sec:Device Types} \input{device-types/scmi/description.tex} \input{device-types/gpio/description.tex} \input{device-types/pmem/description.tex} +\input{device-types/media/description.tex} \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}device-types/media/description.tex new file +58
-0@@ -0,0 +1,575 @@ +\section{Media Device}\label{sec:Device Types / Media Device} + +The virtio media device follows the same model (and structures) as V4L2. It +can be used to virtualize cameras, codec devices, or any other device +supported by V4L2. The complete definition of V4L2 structures and ioctls can +be found under the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/index.html}{V4L2 UAPI documentation}. + +V4L2 is a UAPI that allows a less privileged entity (user-space) to use video +hardware exposed by a more privileged entity (the kernel). Virtio-media is an +encapsulation of this API into virtio, turning it into a virtualization API +for all classes of video devices supported by V4L2, where the device plays the +role of the kernel and the driver the role of user-space. + +The device is therefore responsible for presenting a virtual device that behaves +like an actual V4L2 device, which the driver can control. + +Note that virtio-media does not require the use of a V4L2 device driver or of +Linux on any side - V4L2 is only used as a transport protocol, +and both sides are free to convert it from/to any model that they wish to use. + +\subsection{Device ID}\label{sec:Device Types / Media Device / Device ID} + +48 + +\subsection{Virtqueues}\label{sec:Device Types / Media Device / Virtqueues} + +\begin{description} +\item[0] commandq - used for driver commands and device responses to these +commands. +\item[1] eventq - used for events sent by the device to the driver. +\end{description} + +\subsection{Feature Bits}\label{sec:Device Types / Media Device / Feature Bits} + +None + +\subsection{Device Configuration Layout}\label{sec:Device Types / Media Device / Device Configuration Layout} + +The video device configuration space uses the following layout: + +\begin{lstlisting} +struct virtio_media_config { + le32 device_caps; + le32 device_type; + le8 card[32]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{device_caps}] (driver-read-only) flags representing the device +capabilities as used in +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/vidioc-querycap.html#c.V4L.v4l2_capability}{struct v4l2_capability}. +It corresponds with the \field{device_caps} field in the \textit{struct video_device}. +\item[\field{device_type}] (driver-read-only) informs the driver of the type +of the video device. It corresponds with the \field{vfl_devnode_type} field of the device. +\item[\field{card}] (driver-read-only) name of the device, a NUL-terminated +UTF-8 string. It corresponds with the \field{card} field of the \textit{struct v4l2_capability}.device-types/media/description.tex line 68
Quoted 10 lines
device-types/media/description.tex new file +10
-0+If all the characters of the field are used, it does not need to be NUL-terminated. +\end{description} + +\subsection{Device Initialization} + +\begin{enumerate} +\item Read the \field{device_caps} and \field{device_type} fields +from the configuration layout to identify the device. +\item Set up the \field{commandq} and \field{eventq}. +\item May open a session (see Section \ref{sec:Device Operation: Command Virtqueue: Sessions})This reference does not resolve.
Quoted 48 lines
device-types/media/description.tex new file +48
-0+to use the device and send V4L2 ioctls in order to receive more information +about the device, such as supported formats or controls. +\end{enumerate} + +\subsection{Device Operation}\label{sec:Device Types / Media Device / Device Operation} + +Commands are queued on the command queue by the driver for the device to +process. The errors returned by each command are standard +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/gen-errors.html}{Linux kernel error codes}. +For instance, a command that contains invalid options will return \textit{EINVAL}. + +Events are sent on the event queue by the device for the driver to handle. + +\subsubsection{Command Virtqueue} + +\paragraph{Device Operation: Command headers} + +\begin{lstlisting} +#define VIRTIO_MEDIA_CMD_OPEN 1 +#define VIRTIO_MEDIA_CMD_CLOSE 2 +#define VIRTIO_MEDIA_CMD_IOCTL 3 +#define VIRTIO_MEDIA_CMD_MMAP 4 +#define VIRTIO_MEDIA_CMD_MUNMAP 5 + +/* Header for all virtio commands from the driver to the device on the commandq. */ +struct virtio_media_cmd_header { + le32 cmd; + le32 __reserved; +}; + +/* Header for all virtio responses from the device to the driver on the commandq. */ +struct virtio_media_resp_header { + le32 status; + le32 __reserved; +}; +\end{lstlisting} + +A command consists of a command header \textit{virtio_media_cmd_header} +containing the following device-readable field: + +\begin{description} +\item[\field{cmd}] specifies a device request type (VIRTIO_MEDIA_CMD_*). +\end{description} + +A response consists of a response header \textit{virtio_media_resp_header} +containing the following device-writable field: + +\begin{description}device-types/media/description.tex line 126
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\item[\field{status}] indicates a device request status. +\end{description} + +The status field can take 0 if the command was successful, or one of the +standard Linux error codes if it was not. + +\drivernormative{\paragraph}{Device Operation: Command Virtqueue: Sessions}{Device Types / Media Device / Device Operation / Command Virtqueue} + +Sessions are how the device is multiplexed, allowing several distinct works to +take place simultaneously. Before start operating, the driver needs to open anit: "Before starting operation" maybe?
Quoted 51 lines
device-types/media/description.tex new file +51
-0+session. This is equivalent to opening the \textit{/dev/videoX} file of the +V4L2 device. Each session gets a unique ID assigned, which can be then used +to perform actions on it. + +\paragraph{Device Operation: Open device} + +\textbf{VIRTIO_MEDIA_CMD_OPEN} Command for creating a new session. + +This is the equivalent of calling \textit{open} on a V4L2 device node. +The driver uses \textit{virtio_media_cmd_open} to send an open request. + +\begin{lstlisting} +struct virtio_media_cmd_open { + struct virtio_media_cmd_header hdr; +}; +\end{lstlisting} + +The device responds to \textit{VIRTIO_MEDIA_CMD_OPEN} with \textit{virtio_media_resp_open}. + +\begin{lstlisting} +struct virtio_media_resp_open { + struct virtio_media_resp_header hdr; + le32 session_id; + le32 __reserved; +}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the current session, which is used for +other commands, predominantly ioctls. +\end{description} + +\devicenormative{\subparagraph}{Device Operation: Open device}{Device Types / Media Device / Device Operation / Open device} + +Upon success, the device MUST set a \field{session_id} in \textit{virtio_media_resp_open} +to an integer that is NOT used by any other open session. + +\paragraph{Device Operation: Close device} + +\textbf{VIRTIO_MEDIA_CMD_CLOSE} Command for closing an active session. + +This is the equivalent of calling \textit{close} on a previously opened V4L2 +device node. All resources associated with this session will be freed. + +This command does not require a response from the device. + +\begin{lstlisting} +struct virtio_media_cmd_close { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 __reserved;device-types/media/description.tex line 187
Quoted 10 lines
device-types/media/description.tex new file +10
-0+}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] specifies an identifier for the session to close. +\end{description} + +\drivernormative{\subparagraph}{Device Operation: Close device}{Device Types / Media Device / Device Operation / Close device} + +The session ID SHALL NOT be used again after queueing this command.... until it has been obtained again through a subsequent
VIRTIO_MEDIA_CMD_OPENcall. (session IDs can be recycled ; this should go without saying but it doesn't hurt to mention it)Quoted 5 lines
device-types/media/description.tex new file +5
-0+ +\paragraph{Device Operation: V4L2 ioctls} + +\textbf{VIRTIO_MEDIA_CMD_IOCTL} Command for executing an ioctl on an open +session.device-types/media/description.tex line 202
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +This command tells the device to run one of the `VIDIOC_*` ioctls on the +session identified by \textit{session_id}. + +\begin{lstlisting} +struct virtio_media_cmd_ioctl { + struct virtio_media_cmd_header hdr; + le32 session_id; + le32 code; + /* Followed by the relevant ioctl payload as defined in the macro */"ioctl command payload" maybe.
Quoted 22 lines
device-types/media/description.tex new file +22
-0+}; +\end{lstlisting} + +\begin{description} +\item[\field{session_id}] identifies the session to run the ioctl on. +\item[\field{code}] specifies the code of the \field{VIDIOC_*} ioctl to run. +\end{description} + +The code is extracted from the +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/videodev.html}{videodev2.h}, +header file. The file defines the ioctl's codes, type of payload, and +direction. The code consists of the second argument of the \field{_IO*} macro. + +For example, the \textit{VIDIOC_G_FMT} is defined as follows: + +\begin{lstlisting} +#define VIDIOC_G_FMT _IOWR('V', 4, struct v4l2_format) +\end{lstlisting} + +This means that its ioctl code is \textit{4}, its payload is a +\textit{struct v4l2_format}, and its direction is \textit{WR} (i.e., the +payload is written by both the driver and the device).Quoted 8 lines
device-types/media/description.tex line 229
Quoted 5 lines
device-types/media/description.tex new file +5
-0+See Section \ref{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} +for more information about the direction of ioctls. +Note that although most architectures use this format, there +are some architecture-specific encoding differences. +See \textit{include/ARCH/ioctl.h} for more details.Is the last sentence needed?
videodev2.his not architecture-specific, so I don't think this adds anything for our purpose?Quoted 5 lines
Probably not. The idea was to highlight that in some cases the payload may vary depending on the underlying architecture for some ioctls, due to these architecture differences.
But I agree that it is a detail that should not concern to those wanting to implement the specs. So I will remove the lines.Thanks. My concern was that reader would try to find reasons to care about architecture when all they need to do is look at the ioctls declarations from
videodev2.hand copy the second field.Quoted 43 lines
Quoted 43 lines
Quoted 4 lines
device-types/media/description.tex new file +2
-0+ +The payload struct layout always matches the 64-bit, little-endiandevice-types/media/description.tex line 241
Quoted 10 lines
device-types/media/description.tex new file +10
-0+representation of the corresponding V4L2 structure. For most structs, the +size is identical for both 32 and 64 bits versions. Otherwise, the driver +must translate them to the aforementioned size and endianess. + +The device responds to \textit{VIRTIO_MEDIA_CMD_IOCTL} with \textit{virtio_media_resp_ioctl}. + +\begin{lstlisting} +struct virtio_media_resp_ioctl { + struct virtio_media_resp_header hdr; + /* Followed by the ioctl payload as defined in the macro */Similarly, "ioctl response payload"?
Quoted 12 lines
device-types/media/description.tex new file +12
-0+}; +\end{lstlisting} + +\subparagraph{Ioctls payload}\label{sec:Device Types / Media Device / V4L2 ioctls / Ioctls payload} + +Each ioctl has a payload, which is defined by the third argument of the +\field{_IO*} macro. + +The payload of an ioctl in the descriptor chain follows the command structure, +the response structure, or both depending on the direction: + +\begin{itemize}device-types/media/description.tex line 263
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\item \textbf{_IOR} is read-only for the driver, meaning the payload +follows the response in the device-writable section of the descriptor chain. +\item \textbf{_IOW} is read-only for the device, meaning the payload +follows the command in the driver-writable section of the descriptor chain. +\item \textbf{_IOWR} is writable by both the device and driver, +meaning the payload must follow both the command in the driver-writable section +of the descriptor chain, and the response in the device-writable section. +\end{itemize} + +A common optimization for \textit{WR} ioctls is to provide the payload usings/common/possible? Unless this is indeed commonly used by other devices, but I doubt it.
Quoted 8 lines
Quoted 8 lines
device-types/media/description.tex line 266
Quoted 3 lines
device-types/media/description.tex new file +3
-0+descriptors that both point to the same buffer. This mimics the behavior of +V4L2 ioctls where the data is only passed once and used as both input and +output by the kernel.That's actually something I'd like to confirm - is it ok to have the device-readable and device-writable descriptors point to the same memory? I don't see why it wouldn't, but in case this is a problem we may want to change this and update the driver accordingly.
Thanks to Matias and Michael's confirmation it seems there is no problem doing this.
Quoted 50 lines
Quoted 38 lines
device-types/media/description.tex line 278
Quoted 13 lines
device-types/media/description.tex new file +12
-0+ +\devicenormative{\subparagraph}{Device Operation: V4L2 ioctls}{Device Types / Media Device / Device Operation / V4L2 ioctls} + +In case of success of a device-writable ioctl, the device MUST always write the +payload in the device-writable part of the descriptor chain. + +In case of failure of a device-writable ioctl, the device is free to write the +payload in the device-writable part of the descriptor chain or not. Some errors +may still result in the payload being updated, and in this case the device is +expected to write the updated payload. If the device has not written the +payload after an error (i.e., only the header is returned), the driver MUST +assume that the payload has not been modified.This last sentence sounds like it is driver-normative, not device-normative.
device-types/media/description.tex line 289
Quoted 11 lines
device-types/media/description.tex new file +11
-0+ +\subparagraph{Handling of pointers in ioctl payload} + +A few structures used as ioctl payloads contain pointers to further +data needed for the ioctl. There are notably: + +\begin{itemize} +\item The \field{planes} pointer of +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-buffer}{struct v4l2_buffer}, +which size is determined by the length member. +\item The \field{controls} pointer of \textit{struct v4l2_ext_controls}, whichSince
struct v4l2_bufferabove is a link, do we want to one forstruct v4l2_ext_controlsas well?device-types/media/description.tex line 293
Quoted 4 lines
device-types/media/description.tex new file +4
-0+size is determined by the count member. +\end{itemize} + +If the size of the pointed area is determined to be non-zero, then the mainnit: "If the size of the pointed area is non-zero, ..."
Quoted 7 lines
device-types/media/description.tex line 298
Quoted 5 lines
device-types/media/description.tex new file +5
-0+payload is immediately followed by the pointed data in their order of +appearance in the structure, and the pointer value itself is ignored by the +device, which must also return the value initially passed by the driver. + +\subparagraph{Handling of pointers to userspace memory}s/userspace/guest ?
To me this refers to the fact that the original fields in the related structures refer to userspace memory (outside of the host/guest context). I think I'd keep it as is.
Quoted 6 lines
Maybe changing the titles of this and the previous sections to:
\subparagraph{Handling of pointers to data in ioctl payload} and
\subparagraph{Handling of pointers to userspace memory in ioctl payload}wdyt?
Sounds good to me.
Quoted 10 lines
Quoted 5 lines
I'd also suggest adding a "(
SHARED_PAGESmemory type)" at the end of the title to make it explicit that this is specific to this particular memory type.Maybe a clarification in the first paragraph (and a link to the related section) would be enough? I feel having a memory type in the title when this concept has not been yet introduced may be confusing. Having that in the text allows to link the section and jump ahead to relate both concepts.
Agreed, a clarification in the text is probably better for providing context.
Quoted 124 lines
Quoted 17 lines
device-types/media/description.tex line 309
Quoted 13 lines
device-types/media/description.tex new file +11
-0+\label{sec:Device Types / Media Device / V4L2 ioctls / Userspace memory} + +A few pointers are special in that they point to userspace memory in the +original V4L2 specification. They are: + +\begin{itemize} +\item The \field{m.userptr} member of \textit{struct v4l2_buffer} and +\href{https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/buffer.html#struct-v4l2-plane}{struct v4l2_plane} +(technically an unsigned long, but designated a userspace address). +\item The \field{ptr} member of \textit{struct v4l2_ext_ctrl}. +\end{itemize}Same as above, let's either add links to all V4L2 structs, or just remove them altogether since we point to the documentation in the introduction anyway.
I added the link for your comment above, but I think with that change, this is ok as it is. I only add the link to those structs that have not been linked before.
Quoted 101 lines
Quoted 37 lines
device-types/media/description.tex new file +36
-0+ +These pointers can cover large areas of scattered memory, which has the +potential to require more descriptors than the virtio queue can provide. For +these particular pointers only, a list of \textit{struct virtio_media_sg_entry} +that covers the needed amount of memory for the pointer is used instead of +using descriptors to map the pointed memory directly. + +\begin{lstlisting} +struct virtio_media_sg_entry { + le64 start; + le32 len; + le32 __reserved; +}; +\end{lstlisting} + +For each such pointer to read, the device reads as many SG entries as needed +to cover the length of the pointed buffer, as described by its parent +structure (\field{length} member of \textit{struct v4l2_buffer} or +\textit{struct v4l2_plane} for buffer memory, and \field{size} member of +\textit{struct v4l2_ext_control} for control data). + +Since the device never needs to modify the list of SG entries, it is only +provided by the driver in the device-readable section of the descriptor chain, +and not repeated in the device-writable section, even for WR ioctls. + +\subparagraph{Unsupported ioctls} + +A few ioctls are replaced by other, more suitable mechanisms. + +\begin{itemize} +\item \textit{VIDIOC_QUERYCAP} is replaced by reading the configuration area +(see \ref{sec:Device Types / Media Device / Device Configuration Layout}). +\item \textit{VIDIOC_DQBUF} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Dequeue buffer}). +\item \textit{VIDIOC_DQEVENT} is replaced by a dedicated event +(see \ref{sec:Device Types / Media Device / Device Operation / Emit an event}).device-types/media/description.tex line 355
Quoted 10 lines
device-types/media/description.tex new file +10
-0+\item \textit{VIDIOC_G_JPEGCOMP} and \textit{VIDIOC_S_JPEGCOMP} are deprecated +and replaced by the controls of the JPEG class. +\item \textit{VIDIOC_LOG_STATUS} is a driver-only operation and shall not be +implemented by the device. +\end{itemize} + +\devicenormative{\subparagraph}{Device Operation: Unsupported ioctls}{Device Types / Media Device / Device Operation / Unsupported ioctls} + +When a request is not supported, the device MUST return \textit{ENOTTY}, +which corresponds with the response for unknown ioctls.s/with/to
Quoted 5 lines
device-types/media/description.tex new file +5
-0+ +\paragraph{Device Operation: Mapping a MMAP buffer} + +\textbf{VIRTIO_MEDIA_CMD_MMAP} Command for mapping a MMAP buffer into the +driver's address space.device-types/media/description.tex line 370
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ +Shared memory region ID 0 is used to map MMAP buffers with +the \textit{VIRTIO_MEDIA_CMD_MMAP} command. + +\begin{lstlisting} +#define VIRTIO_MEDIA_MMAP_FLAG_RW (1 << 0) + +struct virtio_media_cmd_mmap { + struct virtio_media_cmd_header hdr; + le32 session_id;
session_idis not mentioned in the member description below.Quoted 18 lines
device-types/media/description.tex new file +18
-0+ le32 flags; + le32 offset; +}; +\end{lstlisting} + +\begin{description} +\item[\field{flags}] is the set of flags for the mapping. \field{VIRTIO_MEDIA_MMAP_FLAG_RW} +can be set if a read-write mapping is desired. Without this flag the mapping +will be read-only. +\item[\field{offset}] corresponds to the \field{mem_offset} field of the +\textit{union v4l2_plane} for the plane to map. This field can be obtained +using the \textit{VIDIOC_QUERYBUF} ioctl. +\end{description} + +The device responds to \textit{VIRTIO_MEDIA_CMD_MMAP} with \textit{virtio_media_resp_mmap}. + +\begin{lstlisting} +struct virtio_media_resp_mmap {device-types/media/description.tex line 398
Quoted 10 lines
device-types/media/description.tex new file +10
-0+ struct virtio_media_resp_header hdr; + le64 driver_addr; + le64 len; +}; +\end{lstlisting} + +\begin{description} +\item[\field{driver_addr}] offset into SHM region ID 0 of the start of the mapping. +\item[\field{len}] length of the mapping as indicated by the \textit{struct v4l2_plane} +the buffer belongs to.This makes me think: do we need to return the length of the mapping when we already know the length of the
v4l2_planeand we only map whole buffers? It probably doesn't hurt to repeat that information here but maybe that's something we can optimize out?Quoted 7 lines
Quoted 4 lines
That's a good point. Indeed, no partial mapping should be allowed, so the length field does sound a bit redundant. Unless it needs error checking, which probably does not, I think it can be skipped. I'll do for the next version. Thanks!
Thanks - I will make sure to update the driver and devices as well to confirm this doesn't introduce any issue in practice.
So after trying to update the code to remove the length field, I have come to the conclusion that it is useful after all.
While it is true that the returned length of the mapping will always be equal to the length of the buffer, the semantics of mmap require that we check that the mapped area fits inside the requested VMA, and in order to do so we need the length of the mapping.
Why can't we use the length of the buffer, since it is the same value? The driver doesn't know that length until a QUERYBUF ioctl is performed, and although it is almost certain that the user will do a QUERYBUF -> mmap in practice (if only to obtain the offset to use as a mmap parameter), there is no formal guarantee of this, and doing so would require some caching mechanism that is a bit more complex than what I am comfortable putting in a kernel driver if it is not absolutely necessary.
Quoted 3 lines
So we should keep this
lenparameter, and maybe just add a device-normative paragraph stating that it must always be equal to the length of the buffer.
Got it. Will do that. Thanks for testing it.
Next week I will post a new version of the spec. But I feel it pretty mature already.
Best regards,
Albert.
Quoted 4 lines
Thanks,
Alex.