Skip to contentSkip to conversation

virtio-rtc: Add device specification.

RFC PATCH Superseded Peter Hilber 2 participants 20 messages Jun 30, 2023 – Oct 18, 2023

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

RFC PATCH v2 1/1 virtio-rtc: Add device specification.

This is a partial draft for a Real-Time Clock (RTC) device. The virtio-rtc device provides information about current time through one or more clocks.

v2 is a minor update which should match the Linux kernel driver patch series to be posted soon. It is planned to add a timer/alarm feature as part of a later draft spec update.

The draft uses the "Timer/Clock" device id which is already part of the specification. This device id was registered a long time ago and should be unused according to the author's information. The name "RTC" was determined to be the best for a device which focuses on current time, but is up to discussion.

For this device, there is a prototypical Linux kernel driver for which upstreaming is planned, and a proprietary device implementation.

In the spec text, there is some redundancy between the informative and the normative text. The intent is that the informative text already gives all the essential information about the device.

In the spec, clock ids are 64 bit wide to have unique ids when supporting clock hot-plugging in the future.

Any feedback is appreciated very much.

v2:

- Add VIRTIO_RTC_M_CROSS_CAP message [1].
- Fix various minor issues and improve wording [1].
- Add several clarifications regarding device error statuses.

[1] https://lists.oasis-open.org/archives/virtio-comment/202304/msg00523.html

---
 conformance.tex                         |   2 +
 content.tex                             |   3 +-
 device-types/rtc/description.tex        | 582 ++++++++++++++++++++++++
 device-types/rtc/device-conformance.tex |  10 +
 device-types/rtc/driver-conformance.tex |   9 +
 5 files changed, 605 insertions(+), 1 deletion(-)
 create mode 100644 device-types/rtc/description.tex
 create mode 100644 device-types/rtc/device-conformance.tex
 create mode 100644 device-types/rtc/driver-conformance.tex
conformance.tex +2 -0
index 01ccd693b245..423dc7af1e3c 100644
@@ -152,6 +152,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \input{device-types/scmi/driver-conformance.tex}
 \input{device-types/gpio/driver-conformance.tex}
 \input{device-types/pmem/driver-conformance.tex}
+\input{device-types/rtc/driver-conformance.tex}
 
 \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}
 
@@ -238,6 +239,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \input{device-types/scmi/device-conformance.tex}
 \input{device-types/gpio/device-conformance.tex}
 \input{device-types/pmem/device-conformance.tex}
+\input{device-types/rtc/device-conformance.tex}
 
 \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}
 A conformant implementation MUST be either transitional or
content.tex +2 -1
index d2ab9eb0d775..f5207a6916b9 100644
@@ -683,7 +683,7 @@ \chapter{Device Types}\label{sec:Device Types}
 \hline
 16         &   GPU device       \\
 \hline
-17         &   Timer/Clock device \\
+17         &   RTC (Timer/Clock) device \\
 \hline
 18         &   Input device \\
 \hline
@@ -765,6 +765,7 @@ \chapter{Device Types}\label{sec:Device Types}
 \input{device-types/scmi/description.tex}
 \input{device-types/gpio/description.tex}
 \input{device-types/pmem/description.tex}
+\input{device-types/rtc/description.tex}
 
 \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 
device-types/rtc/description.tex new file +582 -0

582 lines not on this page · raw message

device-types/rtc/device-conformance.tex new file +10 -0
new file mode 100644
index 000000000000..85fdcd3a4fba
@@ -0,0 +1,10 @@
+\conformance{\subsection}{RTC Device Conformance}\label{sec:Conformance / Device Conformance / RTC Device Conformance}
+
+An RTC device MUST conform to the following normative statements:
+
+\begin{itemize}
+\item \ref{devicenormative:Device Types / RTC Device / Feature bits}
+\item \ref{devicenormative:Device Types / RTC Device / Device Operation}
+\item \ref{devicenormative:Device Types / RTC Device / Device Operation / controlq Operation}
+\item \ref{devicenormative:Device Types / RTC Device / Device Operation / readq Operation}
+\end{itemize}
device-types/rtc/driver-conformance.tex new file +9 -0
new file mode 100644
index 000000000000..18948843e528
@@ -0,0 +1,9 @@
+\conformance{\subsection}{RTC Driver Conformance}\label{sec:Conformance / Driver Conformance / RTC Driver Conformance}
+
+An RTC driver MUST conform to the following normative statements:
+
+\begin{itemize}
+\item \ref{drivernormative:Device Types / RTC Device / Device Operation}
+\item \ref{drivernormative:Device Types / RTC Device / Device Operation / controlq Operation}
+\item \ref{drivernormative:Device Types / RTC Device / Device Operation / readq Operation}
+\end{itemize}

base-commit: 92295f3cb9479ce9fa5f440c3f34af17d45dab3b

Signature · 18 lines
2.39.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#1 lore raw
Quoted 7 lines

From: [email protected] <[email protected] open.org> On Behalf Of Peter Hilber
Sent: Friday, June 30, 2023 5:30 AM
Subject: [virtio-comment] [RFC PATCH v2] virtio-rtc: Add device specification.

Tiny nit: remove full stop from the subject, it usually does not have it.

Quoted 13 lines

This is a partial draft for a Real-Time Clock (RTC) device. The virtio-rtc device provides information about current time through one or more clocks.

v2 is a minor update which should match the Linux kernel driver patch series to be posted soon. It is planned to add a timer/alarm feature as part of a later draft spec update.

The draft uses the "Timer/Clock" device id which is already part of the specification. This device id was registered a long time ago and should be unused according to the author's information. The name "RTC" was determined to be the best for a device which focuses on current time, but is up to discussion.

Quoted 8 lines

For this device, there is a prototypical Linux kernel driver for which upstreaming is planned, and a proprietary device implementation.

In the spec text, there is some redundancy between the informative and the normative text. The intent is that the informative text already gives all the essential information about the device.

In the spec, clock ids are 64 bit wide to have unique ids when supporting clock

It is unclear the namespace of uniqueness.
Do you mean unique id across multiple virtio devices? If so, how is it proposed, we probably need UUID?

Quoted 4 lines

hot-plugging in the future.

Any feedback is appreciated very much.

This is a good feature.

A network device has a PHC which is very similar to what is being proposed here which is super set of this proposal.

A network device will support packet timestamping as phase-1, followed by PTP synchronized UTC/TAI format in phase-2 followed by PTP timestamping offload in phase-3. Phase-1 is already being proposed using monotonic clock method at [1].

[1] https://lists.oasis-open.org/archives/virtio-comment/202306/msg00287.html

We would like to have a single virtio network device to expose the optional RTC without creating a full blown new virtio device.

A quick grep in the industry indicates that at least 15+ nics in industry with dual PTP clock + nic functionality. And we would like virtio network device also to have it.
At minimum such solution avoids creating an additional full blown virtio device.

As proposed here, RTC device as stand alone itself has its merit and use.

So lets build the RTC device interface that can be of dual use.
1. as part of stand alone RTC device
2. as part of virtio network device

Such work will enable to re-invent the wheel 2nd time in virtio net.

What I can think of is following definitions being common between two device types. This will be able to leverage your rtc work done as part of this patchset.

1. _F feature bit that can be common between the rtc and net device
2. control vq command opcode and format be common
3. discovery of the readq id possibly using cvq and it must depend on _F_RESET bit.

device-types/rtc/description.tex line 7

Quoted 10 lines
device-types/rtc/description.tex new file +7 -0
@@ -0,0 +1,582 @@
+\section{RTC Device}\label{sec:Device Types / RTC Device}
+
+The RTC (Real Time Clock) device provides information about current
+time. The device can provide different clocks, e.g.\ for the UTC or TAI
+time standards, or for physical time elapsed since some past epoch.
+Drivers can read the clocks with simple or more accurate methods.
+

The driver can read...

I am not sure if I see two different methods to read (simple and accurate one). Is if the readq that is accurate one?

Quoted 4 lines
device-types/rtc/description.tex new file +4 -0
+\subsection{Device ID}\label{sec:Device Types / RTC Device / Device ID}
+
+17
+

device-types/rtc/description.tex

Quoted 10 lines
device-types/rtc/description.tex new file +10 -0
+\subsection{Virtqueues}\label{sec:Device Types / RTC Device /
+Virtqueues}
+
+\begin{description}
+\item[0] readq
+\item[1] controlq
+\end{description}
+
+The driver initiates clock readings through requests to the
+readq.

The driver initiates (or enqueues) clock reading requests to the readq. Does above read better?

\footnote{The readq is a dedicated virtqueue to facilitate

Quoted 7 lines
device-types/rtc/description.tex new file +7 -0
+reducing latency for clock readings, which may be more
+performance-sensitive than other requests.} The driver uses the
+controlq for other requests to the device.
+
+\subsection{Feature bits}\label{sec:Device Types / RTC Device / Feature
+bits}
+

device-types/rtc/description.tex line 36

Quoted 10 lines
device-types/rtc/description.tex new file +10 -0
+\begin{description}
+\item[VIRTIO_RTC_F_READ_CROSS (0)] Device supports cross-timestamping.
+\end{description}
+
+VIRTIO_RTC_F_READ_CROSS determines whether the device supports clock
+reading with cross-timestamping. Without this feature, the device does
+not support cross-timestamping. Even when this feature was negotiated,
+the device may still not support cross-timestamping for some clocks, or
+for particular requests.
+

Can you please explain why even after feature negotiation, device may still not support? Is it because the cross timestamping is per clock feature?
If so, I guess no need of a feature bit as it is not serving any meaningful purpose. Each clock information can advertise what is supports, right?

Quoted 15 lines
device-types/rtc/description.tex new file +15 -0
+\devicenormative{\subsubsection}{Feature bits}{Device Types / RTC
+Device / Feature bits}
+
+The device SHOULD offer VIRTIO_RTC_F_READ_CROSS if the device can
+support cross-timestamping for any of its clocks.
+
+\subsection{Device configuration layout}\label{sec:Device Types / RTC
+Device / Device configuration layout}
+
+There is no configuration data for the device.
+
+\subsection{Device Initialization}\label{sec:Device Types / RTC Device
+/ Device Initialization}
+
+The device determines the set of clocks. The device can provide zero or

device-types/rtc/description.tex line 57

Quoted 10 lines
device-types/rtc/description.tex new file +10 -0
+more clocks.
+
+\subsection{Device Operation}\label{sec:Device Types / RTC Device /
+Device Operation}
+
+For the readq and the controlq, the driver sends a message with a
+request, and receives the response in the device-writable part of the
+message. readq and controlq messages use common request and response
+headers.
+

Below structure definition is _req and response.
So lets just call it request and request_type as its simple to explain using single thing. Please drop the wording around message.

Quoted 18 lines
device-types/rtc/description.tex new file +18 -0
+\begin{lstlisting}
+/* common request header */
+struct virtio_rtc_req_head {
+	__le16 msg_type;
+	__u8 reserved[2];
+};
+
+/* common response header */
+struct virtio_rtc_resp_head {
+	__u8 status;
+	__u8 reserved[3];
+};
+\end{lstlisting}
+
+The \field{msg_type} field identifies the message type.
+
+The \field{status} field indicates whether the device successfully
+executed the request. The device sets the \field{status} field to one

device-types/rtc/description.tex line 85

Quoted 10 lines
device-types/rtc/description.tex new file +10 -0
+of the following values:
+
+\begin{lstlisting}
+#define VIRTIO_RTC_S_OK 0
+#define VIRTIO_RTC_S_UNSUPP 1
+#define VIRTIO_RTC_S_NODEV 2
+#define VIRTIO_RTC_S_INVAL 3
+#define VIRTIO_RTC_S_DEVERR 4
+\end{lstlisting}
+

Recently for the AQ we picked Linux UAPI specific error codes such as EAGAIN, EINVAL etc. You might want to have similar code reused for EINVAL, NOTSUPPORT, NODEV to be reused here. No strong opinion though.

[..]
When the device and driver normative lines are long like this, it is best to put in separate patch, so that its easy to read (review) through the whole description and conformance as separate.

device-types/rtc/description.tex line 225

Quoted 5 lines
device-types/rtc/description.tex new file +5 -0
+\item[VIRTIO_RTC_M_CFG] discovers the number of clocks.
+
+\begin{lstlisting}
+#define VIRTIO_RTC_M_CFG 0x1000 /* message type */
+

Like above, lets just call it RTC_REQ_CFG as below is a request structure.

Quoted 18 lines
device-types/rtc/description.tex new file +18 -0
+struct virtio_rtc_req_cfg {
+	struct virtio_rtc_req_head head;
+	/* no request params */
+	__u8 reserved[4];
+};
+
+struct virtio_rtc_resp_cfg {
+	struct virtio_rtc_resp_head head;
+	__le16 num_clocks;
+	__u8 reserved[10];
+};
+\end{lstlisting}
+
+Field \field{num_clocks} contains the number of clocks. A device
+provides zero or more clocks. Valid clock ids are those smaller than
+\field{num_clocks}.
+
+\item[VIRTIO_RTC_M_CLOCK_CAP] discovers the capabilities of the clock

device-types/rtc/description.tex line 253

Quoted 10 lines
device-types/rtc/description.tex new file +10 -0
+identified by the \field{clock_id} field.
+
+\begin{lstlisting}
+#define VIRTIO_RTC_M_CLOCK_CAP 0x1001 /* message type */
+
+struct virtio_rtc_req_clock_cap {
+	struct virtio_rtc_req_head head;
+	__u8 reserved[4];
+	__le64 clock_id;
+};

Please drop "__" from the field data type definition.

Quoted 117 lines
device-types/rtc/description.tex new file +117 -0
+
+struct virtio_rtc_resp_clock_cap {
+	struct virtio_rtc_resp_head head;
+	__le16 type;
+	__u8 reserved[10];
+};
+\end{lstlisting}
+
+The \field{type} field identifies the clock type. A device provides
+zero or more clocks for a clock type. The following clock types are
+defined:
+
+\begin{lstlisting}
+#define VIRTIO_RTC_CLOCK_UTC 0
+#define VIRTIO_RTC_CLOCK_TAI 1
+#define VIRTIO_RTC_CLOCK_MONO 2
+\end{lstlisting}
+
+VIRTIO_RTC_CLOCK_UTC uses the UTC (Coordinated Universal Time) time
+standard. The device uses the time epoch of January 1, 1970, 00:00 UTC.
+This is the same epoch as \emph{Unix time}.
+
+VIRTIO_RTC_CLOCK_TAI uses the TAI (International Atomic Time) time
+standard. The device uses the time epoch of January 1, 1970, 00:00 TAI.
+
+VIRTIO_RTC_CLOCK_MONO uses monotonic physical time (SI seconds
+subdivisions) since some unspecified epoch. The VIRTIO_RTC_CLOCK_MONO
+epoch is before or during device reset.
+
+Additional clock types may be standardized in the future.
+Implementation-specific clock type definitions are not recommended and
+use a clock type id greater than or equal to 0xF000.
+
+\item[VIRTIO_RTC_M_CROSS_CAP] discovers whether the device supports
+cross-timestamping for a particular pair of clock and hardware counter.
+If the VIRTIO_RTC_F_READ_CROSS feature has been negotiated, the driver
+can send the VIRTIO_RTC_M_CROSS_CAP message.
+
+\begin{lstlisting}
+#define VIRTIO_RTC_M_CROSS_CAP 0x1002 /* message type */
+
+struct virtio_rtc_req_cross_cap {
+	struct virtio_rtc_req_head head;
+	__le16 hw_counter;
+	__u8 reserved[2];
+	__le64 clock_id;
+};
+
+#define VIRTIO_RTC_FLAG_CROSS_CAP 0
+
+struct virtio_rtc_resp_cross_cap {
+	struct virtio_rtc_resp_head head;
+	__u8 flags;
+	__u8 reserved[11];
+};
+\end{lstlisting}
+
+The \field{clock_id} field identifies the clock, and the
+\field{hw_counter} field identifies the hardware counter, for which
+cross-timestamp support is probed. The device sets bit
+\field{VIRTIO_RTC_FLAG_CROSS_CAP} in the \field{flags} field if the
+clock supports cross-timestamping for the particular clock and hardware
+counter, and clears the bit otherwise.
+
+\end{description}
+
+\devicenormative{\paragraph}{controlq Operation}{Device Types / RTC
+Device / Device Operation / controlq Operation}
+
+For a fixed controlq message request value V, the device SHOULD either,
+for every request with value V, always execute successfully, or, for
+every request with value V, always set a fixed \field{status} other
+than VIRTIO_RTC_S_OK.
+
+For any clock of type VIRTIO_RTC_CLOCK_UTC, the device MUST use the UTC
+time standard (Coordinated Universal Time).
+
+For any clock of type VIRTIO_RTC_CLOCK_UTC, the device MUST use the
+time epoch of January 1, 1970, 00:00 UTC.
+
+For any clock of type VIRTIO_RTC_CLOCK_TAI, the device MUST use the TAI
+time standard (International Atomic Time).
+
+For any clock of type VIRTIO_RTC_CLOCK_TAI, the device MUST use the
+time epoch of January 1, 1970, 00:00 TAI.
+
+For any clock of type VIRTIO_RTC_CLOCK_MONO, the device MUST use SI
+seconds subdivisions.
+
+For any clock of type VIRTIO_RTC_CLOCK_MONO, the device MUST use an
+epoch at a time instant before or during device reset.
+
+For message VIRTIO_RTC_M_CLOCK_CAP, the device MUST set \field{type} to
+one of the clock types defined in this specification, or to a value
+greater than or equal to 0xF000.
+
+For a VIRTIO_RTC_M_CROSS_CAP message M, the device MUST set the
+VIRTIO_RTC_FLAG_CROSS_CAP bit in the \field{flags} field if the device
+would respond to a VIRTIO_RTC_M_READ_CROSS message with the same
+\field{hw_counter} and \field{clock_id} values as in M with status
+VIRTIO_RTC_S_OK, and clear the bit otherwise.
+
+\drivernormative{\paragraph}{controlq Operation}{Device Types / RTC
+Device / Device Operation / controlq Operation}
+
+The driver MUST put the request into the device-readable part of the
+controlq message.
+
+The driver MUST allocate enough space for the response in the
+device-writable part of the controlq message.
+
+The driver MUST negotiate VIRTIO_RTC_F_READ_CROSS before sending a
+VIRTIO_RTC_M_CROSS_CAP message.
+
+For VIRTIO_RTC_M_CROSS_CAP, the driver MUST set \field{hw_counter} to
+one of the hardware counter identifiers defined in this specification,
+or to a value greater than or equal to 0xF000.

device-types/rtc/description.tex line 378

Quoted 10 lines
device-types/rtc/description.tex new file +10 -0
+
+\subsubsection{readq Operation}\label{sec:Device Types / RTC Device /
+Device Operation / readq Operation}
+
+The driver requests clock readings through readq messages. The device
+obtains device-side clock readings and forwards these clock readings to
+the driver. The driver may enhance and interpret the clock readings
+through methods which are beyond the scope of this specification.
+
+Once DRIVER_OK has been set, the device should continuously support

The word "continuously" is likely not needed.
As long as readq is created, driver can read it.

Quoted 38 lines
device-types/rtc/description.tex new file +38 -0
+reading every clock, even when a clock may yet have to be aligned to
+reference time sources.
+
+In general,
+
+\begin{itemize}
+\item clocks may jump backwards or forward, and \item the clock
+frequency may change. Clocks may be \emph{slewed},
+	i.e.\ clocks may run at a frequency other than their current
+	best frequency estimate.
+\end{itemize}
+
+E.g., a clock of type VIRTIO_RTC_CLOCK_UTC can insert UTC leap seconds.
+
+As long as a clock does not jump backwards, the driver clock readings
+increase monotonically:
+
+\begin{itemize}
+\item As long as a clock does not jump backwards in-between device-side
+	clock readings, the driver-side readings for that clock increase
+	monotonically as well, in the order in which the driver
+	marks readq messages as available.
+
+\item The device marks readq messages for the same clock as used in
+	the order in which the messages were marked as available.
+\end{itemize}
+
+For a clock of type VIRTIO_RTC_CLOCK_MONO, the device always returns
+monotonically increasing clock readings.
+
+The unit of all \field{clock_reading} fields is 1 nanosecond for all
+clock types which are part of the device specification so
+far.\footnote{For time epochs in year 1970 or later, this means that
+time until at least year 2553 can be represented in the \field{__le64
+clock_reading} fields.}
+
+\begin{description}
+

device-types/rtc/description.tex line 426

Quoted 10 lines
device-types/rtc/description.tex new file +10 -0
+\item[VIRTIO_RTC_M_READ] reads the clock identified by the
+\field{clock_id} field. The device supports this message for every
+clock.
+
+\begin{lstlisting}
+#define VIRTIO_RTC_M_READ 0x0001 /* message type */
+
+struct virtio_rtc_req_read {
+	struct virtio_rtc_req_head head;
+	__u8 reserved[4];

I think the req_head structure to be 64-bit aligned, so that here individual requests does not have to align it.

device-types/rtc/description.tex line 439

Quoted 13 lines
device-types/rtc/description.tex new file +13 -0
+	__le64 clock_id;
+};
+
+struct virtio_rtc_resp_read {
+	struct virtio_rtc_resp_head head;
+	__u8 reserved[4];
+	__le64 clock_reading;
+};
+\end{lstlisting}
+
+\field{clock_reading} is a device-side clock reading obtained after the
+message was marked as available.
+

If this clock device is a PCI device, on an empty system, a PCI device has to do at least couple of DMA.
1. read the descriptor (250 to 300nsec)
2. Read the request (250 to 300nsec)
3. Write the response (50 nsec+)
4. raise interrupt.

How is this time delta accounted in clock reading?
Is driver expected to record system time and subtract it on receiving response? Or?

Ran out of time.
Will continue after I get above base line alignment.

Thanks.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#2 lore raw
Quoted 7 lines · Parav Pandit
Quoted 5 lines

From: [email protected] <[email protected] open.org> On Behalf Of Peter Hilber
Sent: Friday, June 30, 2023 5:30 AM
Subject: [virtio-comment] [RFC PATCH v2] virtio-rtc: Add device specification.

Tiny nit: remove full stop from the subject, it usually does not have it.

OK.

Quoted 16 lines
Quoted 16 lines

This is a partial draft for a Real-Time Clock (RTC) device. The virtio-rtc device provides information about current time through one or more clocks.

v2 is a minor update which should match the Linux kernel driver patch series to be posted soon. It is planned to add a timer/alarm feature as part of a later draft spec update.

The draft uses the "Timer/Clock" device id which is already part of the specification. This device id was registered a long time ago and should be unused according to the author's information. The name "RTC" was determined to be the best for a device which focuses on current time, but is up to discussion.

For this device, there is a prototypical Linux kernel driver for which upstreaming is planned, and a proprietary device implementation.

Quoted 7 lines
Quoted 5 lines

In the spec text, there is some redundancy between the informative and the normative text. The intent is that the informative text already gives all the essential information about the device.

In the spec, clock ids are 64 bit wide to have unique ids when supporting clock

It is unclear the namespace of uniqueness.
Do you mean unique id across multiple virtio devices?

The idea was not to have a unique id across multiple virtio devices. Rather, the idea was to simplify lifetime management for clocks in a single virtio device (if such lifetimes would be introduced in the future) by not having to worry about race conditions due to clock id reuse.

Quoted 2 lines

If so, how is it proposed, we probably need UUID?

So it would be up to the device implementation to assign unique ids for each clock lifetime; incrementing a counter should do.

Quoted 19 lines
Quoted 4 lines

hot-plugging in the future.

Any feedback is appreciated very much.

This is a good feature.

A network device has a PHC which is very similar to what is being proposed here which is super set of this proposal.

A network device will support packet timestamping as phase-1, followed by PTP synchronized UTC/TAI format in phase-2 followed by PTP timestamping offload in phase-3. Phase-1 is already being proposed using monotonic clock method at [1].

[1] https://lists.oasis-open.org/archives/virtio-comment/202306/msg00287.html

We would like to have a single virtio network device to expose the optional RTC without creating a full blown new virtio device.

A quick grep in the industry indicates that at least 15+ nics in industry with dual PTP clock + nic functionality. And we would like virtio network device also to have it.
At minimum such solution avoids creating an additional full blown virtio device.

Quoted 6 lines

As proposed here, RTC device as stand alone itself has its merit and use.

So lets build the RTC device interface that can be of dual use.
1. as part of stand alone RTC device
2. as part of virtio network device

To me it seems preferable to first try to keep the two devices separated. AFAIU PTP clocks are also usually represented as a device separate from network devices.

The extra effort to implement another simple virtio device (rather than putting everything in a single device) seems to be small when compared to the complexity of creating an extensible specification which should work both standalone and as part of an already existing, but still growing, device. E.g., the RTC device should also get an alarm, with an associated eventq, in the future. Also, I wonder if the driver implementation should then work both as a standalone Virtio driver and as part of the net driver.

Maybe the above use case could also be addressed by a standalone RTC device as follows:

1) The net device exports a UUID object to which the RTC device can refer to (so the drivers can link the net device to the RTC clock).

2) If needed, the RTC device aligns the lifetime of the clock to the net device. Then, clock lifetimes are no longer static, and can be aligned with the net device. Through an eventq (to be added), the RTC device announces clock creation/destruction, which can be synchronized to the net device state.

Quoted 9 lines

Such work will enable to re-invent the wheel 2nd time in virtio net.

What I can think of is following definitions being common between two device types. This will be able to leverage your rtc work done as part of this patchset.

1. _F feature bit that can be common between the rtc and net device
2. control vq command opcode and format be common
3. discovery of the readq id possibly using cvq and it must depend on _F_RESET bit.

Not sure what _F_RESET refers to, is it VIRTIO_F_RING_RESET? And why?

Quoted 12 lines

device-types/rtc/description.tex line 7

Quoted 10 lines
device-types/rtc/description.tex new file +7 -0
@@ -0,0 +1,582 @@
+\section{RTC Device}\label{sec:Device Types / RTC Device}
+
+The RTC (Real Time Clock) device provides information about current
+time. The device can provide different clocks, e.g.\ for the UTC or TAI
+time standards, or for physical time elapsed since some past epoch.
+Drivers can read the clocks with simple or more accurate methods.
+

The driver can read...

I assume this is a proposal to rephrase, I can change it like this.

Quoted 3 lines

I am not sure if I see two different methods to read (simple and accurate one). Is if the readq that is accurate one?

The readq is the only vqueue through which the clock can be read, through VIRTIO_RTC_M_READ (imprecise) and VIRTIO_RTC_M_READ_CROSS (allows for precise reading).

Quoted 8 lines
Quoted 8 lines
device-types/rtc/description.tex new file +8 -0
+\subsection{Device ID}\label{sec:Device Types / RTC Device / Device ID}
+
+17
+
+\subsection{Virtqueues}\label{sec:Device Types / RTC Device /
+Virtqueues}
+
+\begin{description}
Quoted 9 lines

device-types/rtc/description.tex

Quoted 6 lines
device-types/rtc/description.tex new file +6 -0
+\item[0] readq
+\item[1] controlq
+\end{description}
+
+The driver initiates clock readings through requests to the
+readq.

The driver initiates (or enqueues) clock reading requests to the readq. Does above read better?

If this sounds better, I could rephrase to "The driver enqueues clock reading requests to the readq."

Quoted 12 lines

\footnote{The readq is a dedicated virtqueue to facilitate

Quoted 11 lines
device-types/rtc/description.tex new file +11 -0
+reducing latency for clock readings, which may be more
+performance-sensitive than other requests.} The driver uses the
+controlq for other requests to the device.
+
+\subsection{Feature bits}\label{sec:Device Types / RTC Device / Feature
+bits}
+
+\begin{description}
+\item[VIRTIO_RTC_F_READ_CROSS (0)] Device supports cross-timestamping.
+\end{description}
+
Quoted 8 lines

device-types/rtc/description.tex line 36

Quoted 6 lines
device-types/rtc/description.tex new file +6 -0
+VIRTIO_RTC_F_READ_CROSS determines whether the device supports clock
+reading with cross-timestamping. Without this feature, the device does
+not support cross-timestamping. Even when this feature was negotiated,
+the device may still not support cross-timestamping for some clocks, or
+for particular requests.
+

Can you please explain why even after feature negotiation, device may still not support? Is it because the cross timestamping is per clock feature?

Actually, the cross-timestamping support can be specific to the combination of clock and hardware counter. This is explained in more detail in the context of VIRTIO_RTC_M_READ_CROSS. I will mention this here as well.

Quoted 1 line

If so, I guess no need of a feature bit as it is not serving any meaningful purpose.

The only purpose is to potentially save the device some initialization work or resources in case the feature is not negotiated.

Quoted 2 lines

Each clock information can advertise what is supports, right?

The driver can use VIRTIO_RTC_M_CROSS_CAP to inquire about this (or the driver could also just try if VIRTIO_RTC_M_READ_CROSS works).

Quoted 20 lines
Quoted 20 lines
device-types/rtc/description.tex new file +20 -0
+\devicenormative{\subsubsection}{Feature bits}{Device Types / RTC
+Device / Feature bits}
+
+The device SHOULD offer VIRTIO_RTC_F_READ_CROSS if the device can
+support cross-timestamping for any of its clocks.
+
+\subsection{Device configuration layout}\label{sec:Device Types / RTC
+Device / Device configuration layout}
+
+There is no configuration data for the device.
+
+\subsection{Device Initialization}\label{sec:Device Types / RTC Device
+/ Device Initialization}
+
+The device determines the set of clocks. The device can provide zero or
+more clocks.
+
+\subsection{Device Operation}\label{sec:Device Types / RTC Device /
+Device Operation}
+
Quoted 9 lines

device-types/rtc/description.tex line 57

Quoted 5 lines
device-types/rtc/description.tex new file +5 -0
+For the readq and the controlq, the driver sends a message with a
+request, and receives the response in the device-writable part of the
+message. readq and controlq messages use common request and response
+headers.
+

Below structure definition is _req and response.
So lets just call it request and request_type as its simple to explain using single thing. Please drop the wording around message.

OK.

Quoted 23 lines
Quoted 23 lines
device-types/rtc/description.tex new file +23 -0
+\begin{lstlisting}
+/* common request header */
+struct virtio_rtc_req_head {
+	__le16 msg_type;
+	__u8 reserved[2];
+};
+
+/* common response header */
+struct virtio_rtc_resp_head {
+	__u8 status;
+	__u8 reserved[3];
+};
+\end{lstlisting}
+
+The \field{msg_type} field identifies the message type.
+
+The \field{status} field indicates whether the device successfully
+executed the request. The device sets the \field{status} field to one
+of the following values:
+
+\begin{lstlisting}
+#define VIRTIO_RTC_S_OK 0
+#define VIRTIO_RTC_S_UNSUPP 1
Quoted 8 lines

device-types/rtc/description.tex line 85

Quoted 5 lines
device-types/rtc/description.tex new file +5 -0
+#define VIRTIO_RTC_S_NODEV 2
+#define VIRTIO_RTC_S_INVAL 3
+#define VIRTIO_RTC_S_DEVERR 4
+\end{lstlisting}
+

Recently for the AQ we picked Linux UAPI specific error codes such as EAGAIN, EINVAL etc. You might want to have similar code reused for EINVAL, NOTSUPPORT, NODEV to be reused here. No strong opinion though.

I think I will change as suggested.

Quoted 4 lines

[..]
When the device and driver normative lines are long like this, it is best to put in separate patch, so that its easy to read (review) through the whole description and conformance as separate.

I will separate non-normative and normative parts in the next iteration.

Quoted 7 lines

device-types/rtc/description.tex line 225

Quoted 5 lines
device-types/rtc/description.tex new file +5 -0
+\item[VIRTIO_RTC_M_CFG] discovers the number of clocks.
+
+\begin{lstlisting}
+#define VIRTIO_RTC_M_CFG 0x1000 /* message type */
+

Like above, lets just call it RTC_REQ_CFG as below is a request structure.

OK.

Quoted 21 lines
Quoted 21 lines
device-types/rtc/description.tex new file +21 -0
+struct virtio_rtc_req_cfg {
+	struct virtio_rtc_req_head head;
+	/* no request params */
+	__u8 reserved[4];
+};
+
+struct virtio_rtc_resp_cfg {
+	struct virtio_rtc_resp_head head;
+	__le16 num_clocks;
+	__u8 reserved[10];
+};
+\end{lstlisting}
+
+Field \field{num_clocks} contains the number of clocks. A device
+provides zero or more clocks. Valid clock ids are those smaller than
+\field{num_clocks}.
+
+\item[VIRTIO_RTC_M_CLOCK_CAP] discovers the capabilities of the clock
+identified by the \field{clock_id} field.
+
+\begin{lstlisting}
Quoted 9 lines

device-types/rtc/description.tex line 253

Quoted 7 lines
device-types/rtc/description.tex new file +7 -0
+#define VIRTIO_RTC_M_CLOCK_CAP 0x1001 /* message type */
+
+struct virtio_rtc_req_clock_cap {
+	struct virtio_rtc_req_head head;
+	__u8 reserved[4];
+	__le64 clock_id;
+};

Please drop "__" from the field data type definition.

I will fix this.

Quoted 122 lines
Quoted 122 lines
device-types/rtc/description.tex new file +122 -0
+
+struct virtio_rtc_resp_clock_cap {
+	struct virtio_rtc_resp_head head;
+	__le16 type;
+	__u8 reserved[10];
+};
+\end{lstlisting}
+
+The \field{type} field identifies the clock type. A device provides
+zero or more clocks for a clock type. The following clock types are
+defined:
+
+\begin{lstlisting}
+#define VIRTIO_RTC_CLOCK_UTC 0
+#define VIRTIO_RTC_CLOCK_TAI 1
+#define VIRTIO_RTC_CLOCK_MONO 2
+\end{lstlisting}
+
+VIRTIO_RTC_CLOCK_UTC uses the UTC (Coordinated Universal Time) time
+standard. The device uses the time epoch of January 1, 1970, 00:00 UTC.
+This is the same epoch as \emph{Unix time}.
+
+VIRTIO_RTC_CLOCK_TAI uses the TAI (International Atomic Time) time
+standard. The device uses the time epoch of January 1, 1970, 00:00 TAI.
+
+VIRTIO_RTC_CLOCK_MONO uses monotonic physical time (SI seconds
+subdivisions) since some unspecified epoch. The VIRTIO_RTC_CLOCK_MONO
+epoch is before or during device reset.
+
+Additional clock types may be standardized in the future.
+Implementation-specific clock type definitions are not recommended and
+use a clock type id greater than or equal to 0xF000.
+
+\item[VIRTIO_RTC_M_CROSS_CAP] discovers whether the device supports
+cross-timestamping for a particular pair of clock and hardware counter.
+If the VIRTIO_RTC_F_READ_CROSS feature has been negotiated, the driver
+can send the VIRTIO_RTC_M_CROSS_CAP message.
+
+\begin{lstlisting}
+#define VIRTIO_RTC_M_CROSS_CAP 0x1002 /* message type */
+
+struct virtio_rtc_req_cross_cap {
+	struct virtio_rtc_req_head head;
+	__le16 hw_counter;
+	__u8 reserved[2];
+	__le64 clock_id;
+};
+
+#define VIRTIO_RTC_FLAG_CROSS_CAP 0
+
+struct virtio_rtc_resp_cross_cap {
+	struct virtio_rtc_resp_head head;
+	__u8 flags;
+	__u8 reserved[11];
+};
+\end{lstlisting}
+
+The \field{clock_id} field identifies the clock, and the
+\field{hw_counter} field identifies the hardware counter, for which
+cross-timestamp support is probed. The device sets bit
+\field{VIRTIO_RTC_FLAG_CROSS_CAP} in the \field{flags} field if the
+clock supports cross-timestamping for the particular clock and hardware
+counter, and clears the bit otherwise.
+
+\end{description}
+
+\devicenormative{\paragraph}{controlq Operation}{Device Types / RTC
+Device / Device Operation / controlq Operation}
+
+For a fixed controlq message request value V, the device SHOULD either,
+for every request with value V, always execute successfully, or, for
+every request with value V, always set a fixed \field{status} other
+than VIRTIO_RTC_S_OK.
+
+For any clock of type VIRTIO_RTC_CLOCK_UTC, the device MUST use the UTC
+time standard (Coordinated Universal Time).
+
+For any clock of type VIRTIO_RTC_CLOCK_UTC, the device MUST use the
+time epoch of January 1, 1970, 00:00 UTC.
+
+For any clock of type VIRTIO_RTC_CLOCK_TAI, the device MUST use the TAI
+time standard (International Atomic Time).
+
+For any clock of type VIRTIO_RTC_CLOCK_TAI, the device MUST use the
+time epoch of January 1, 1970, 00:00 TAI.
+
+For any clock of type VIRTIO_RTC_CLOCK_MONO, the device MUST use SI
+seconds subdivisions.
+
+For any clock of type VIRTIO_RTC_CLOCK_MONO, the device MUST use an
+epoch at a time instant before or during device reset.
+
+For message VIRTIO_RTC_M_CLOCK_CAP, the device MUST set \field{type} to
+one of the clock types defined in this specification, or to a value
+greater than or equal to 0xF000.
+
+For a VIRTIO_RTC_M_CROSS_CAP message M, the device MUST set the
+VIRTIO_RTC_FLAG_CROSS_CAP bit in the \field{flags} field if the device
+would respond to a VIRTIO_RTC_M_READ_CROSS message with the same
+\field{hw_counter} and \field{clock_id} values as in M with status
+VIRTIO_RTC_S_OK, and clear the bit otherwise.
+
+\drivernormative{\paragraph}{controlq Operation}{Device Types / RTC
+Device / Device Operation / controlq Operation}
+
+The driver MUST put the request into the device-readable part of the
+controlq message.
+
+The driver MUST allocate enough space for the response in the
+device-writable part of the controlq message.
+
+The driver MUST negotiate VIRTIO_RTC_F_READ_CROSS before sending a
+VIRTIO_RTC_M_CROSS_CAP message.
+
+For VIRTIO_RTC_M_CROSS_CAP, the driver MUST set \field{hw_counter} to
+one of the hardware counter identifiers defined in this specification,
+or to a value greater than or equal to 0xF000.
+
+\subsubsection{readq Operation}\label{sec:Device Types / RTC Device /
+Device Operation / readq Operation}
+
+The driver requests clock readings through readq messages. The device
Quoted 9 lines

device-types/rtc/description.tex line 378

Quoted 5 lines
device-types/rtc/description.tex new file +5 -0
+obtains device-side clock readings and forwards these clock readings to
+the driver. The driver may enhance and interpret the clock readings
+through methods which are beyond the scope of this specification.
+
+Once DRIVER_OK has been set, the device should continuously support

The word "continuously" is likely not needed.
As long as readq is created, driver can read it.

I will drop "continuously".

Quoted 42 lines
Quoted 42 lines
device-types/rtc/description.tex new file +42 -0
+reading every clock, even when a clock may yet have to be aligned to
+reference time sources.
+
+In general,
+
+\begin{itemize}
+\item clocks may jump backwards or forward, and \item the clock
+frequency may change. Clocks may be \emph{slewed},
+	i.e.\ clocks may run at a frequency other than their current
+	best frequency estimate.
+\end{itemize}
+
+E.g., a clock of type VIRTIO_RTC_CLOCK_UTC can insert UTC leap seconds.
+
+As long as a clock does not jump backwards, the driver clock readings
+increase monotonically:
+
+\begin{itemize}
+\item As long as a clock does not jump backwards in-between device-side
+	clock readings, the driver-side readings for that clock increase
+	monotonically as well, in the order in which the driver
+	marks readq messages as available.
+
+\item The device marks readq messages for the same clock as used in
+	the order in which the messages were marked as available.
+\end{itemize}
+
+For a clock of type VIRTIO_RTC_CLOCK_MONO, the device always returns
+monotonically increasing clock readings.
+
+The unit of all \field{clock_reading} fields is 1 nanosecond for all
+clock types which are part of the device specification so
+far.\footnote{For time epochs in year 1970 or later, this means that
+time until at least year 2553 can be represented in the \field{__le64
+clock_reading} fields.}
+
+\begin{description}
+
+\item[VIRTIO_RTC_M_READ] reads the clock identified by the
+\field{clock_id} field. The device supports this message for every
+clock.
+
Quoted 9 lines

device-types/rtc/description.tex line 426

Quoted 6 lines
device-types/rtc/description.tex new file +6 -0
+\begin{lstlisting}
+#define VIRTIO_RTC_M_READ 0x0001 /* message type */
+
+struct virtio_rtc_req_read {
+	struct virtio_rtc_req_head head;
+	__u8 reserved[4];

I think the req_head structure to be 64-bit aligned, so that here individual requests does not have to align it.

I will pad virtio_rtc_req_head and virtio_rtc_resp_head to 64 bit.

Quoted 19 lines

device-types/rtc/description.tex line 439

Quoted 13 lines
device-types/rtc/description.tex new file +13 -0
+	__le64 clock_id;
+};
+
+struct virtio_rtc_resp_read {
+	struct virtio_rtc_resp_head head;
+	__u8 reserved[4];
+	__le64 clock_reading;
+};
+\end{lstlisting}
+
+\field{clock_reading} is a device-side clock reading obtained after the
+message was marked as available.
+

If this clock device is a PCI device, on an empty system, a PCI device has to do at least couple of DMA.
1. read the descriptor (250 to 300nsec)
2. Read the request (250 to 300nsec)
3. Write the response (50 nsec+)
4. raise interrupt.

Quoted 3 lines

How is this time delta accounted in clock reading?
Is driver expected to record system time and subtract it on receiving response? Or?

As a disclaimer, I am no expert for clock reading, so any additional insights are appreciated very much.

I think how the driver accounts for the time delta should remain outside of the scope of the specification. In the current spec the interpretation of clock reading is behavior internal to the driver. Also, AFAIU interpretation of clock readings is a scientific and technical topic of its own, which has been extensively discussed in the past decades. The choice of a clock reading algorithm implies trade-offs between precision, implementation complexity, and resource usage/clock reading speed (when re-reading the clock).

The Linux kernel has ioctl PTP_SYS_OFFSET_EXTENDED, which will read the system time immediately before and after each PTP clock reading, leaving it up to userspace how to interpret the clock reading. This ioctl can also be used with the RFC Linux kernel driver implementing this spec [2], which exposes a virtual PTP clock. So in this case, it would hardly help to stipulate Virtio driver behavior, when the actual clock reading interpretation happens in userspace daemons using PTP_SYS_OFFSET_EXTENDED, such as chrony.

I see the above discussed VIRTIO_RTC_M_READ method mainly as a fallback in cases where the VIRTIO_RTC_M_READ_CROSS method would not be available. The VIRTIO_RTC_M_READ_CROSS method is more or less independent of the time delta (which can be in the 100 ms range, as discussed in the driver patch series cover letter [2]).

The above question is addressed in the introductory paragraph for "readq Operation" as follows:

Quoted 2 lines

The driver may enhance and interpret the clock readings through methods which are beyond the scope of this specification.

I think I will add part of the above justification to the commit message.

Quoted 5 lines

Ran out of time.
Will continue after I get above base line alignment.

Thanks.

Thanks a lot for the review comments. Looking forward to continue the discussion.

Peter

[2] https://lore.kernel.org/lkml/[email protected]/T/#me7df2d4db4fe1119d821dc9c4054b9404c15b02d

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#3 lore raw
Quoted 7 lines

From: Peter Hilber <[email protected]>
Sent: Thursday, July 27, 2023 3:51 PM

Quoted 2 lines
Quoted 2 lines

In the spec, clock ids are 64 bit wide to have unique ids when supporting clock

Quoted 8 lines
Quoted 2 lines

It is unclear the namespace of uniqueness.
Do you mean unique id across multiple virtio devices?

The idea was not to have a unique id across multiple virtio devices. Rather, the idea was to simplify lifetime management for clocks in a single virtio device (if such lifetimes would be introduced in the future) by not having to worry about race conditions due to clock id reuse.

If the id not unique, 16-bit is likely more than enough.
There should not be any race condition with respect to clock id as its per device.

Quoted 6 lines
Quoted 2 lines

If so, how is it proposed, we probably need UUID?

So it would be up to the device implementation to assign unique ids for each clock lifetime; incrementing a counter should do.

Ok. uniqueness is per device hence any scheme by device is fine as you explained.

Quoted 27 lines
Quoted 7 lines
Quoted 4 lines

hot-plugging in the future.

Any feedback is appreciated very much.

This is a good feature.

A network device has a PHC which is very similar to what is being proposed

here which is super set of this proposal.

Quoted 2 lines

A network device will support packet timestamping as phase-1, followed by

PTP synchronized UTC/TAI format in phase-2 followed by PTP timestamping offload in phase-3.

Quoted 7 lines

Phase-1 is already being proposed using monotonic clock method at [1].

[1]
https://lists.oasis-open.org/archives/virtio-comment/202306/msg00287.h
tml

We would like to have a single virtio network device to expose the optional

RTC without creating a full blown new virtio device.

Quoted 2 lines

A quick grep in the industry indicates that at least 15+ nics in industry with

dual PTP clock + nic functionality.

Quoted 2 lines

And we would like virtio network device also to have it.
At minimum such solution avoids creating an additional full blown virtio

device.

Quoted 1 line
Quoted 8 lines
Quoted 6 lines

As proposed here, RTC device as stand alone itself has its merit and use.

So lets build the RTC device interface that can be of dual use.
1. as part of stand alone RTC device
2. as part of virtio network device

To me it seems preferable to first try to keep the two devices separated.

Devices can be separate, the driver to device interface between them should be common to reuse it.

Quoted 4 lines

AFAIU
PTP clocks are also usually represented as a device separate from network devices.

Both the modes are in use, hence we want to support #1 and #2 both without duplicating the interfaces.

Quoted 4 lines

The extra effort to implement another simple virtio device (rather than putting everything in a single device) seems to be small when compared to the complexity of creating an extensible specification which should work both standalone and as part of an already existing, but still growing, device.

It may sound as small effort with only a focused view on RTC only device.
But both types are needed.
From actual implementation point of view and use case wise, yet another PCI device at scale of virtual machines is not desired, especially when it has tight usage completing on network device.

Quoted 2 lines

E.g., the
RTC device should also get an alarm, with an associated eventq, in the future.

Sure, it is ok. it will be optional when it is RTC device.

Quoted 3 lines

Also, I wonder if the driver implementation should then work both as a standalone Virtio driver and as part of the net driver.

Driver layer will be fine. Linux stack evolved over last 3 years to support multi-functionality device. A RTC/PTP driver will be a stand alone entity that works on underlying layered way to issue read cmd, config etc.

Quoted 11 lines

Maybe the above use case could also be addressed by a standalone RTC device as follows:

1) The net device exports a UUID object to which the RTC device can refer to (so the drivers can link the net device to the RTC clock).

2) If needed, the RTC device aligns the lifetime of the clock to the net device. Then, clock lifetimes are no longer static, and can be aligned with the net device. Through an eventq (to be added), the RTC device announces clock creation/destruction, which can be synchronized to the net device state.

All of the infrastructure, cross device synchronization, its device side management will come with equal complexity. And we will end up with sub-optimal way to using multiple PCI devices.

Net already has cvq required by RTC. So cvq commands can be common. Readq is additional, we need to define common between the two and way to discover it.

Quoted 4 lines
Quoted 3 lines

Such work will enable to re-invent the wheel 2nd time in virtio net.

What I can think of is following definitions being common between two device

types.

Quoted 8 lines
Quoted 6 lines

This will be able to leverage your rtc work done as part of this patchset.

1. _F feature bit that can be common between the rtc and net device 2. control vq command opcode and format be common 3. discovery of the readq id possibly using cvq and it must depend on _F_RESET bit.

Not sure what _F_RESET refers to, is it VIRTIO_F_RING_RESET? And why?

Yes, because one needs the ability to create the readq dynamically only when/if such virtio_rtc driver is loaded when a device is multi-functionality device.

Quoted 8 lines
Quoted 3 lines

device-types/rtc/description.tex

Quoted 3 lines
device-types/rtc/description.tex new file +0 -0

If this sounds better, I could rephrase to "The driver enqueues clock reading requests to the readq."

Looks good.

Quoted 20 lines
Quoted 10 lines

\footnote{The readq is a dedicated virtqueue to facilitate

device-types/rtc/description.tex

Quoted 9 lines
device-types/rtc/description.tex new file +9 -0
+reducing latency for clock readings, which may be more
+performance-sensitive than other requests.} The driver uses the
+controlq for other requests to the device.
+
+\subsection{Feature bits}\label{sec:Device Types / RTC Device /
+Feature bits}
+
+\begin{description}
+\item[VIRTIO_RTC_F_READ_CROSS (0)] Device supports cross-

timestamping.

Quoted 9 lines

device-types/rtc/description.tex

Quoted 8 lines

+\end{description}

device-types/rtc/description.tex new file +7 -0
+
+VIRTIO_RTC_F_READ_CROSS determines whether the device supports clock
+reading with cross-timestamping. Without this feature, the device
+does not support cross-timestamping. Even when this feature was
+negotiated, the device may still not support cross-timestamping for
+some clocks, or for particular requests.
+

Can you please explain why even after feature negotiation, device may still

Quoted 7 lines

not support?

Quoted 1 line

Is it because the cross timestamping is per clock feature?

Actually, the cross-timestamping support can be specific to the combination of clock and hardware counter. This is explained in more detail in the context of VIRTIO_RTC_M_READ_CROSS. I will mention this here as well.

Hardware counter is a misleading term here. It is really platform (system) specific identifier.

Maybe I am wrong, but I think cross timestamping functionality can be achieved at the driver level, reading counter value from two sources? Unlikely? because the variance is lot?

If so, why do we need to annotate arm/x86, can it be done without it as just system clock/cycles?

Quoted 6 lines
Quoted 1 line

If so, I guess no need of a feature bit as it is not serving any meaningful

purpose.

The only purpose is to potentially save the device some initialization work or resources in case the feature is not negotiated.

Ok. In that case we should move it to per clock enablement instead of feature bit.

Quoted 6 lines
Quoted 2 lines

Each clock information can advertise what is supports, right?

The driver can use VIRTIO_RTC_M_CROSS_CAP to inquire about this (or the driver could also just try if VIRTIO_RTC_M_READ_CROSS works).

VIRTIO_RTC_M_CROSS_CAP is reliable way to discover the device support.
Direct command if it results in error can throw unwanted kernel errors resulting in false alarms.

Quoted 8 lines
Quoted 1 line

If this clock device is a PCI device, on an empty system, a PCI device has to do

at least couple of DMA.

Quoted 5 lines

1. read the descriptor (250 to 300nsec) 2. Read the request (250 to 300nsec) 3. Write the response (50 nsec+) 4. raise interrupt.

How is this time delta accounted in clock reading?
Is driver expected to record system time and subtract it on receiving

response?

Quoted 8 lines
Quoted 1 line

Or?

As a disclaimer, I am no expert for clock reading, so any additional insights are appreciated very much.

I think how the driver accounts for the time delta should remain outside of the scope of the specification. In the current spec the interpretation of clock reading is behavior internal to the driver.

Well, directly reading the clock as device register can be more precise. But this can be limited to small number of clocks.

Quoted 14 lines

Also, AFAIU interpretation of clock readings is a scientific and technical topic of its own, which has been extensively discussed in the past decades. The choice of a clock reading algorithm implies trade-offs between precision, implementation complexity, and resource usage/clock reading speed (when re-reading the clock).

The Linux kernel has ioctl PTP_SYS_OFFSET_EXTENDED, which will read the system time immediately before and after each PTP clock reading, leaving it up to userspace how to interpret the clock reading. This ioctl can also be used with the RFC Linux kernel driver implementing this spec [2], which exposes a virtual PTP clock. So in this case, it would hardly help to stipulate Virtio driver behavior, when the actual clock reading interpretation happens in userspace daemons using PTP_SYS_OFFSET_EXTENDED, such as chrony.

Quoted 6 lines

I see the above discussed VIRTIO_RTC_M_READ method mainly as a fallback in cases where the VIRTIO_RTC_M_READ_CROSS method would not be available. The VIRTIO_RTC_M_READ_CROSS method is more or less independent of the time delta (which can be in the 100 ms range, as discussed in the driver patch series cover letter [2]).

Do you mean that the system clock and device clock reading happening in single call, hence the _CROSS gives more precise result?

Quoted 5 lines

The above question is addressed in the introductory paragraph for "readq Operation" as follows:

Quoted 2 lines

The driver may enhance and interpret the clock readings through methods which are beyond the scope of this specification.

#4 lore raw
Quoted 10 lines · Parav Pandit
Quoted 10 lines

From: Peter Hilber <[email protected]>
Sent: Thursday, July 27, 2023 3:51 PM

Quoted 4 lines
Quoted 2 lines

In the spec, clock ids are 64 bit wide to have unique ids when supporting clock

It is unclear the namespace of uniqueness.
Do you mean unique id across multiple virtio devices?

Quoted 8 lines
Quoted 5 lines

The idea was not to have a unique id across multiple virtio devices. Rather, the idea was to simplify lifetime management for clocks in a single virtio device (if such lifetimes would be introduced in the future) by not having to worry about race conditions due to clock id reuse.

If the id not unique, 16-bit is likely more than enough.
There should not be any race condition with respect to clock id as its per device.

I think 64 bit is better since the cost is negligible, and e.g. the device then does not have to care how it issues ids while the driver is suspended.

Quoted 42 lines
Quoted 6 lines
Quoted 2 lines

If so, how is it proposed, we probably need UUID?

So it would be up to the device implementation to assign unique ids for each clock lifetime; incrementing a counter should do.

Ok. uniqueness is per device hence any scheme by device is fine as you explained.

Quoted 34 lines
Quoted 7 lines
Quoted 4 lines

hot-plugging in the future.

Any feedback is appreciated very much.

This is a good feature.

A network device has a PHC which is very similar to what is being proposed

here which is super set of this proposal.

Quoted 2 lines

A network device will support packet timestamping as phase-1, followed by

PTP synchronized UTC/TAI format in phase-2 followed by PTP timestamping offload in phase-3.

Quoted 7 lines

Phase-1 is already being proposed using monotonic clock method at [1].

[1]
https://lists.oasis-open.org/archives/virtio-comment/202306/msg00287.h
tml

We would like to have a single virtio network device to expose the optional

RTC without creating a full blown new virtio device.

Quoted 2 lines

A quick grep in the industry indicates that at least 15+ nics in industry with

dual PTP clock + nic functionality.

Quoted 2 lines

And we would like virtio network device also to have it.
At minimum such solution avoids creating an additional full blown virtio

device.

Quoted 7 lines

As proposed here, RTC device as stand alone itself has its merit and use.

So lets build the RTC device interface that can be of dual use.
1. as part of stand alone RTC device
2. as part of virtio network device

Quoted 9 lines
Quoted 1 line

To me it seems preferable to first try to keep the two devices separated.

Devices can be separate, the driver to device interface between them should be common to reuse it.

Quoted 4 lines

AFAIU
PTP clocks are also usually represented as a device separate from network devices.

Both the modes are in use, hence we want to support #1 and #2 both without duplicating the interfaces.

What I wanted to refer to in my reply are the Linux kernel /dev/ptp* interfaces, which are separate from the network device interfaces.

Quoted 9 lines
Quoted 4 lines

The extra effort to implement another simple virtio device (rather than putting everything in a single device) seems to be small when compared to the complexity of creating an extensible specification which should work both standalone and as part of an already existing, but still growing, device.

It may sound as small effort with only a focused view on RTC only device.
But both types are needed.
From actual implementation point of view and use case wise, yet another PCI device at scale of virtual machines is not desired, especially when it has tight usage completing on network device.

If the issue is PCI transport related resource consumption, the PCI transport could be extended to support multi-function devices, in a way which is agnostic to the device types involved.

Quoted 10 lines
Quoted 2 lines

E.g., the
RTC device should also get an alarm, with an associated eventq, in the future.

Sure, it is ok. it will be optional when it is RTC device.

Quoted 3 lines

Also, I wonder if the driver implementation should then work both as a standalone Virtio driver and as part of the net driver.

Driver layer will be fine. Linux stack evolved over last 3 years to support multi-functionality device. A RTC/PTP driver will be a stand alone entity that works on underlying layered way to issue read cmd, config etc.

As per my above reply, I think it would be better to tackle the problem at the PCI transport level, then there would be no impact on individual device drivers.

Quoted 6 lines
Quoted 6 lines

Maybe the above use case could also be addressed by a standalone RTC device as follows:

1) The net device exports a UUID object to which the RTC device can refer to (so the drivers can link the net device to the RTC clock).

Quoted 8 lines
Quoted 5 lines

2) If needed, the RTC device aligns the lifetime of the clock to the net device. Then, clock lifetimes are no longer static, and can be aligned with the net device. Through an eventq (to be added), the RTC device announces clock creation/destruction, which can be synchronized to the net device state.

All of the infrastructure, cross device synchronization, its device side management will come with equal complexity. And we will end up with sub-optimal way to using multiple PCI devices.

As per my above reply, I think it would be better to tackle the problem at the PCI transport level...

Quoted 3 lines

Net already has cvq required by RTC. So cvq commands can be common. Readq is additional, we need to define common between the two and way to discover it.

... but if not doing that, I would still try to avoid mixing net and RTC details. RTC should keep a dedicated cvq. Then the RTC driver could use generic Virtio functionality to send/receive rather than using an interface which is specific to the net driver cvq.

[snip]

Quoted 20 lines
Quoted 20 lines
Quoted 10 lines

\footnote{The readq is a dedicated virtqueue to facilitate

Review

Quoted 9 lines
(quoted diff) +9 -0
+reducing latency for clock readings, which may be more
+performance-sensitive than other requests.} The driver uses the
+controlq for other requests to the device.
+
+\subsection{Feature bits}\label{sec:Device Types / RTC Device /
+Feature bits}
+
+\begin{description}
+\item[VIRTIO_RTC_F_READ_CROSS (0)] Device supports cross-

timestamping.

Quoted 9 lines

Review

Quoted 8 lines

+\end{description}

(quoted diff) +7 -0
+
+VIRTIO_RTC_F_READ_CROSS determines whether the device supports clock
+reading with cross-timestamping. Without this feature, the device
+does not support cross-timestamping. Even when this feature was
+negotiated, the device may still not support cross-timestamping for
+some clocks, or for particular requests.
+

Can you please explain why even after feature negotiation, device may still

Quoted 9 lines
Quoted 7 lines

not support?

Quoted 1 line

Is it because the cross timestamping is per clock feature?

Actually, the cross-timestamping support can be specific to the combination of clock and hardware counter. This is explained in more detail in the context of VIRTIO_RTC_M_READ_CROSS. I will mention this here as well.

Hardware counter is a misleading term here. It is really platform (system) specific identifier.

Maybe with the currently defined hardware counters there is a one-to-one correspondence to platforms, but that need not be the case. E.g. it might be more convenient for a device on an Intel x86 platform to read the Always Running Timer (ART) rather than the TSC derived from the ART, and then the ART could become another HW counter in the spec for x86.

Quoted 3 lines

Maybe I am wrong, but I think cross timestamping functionality can be achieved at the driver level, reading counter value from two sources? Unlikely? because the variance is lot?

In short, cross-timestamping at the device side can be

- more precise,
- practically insensitive to temporal behavior, and
- less resource demanding due to less need to repeat clock readings,

if the device has a clock based on an appropriate hardware counter.

With the cross-timestamping, the device can convey its understanding of the current time at the instant at which the hardware counter has a specific value. Both the device and the driver should be able to read the same hardware counter. When the device repeatedly conveys its understanding of the current time and corresponding hardware counter reading, a clock synchronization daemon at the driver side can infer the offset and speed of the device's clock (relative to the driver's system time, which can also be related to the same hardware counter).

If the driver side derives system time from the hardware counter (and if the device does not manipulate the clock ATM), then a driver-side clock synchronization daemon can synchronize the system time to a precision of < 10 ns. For this, there are no special requirements on calibration or clock reading performance, except that the hardware counter view by device and driver should be synchronized.

A good illustration of cross-timestamping at the (virtual) device side is the ptp_kvm arm64 implementation [3], which uses ktime_get_snapshot() for cross-timestamping.

By contrast, cross-timestamping at the driver side (as one could say PTP_SYS_OFFSET_EXTENDED does in the Linux kernel) results in inferior precision, is sensitive to clock reading performance, and probably has to be repeated more often. From my POV cross-timestamping should be done at the device side, if the device has access to suitable hardware counters.

Quoted 2 lines

If so, why do we need to annotate arm/x86, can it be done without it as just system clock/cycles?

As discussed above, additional counters may be used in the future.

Also, on x86, no TSC might be present or used (the Linux on my own notebook until recently refused to keep using TSC due to a firmware issue).

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

If so, I guess no need of a feature bit as it is not serving any meaningful

purpose.

The only purpose is to potentially save the device some initialization work or resources in case the feature is not negotiated.

Ok. In that case we should move it to per clock enablement instead of feature bit.

As per my above replies, I think it should be kept clock and hardware counter specific. The feature bit could be removed if it seems not helpful.

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

Each clock information can advertise what is supports, right?

The driver can use VIRTIO_RTC_M_CROSS_CAP to inquire about this (or the driver could also just try if VIRTIO_RTC_M_READ_CROSS works).

VIRTIO_RTC_M_CROSS_CAP is reliable way to discover the device support.
Direct command if it results in error can throw unwanted kernel errors resulting in false alarms.

Agreed. The second option is not mentioned in the v2 spec so no action should be necessary.

Quoted 10 lines
Quoted 10 lines
Quoted 1 line

If this clock device is a PCI device, on an empty system, a PCI device has to do

at least couple of DMA.

Quoted 5 lines

1. read the descriptor (250 to 300nsec) 2. Read the request (250 to 300nsec) 3. Write the response (50 nsec+) 4. raise interrupt.

How is this time delta accounted in clock reading?
Is driver expected to record system time and subtract it on receiving

response?

Quoted 1 line

Or?

Quoted 9 lines
Quoted 6 lines

As a disclaimer, I am no expert for clock reading, so any additional insights are appreciated very much.

I think how the driver accounts for the time delta should remain outside of the scope of the specification. In the current spec the interpretation of clock reading is behavior internal to the driver.

Well, directly reading the clock as device register can be more precise. But this can be limited to small number of clocks.

This could be added if required. But to me it would only make sense where cross-timestamping doesn't work (and where precision is important).

Quoted 14 lines
Quoted 14 lines

Also, AFAIU interpretation of clock readings is a scientific and technical topic of its own, which has been extensively discussed in the past decades. The choice of a clock reading algorithm implies trade-offs between precision, implementation complexity, and resource usage/clock reading speed (when re-reading the clock).

The Linux kernel has ioctl PTP_SYS_OFFSET_EXTENDED, which will read the system time immediately before and after each PTP clock reading, leaving it up to userspace how to interpret the clock reading. This ioctl can also be used with the RFC Linux kernel driver implementing this spec [2], which exposes a virtual PTP clock. So in this case, it would hardly help to stipulate Virtio driver behavior, when the actual clock reading interpretation happens in userspace daemons using PTP_SYS_OFFSET_EXTENDED, such as chrony.

Quoted 8 lines
Quoted 6 lines

I see the above discussed VIRTIO_RTC_M_READ method mainly as a fallback in cases where the VIRTIO_RTC_M_READ_CROSS method would not be available. The VIRTIO_RTC_M_READ_CROSS method is more or less independent of the time delta (which can be in the 100 ms range, as discussed in the driver patch series cover letter [2]).

Do you mean that the system clock and device clock reading happening in single call, hence the _CROSS gives more precise result?

The clock reading duration is only relevant for the simple VIRTIO_RTC_M_READ method. For VIRTIO_RTC_M_READ_CROSS, the clock reading duration is not critical, since the measurements are done at the device side. The driver side just needs to know what time it would have associated to the hardware counter reading returned by the device.

I think I will add some contents from my replies to the commit message.

Thanks for the feedback,

Peter

[3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3bf725699bf62494b3e179f1795f08c7d749f061

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#5 lore raw
Quoted 20 lines

From: Peter Hilber <[email protected]>
Sent: Friday, August 4, 2023 9:38 PM

Quoted 15 lines · Parav Pandit
Quoted 15 lines

From: Peter Hilber <[email protected]>
Sent: Thursday, July 27, 2023 3:51 PM

Quoted 4 lines
Quoted 2 lines

In the spec, clock ids are 64 bit wide to have unique ids when supporting clock

It is unclear the namespace of uniqueness.
Do you mean unique id across multiple virtio devices?

The idea was not to have a unique id across multiple virtio devices. Rather, the idea was to simplify lifetime management for clocks in a single virtio device (if such lifetimes would be introduced in the future) by not having to worry about race conditions due to clock id reuse.

Quoted 7 lines
Quoted 2 lines

If the id not unique, 16-bit is likely more than enough.
There should not be any race condition with respect to clock id as its per

device.

Quoted 1 line

I think 64 bit is better since the cost is negligible, and e.g. the device then does not have to care how it issues ids while the driver is suspended.

Can you please explain how 64-bit better when the number of clocks per device is not so much? Even with 256 or 64K clocks device wont care how the ids are issued.

Quoted 7 lines
Quoted 1 line

Both the modes are in use, hence we want to support #1 and #2 both without

duplicating the interfaces.

Quoted 1 line

What I wanted to refer to in my reply are the Linux kernel /dev/ptp* interfaces, which are separate from the network device interfaces.

Sure. It can be.
When the netdevice has the RTC features supports it will create /dev/ptp* and /sys/class/net/netdev devices.

All the plumbing that is done is for dual use.
So command etc definitions is for dual use as I explained before. Those can be used for virtio-net device when it has in-built rtc.

When there is only rtc device with its own device-type, same functionality and commands are there. Driver will create only /dev/ptp* device.

From SW point of view it is just matter of layering.. Already done by more than 13+ industry devices..

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

The extra effort to implement another simple virtio device (rather than putting everything in a single device) seems to be small when compared to the complexity of creating an extensible specification which should work both standalone and as part of an already existing, but

still growing, device.

Quoted 3 lines

It may sound as small effort with only a focused view on RTC only device. But both types are needed.
From actual implementation point of view and use case wise, yet another PCI

Quoted 9 lines

device at scale of virtual machines is not desired, especially when it has tight usage completing on network device.

Quoted 2 lines

If the issue is PCI transport related resource consumption, the PCI transport could be extended to support multi-function devices, in a way which is agnostic to the device types involved.

All things needed by rtc is already present on net device. So there is no point of improving PCI transport etc. PCI transport resource improvement goes just beyond PCI transport improvement. Such as APIC, IOMMU, device registers and more. This is equally an overkill.

And defining and duplicating exact same commands in net is equally sub-optimal, specially when we know its need for last several months.

This is the right junction to reuse functionality on two device types without duplication.

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

E.g., the
RTC device should also get an alarm, with an associated eventq, in the

future.

Quoted 5 lines

Sure, it is ok. it will be optional when it is RTC device.

Quoted 3 lines

Also, I wonder if the driver implementation should then work both as a standalone Virtio driver and as part of the net driver.

Quoted 9 lines
Quoted 1 line

Driver layer will be fine. Linux stack evolved over last 3 years to support

multi-functionality device.

Quoted 1 line

A RTC/PTP driver will be a stand alone entity that works on underlying layered

way to issue read cmd, config etc.

Quoted 1 line

As per my above reply, I think it would be better to tackle the problem at the PCI transport level, then there would be no impact on individual device drivers.

There is no impact by better definition usable on two devices.

Quoted 20 lines
Quoted 10 lines
Quoted 10 lines

Maybe the above use case could also be addressed by a standalone RTC device as follows:

1) The net device exports a UUID object to which the RTC device can refer to (so the drivers can link the net device to the RTC clock).

2) If needed, the RTC device aligns the lifetime of the clock to the net device. Then, clock lifetimes are no longer static, and can be aligned with the net device. Through an eventq (to be added), the RTC device announces clock creation/destruction, which can be synchronized to the net

device state.

Quoted 2 lines
Quoted 1 line

All of the infrastructure, cross device synchronization, its device side

management will come with equal complexity.

Quoted 2 lines

And we will end up with sub-optimal way to using multiple PCI devices.

As per my above reply, I think it would be better to tackle the problem at the PCI transport level...

Quoted 7 lines
Quoted 2 lines

Net already has cvq required by RTC. So cvq commands can be common. Readq is additional, we need to define common between the two and way to

discover it.

Quoted 1 line

I would still try to avoid mixing net and RTC details. RTC
should keep a dedicated cvq.

Can you please explain why a dedicate cvq is needed on net device?

Quoted 4 lines

Then the RTC driver could use generic Virtio
functionality to send/receive rather than using an interface which is specific to the net driver cvq.

Please explain why?
We have dedicated vqs work ongoing for flow filters due to high rate of requests requirement handled by hw.

So would like to understand if we can extend similar dedicated cvq. Given most of the queries done via cvq for rts are not time sensitive. Hence net cvq seems enough.

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

Actually, the cross-timestamping support can be specific to the combination of clock and hardware counter. This is explained in more detail in the context of VIRTIO_RTC_M_READ_CROSS. I will mention this

here as well.

Quoted 2 lines
Quoted 1 line

Hardware counter is a misleading term here. It is really platform (system)

Quoted 9 lines

specific identifier.

Quoted 1 line

Maybe with the currently defined hardware counters there is a one-to-one correspondence to platforms, but that need not be the case. E.g. it might be more convenient for a device on an Intel x86 platform to read the Always Running Timer (ART) rather than the TSC derived from the ART, and then the ART could become another HW counter in the spec for x86.

Maybe I am wrong, but it looks like this is platform feature where driver can device to read ART or ARM timer without any device knowledge?

Can you please explain, if the device does not have the knowledge of platform/system timer, can it still achieve the cross timestamping at driver level?

Quoted 33 lines
Quoted 1 line

Maybe I am wrong, but I think cross timestamping functionality can be

achieved at the driver level, reading counter value from two sources?

Quoted 2 lines

Unlikely? because the variance is lot?

In short, cross-timestamping at the device side can be

- more precise,
- practically insensitive to temporal behavior, and
- less resource demanding due to less need to repeat clock readings,

if the device has a clock based on an appropriate hardware counter.

With the cross-timestamping, the device can convey its understanding of the current time at the instant at which the hardware counter has a specific value. Both the device and the driver should be able to read the same hardware counter. When the device repeatedly conveys its understanding of the current time and corresponding hardware counter reading, a clock synchronization daemon at the driver side can infer the offset and speed of the device's clock (relative to the driver's system time, which can also be related to the same hardware counter).

If the driver side derives system time from the hardware counter (and if the device does not manipulate the clock ATM), then a driver-side clock synchronization daemon can synchronize the system time to a precision of < 10 ns. For this, there are no special requirements on calibration or clock reading performance, except that the hardware counter view by device and driver should be synchronized.

A good illustration of cross-timestamping at the (virtual) device side is the ptp_kvm arm64 implementation [3], which uses ktime_get_snapshot() for cross- timestamping.

Quoted 10 lines

By contrast, cross-timestamping at the driver side (as one could say PTP_SYS_OFFSET_EXTENDED does in the Linux kernel) results in inferior precision, is sensitive to clock reading performance, and probably has to be repeated more often. From my POV cross-timestamping should be done at the device side, if the device has access to suitable hardware counters.

Quoted 1 line

If so, why do we need to annotate arm/x86, can it be done without it as just

system clock/cycles?

Quoted 1 line

Your above detail explanation likely answers why device view of system counter is more precise.

Quoted 2 lines

As discussed above, additional counters may be used in the future.

Do you mean additional system counter?

Quoted 10 lines

Also, on x86, no TSC might be present or used (the Linux on my own notebook until recently refused to keep using TSC due to a firmware issue).

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

If so, I guess no need of a feature bit as it is not serving any meaningful

purpose.

The only purpose is to potentially save the device some initialization work or resources in case the feature is not negotiated.

Quoted 7 lines
Quoted 1 line

Ok. In that case we should move it to per clock enablement instead of feature

bit.

Quoted 1 line

As per my above replies, I think it should be kept clock and hardware counter specific. The feature bit could be removed if it seems not helpful.

Ok. lets remove it.

Quoted 32 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Each clock information can advertise what is supports, right?

The driver can use VIRTIO_RTC_M_CROSS_CAP to inquire about this (or the driver could also just try if VIRTIO_RTC_M_READ_CROSS works).

VIRTIO_RTC_M_CROSS_CAP is reliable way to discover the device support. Direct command if it results in error can throw unwanted kernel errors

resulting in false alarms.

Quoted 1 line

Agreed. The second option is not mentioned in the v2 spec so no action should be necessary.

Quoted 18 lines
Quoted 18 lines
Quoted 2 lines

If this clock device is a PCI device, on an empty system, a PCI device has to do

at least couple of DMA.

Quoted 6 lines

1. read the descriptor (250 to 300nsec) 2. Read the request (250 to 300nsec) 3. Write the response (50 nsec+) 4. raise interrupt.

How is this time delta accounted in clock reading? Is driver expected to record system time and subtract it on receiving

response?

Quoted 1 line

Or?

As a disclaimer, I am no expert for clock reading, so any additional insights are appreciated very much.

I think how the driver accounts for the time delta should remain outside of the scope of the specification. In the current spec the interpretation of clock reading is behavior internal to the driver.

Quoted 7 lines
Quoted 3 lines

Well, directly reading the clock as device register can be more precise. But this can be limited to small number of clocks.

This could be added if required. But to me it would only make sense where cross-timestamping doesn't work (and where precision is important).

Ok. So this can be follow up post this work.

Quoted 33 lines
Quoted 6 lines
Quoted 6 lines

Also, AFAIU interpretation of clock readings is a scientific and technical topic of its own, which has been extensively discussed in the past decades. The choice of a clock reading algorithm implies trade-offs between precision, implementation complexity, and resource usage/clock reading speed (when re-reading the clock).

Quoted 7 lines
Quoted 7 lines

The Linux kernel has ioctl PTP_SYS_OFFSET_EXTENDED, which will read the system time immediately before and after each PTP clock reading, leaving it up to userspace how to interpret the clock reading. This ioctl can also be used with the RFC Linux kernel driver implementing this spec [2], which exposes a virtual PTP clock. So in this case, it would hardly help to stipulate Virtio driver behavior, when the actual clock reading interpretation happens in userspace daemons using

PTP_SYS_OFFSET_EXTENDED, such as chrony.

Quoted 3 lines
Quoted 3 lines

I see the above discussed VIRTIO_RTC_M_READ method mainly as a fallback in cases where the VIRTIO_RTC_M_READ_CROSS method would not

be available.

Quoted 1 line
Quoted 1 line

The VIRTIO_RTC_M_READ_CROSS method is more or less independent of

the

Quoted 4 lines
Quoted 3 lines

time delta (which can be in the 100 ms range, as discussed in the driver patch series cover letter [2]).

Do you mean that the system clock and device clock reading happening in

single call, hence the _CROSS gives more precise result?

Quoted 1 line

The clock reading duration is only relevant for the simple VIRTIO_RTC_M_READ method. For VIRTIO_RTC_M_READ_CROSS, the clock reading duration is not critical, since the measurements are done at the device side. The driver side just needs to know what time it would have associated to the hardware counter reading returned by the device.

Quoted 4 lines

I think I will add some contents from my replies to the commit message.

Thanks for the feedback,

I apologize for my slow response. I was not well when you replied.
After that lost track of your email in stacks of email until you sent the newer version.

Lets bring rtc functionality in the two devices. We are almost there, just need to refine the definitions.

#6 lore raw
Quoted 21 lines · Parav Pandit
Quoted 21 lines

From: Peter Hilber <[email protected]>
Sent: Friday, August 4, 2023 9:38 PM

Quoted 17 lines · Parav Pandit
Quoted 7 lines

From: Peter Hilber <[email protected]>
Sent: Thursday, July 27, 2023 3:51 PM

Quoted 2 lines
Quoted 2 lines

In the spec, clock ids are 64 bit wide to have unique ids when supporting clock

Quoted 8 lines
Quoted 2 lines

It is unclear the namespace of uniqueness.
Do you mean unique id across multiple virtio devices?

The idea was not to have a unique id across multiple virtio devices. Rather, the idea was to simplify lifetime management for clocks in a single virtio device (if such lifetimes would be introduced in the future) by not having to worry about race conditions due to clock id reuse.

If the id not unique, 16-bit is likely more than enough.
There should not be any race condition with respect to clock id as its per

Quoted 8 lines
Quoted 5 lines

device.

Quoted 1 line

I think 64 bit is better since the cost is negligible, and e.g. the device then does not have to care how it issues ids while the driver is suspended.

Can you please explain how 64-bit better when the number of clocks per device is not so much? Even with 256 or 64K clocks device wont care how the ids are issued.

64 bit width should address the potential future case where two clocks get hot-plugged and hot-unplugged at a high rate. If the device hands out a new id each time a clock is re-plugged, and if the driver suspends and resumes during this, an 8 bit id could result in the driver interpreting ids wrongly.

Quoted 14 lines
Quoted 7 lines
Quoted 1 line

Both the modes are in use, hence we want to support #1 and #2 both without

duplicating the interfaces.

Quoted 1 line

What I wanted to refer to in my reply are the Linux kernel /dev/ptp* interfaces, which are separate from the network device interfaces.

Sure. It can be.
When the netdevice has the RTC features supports it will create /dev/ptp* and /sys/class/net/netdev devices.

All the plumbing that is done is for dual use.
So command etc definitions is for dual use as I explained before. Those can be used for virtio-net device when it has in-built rtc.

Quoted 5 lines

When there is only rtc device with its own device-type, same functionality and commands are there. Driver will create only /dev/ptp* device.

From SW point of view it is just matter of layering.. Already done by more than 13+ industry devices..

I think there was a misunderstanding. I was trying to say: When it is better to show distinct devices to userspace, would it not also be better to show distinct devices at the Virtio level?

Quoted 13 lines
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines

The extra effort to implement another simple virtio device (rather than putting everything in a single device) seems to be small when compared to the complexity of creating an extensible specification which should work both standalone and as part of an already existing, but

still growing, device.

Quoted 3 lines

It may sound as small effort with only a focused view on RTC only device. But both types are needed.
From actual implementation point of view and use case wise, yet another PCI

device at scale of virtual machines is not desired, especially when it has tight usage completing on network device.

Quoted 2 lines
Quoted 7 lines
Quoted 4 lines

If the issue is PCI transport related resource consumption, the PCI transport could be extended to support multi-function devices, in a way which is agnostic to the device types involved.

All things needed by rtc is already present on net device. So there is no point of improving PCI transport etc. PCI transport resource improvement goes just beyond PCI transport improvement. Such as APIC, IOMMU, device registers and more. This is equally an overkill.

I understand it would be more effort. But to me it looks better as a generic solution, which avoids imposing restrictions on the RTC device.

Quoted 3 lines

And defining and duplicating exact same commands in net is equally sub-optimal, specially when we know its need for last several months.

This is the right junction to reuse functionality on two device types without duplication.

In my opinion it would still be preferable to have one RTC device, and have the RTC device cooperate with the net device where needed.

Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines

E.g., the
RTC device should also get an alarm, with an associated eventq, in the

future.

Quoted 6 lines

Sure, it is ok. it will be optional when it is RTC device.

Quoted 3 lines

Also, I wonder if the driver implementation should then work both as a standalone Virtio driver and as part of the net driver.

Driver layer will be fine. Linux stack evolved over last 3 years to support

multi-functionality device.

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

A RTC/PTP driver will be a stand alone entity that works on underlying layered

way to issue read cmd, config etc.

Quoted 1 line

As per my above reply, I think it would be better to tackle the problem at the PCI transport level, then there would be no impact on individual device drivers.

There is no impact by better definition usable on two devices.

My main concerns about sharing the spec with the net device are:

- the RTC device would act as a guinea pig, since no other devices share device-specific requirements so far

- a more complex spec impacting two devices

- strong coupling between net and RTC driver implementation (or code duplication)

- using custom interfaces (between net and RTC driver), rather than generic interfaces

- RTC could not use device configuration

- unrecognized issues, or future generic changes to the Virtio standard, may inhibit future development of the RTC device or the net device

- unclear responsibility and timeline for net device related spec and proof-of-concept

Quoted 20 lines
Quoted 20 lines
Quoted 10 lines
Quoted 10 lines

Maybe the above use case could also be addressed by a standalone RTC device as follows:

1) The net device exports a UUID object to which the RTC device can refer to (so the drivers can link the net device to the RTC clock).

2) If needed, the RTC device aligns the lifetime of the clock to the net device. Then, clock lifetimes are no longer static, and can be aligned with the net device. Through an eventq (to be added), the RTC device announces clock creation/destruction, which can be synchronized to the net

device state.

Quoted 2 lines
Quoted 1 line

All of the infrastructure, cross device synchronization, its device side

management will come with equal complexity.

Quoted 2 lines

And we will end up with sub-optimal way to using multiple PCI devices.

As per my above reply, I think it would be better to tackle the problem at the PCI transport level...

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Net already has cvq required by RTC. So cvq commands can be common. Readq is additional, we need to define common between the two and way to

discover it.

Quoted 1 line

I would still try to avoid mixing net and RTC details. RTC
should keep a dedicated cvq.

Can you please explain why a dedicate cvq is needed on net device?

I think the concern about redundant queues could be addressed by merging the RTC readq and controlq. At the moment separating these queues would not yield performance benefits, so I think I will just merge the readq and controlq RTC queues in the next spec iteration.

As to why keeping the net and RTC queues separated: With a dedicated queue, the RTC driver message processing could remain decoupled from the net driver. Also, the net cvq format does not align well with the RTC messages. RTC messages have detailed status codes, while net only has VIRTIO_NET_OK and VIRTIO_NET_ERR.

Quoted 9 lines
Quoted 4 lines

Then the RTC driver could use generic Virtio
functionality to send/receive rather than using an interface which is specific to the net driver cvq.

Please explain why?
We have dedicated vqs work ongoing for flow filters due to high rate of requests requirement handled by hw.

So would like to understand if we can extend similar dedicated cvq. Given most of the queries done via cvq for rts are not time sensitive. Hence net cvq seems enough.

This should have been addressed by merging the RTC readq and controlq, as per my above reply.

Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
Quoted 3 lines

Actually, the cross-timestamping support can be specific to the combination of clock and hardware counter. This is explained in more detail in the context of VIRTIO_RTC_M_READ_CROSS. I will mention this

here as well.

Quoted 2 lines
Quoted 1 line

Hardware counter is a misleading term here. It is really platform (system)

specific identifier.

Quoted 1 line
Quoted 7 lines
Quoted 6 lines

Maybe with the currently defined hardware counters there is a one-to-one correspondence to platforms, but that need not be the case. E.g. it might be more convenient for a device on an Intel x86 platform to read the Always Running Timer (ART) rather than the TSC derived from the ART, and then the ART could become another HW counter in the spec for x86.

Maybe I am wrong, but it looks like this is platform feature where driver can device to read ART or ARM timer without any device knowledge?

Not sure what is meant by "driver can device".

What do you mean by "platform feature"? As I argued in my previous reply, the ART could be used instead of the TSC, and then would need to be distinguished from the TSC. Also, there is no guarantee that all platforms will always have the same single relevant counter. E.g. on Arm there is the physical counter view and the virtual counter view (though distinguishing those might not be so important). So device knowledge is required for reliable operation.

To me there is also no advantage in narrowing down the hardware counter terminology to "platform", when it might in principle be other counters, maybe from peripheral devices.

Quoted 1 line

Can you please explain, if the device does not have the knowledge of platform/system timer, can it still achieve the cross timestamping at driver level?

In a simple scenario, if everything is working out, the driver might just guess the right hardware counter. But it should make sense to be able to detect error conditions, and to be able to choose among multiple hardware counters should the device support those.

Rationale: For cross-timestamping, the device sends the driver a hardware counter reading. The driver needs to know how to interpret the hardware counter reading. If the driver assumes an incompatible hardware counter (e.g. TSC instead of ART, or HPET instead of TSC), the cross-timestamp interpretation would be nonsense.

Of course, in general indicating the type of hardware counter might not be sufficient, e.g. when a heterogeneous multicore platform might not have all counters synchronized. In general, the device would need to have some extra implementation-specific knowledge about which hardware counter view the driver has.

Quoted 3 lines
Quoted 2 lines

As discussed above, additional counters may be used in the future.

Do you mean additional system counter?

This was meant to refer to the ART, which would need to be distinguished from the TSC.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

If so, I guess no need of a feature bit as it is not serving any meaningful

purpose.

The only purpose is to potentially save the device some initialization work or resources in case the feature is not negotiated.

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Ok. In that case we should move it to per clock enablement instead of feature

bit.

Quoted 1 line

As per my above replies, I think it should be kept clock and hardware counter specific. The feature bit could be removed if it seems not helpful.

Ok. lets remove it.

OK.

Quoted 2 lines

I apologize for my slow response. I was not well when you replied.
After that lost track of your email in stacks of email until you sent the newer version.

No worries. I was also on leave and I am unfortunately also slow to respond.

Quoted 1 line

Lets bring rtc functionality in the two devices. We are almost there, just need to refine the definitions.

I would still prefer to keep the RTC spec separated from the net spec. But in my understanding the issues which were raised about sharing the RTC spec with the net spec have been addressed (when taking this reply into account, too).

I would prefer to first try to upstream the RTC spec without altering the net spec, while trying to avoid conflicts with the net spec. Maybe most of the RTC spec could then, as a second step, also be applied to the net device. The responsibility for this second step and the related proof of concept work would need to be clarified. To me it still seems better to not share the spec and address resource problems at the transport level.

Let me go into some detail regarding the above. My main concerns about sharing the RTC spec with the net device are (as per an above reply):

- the RTC device would act as a guinea pig, since no other devices share device-specific requirements so far

- a more complex spec impacting two devices

- strong coupling between net and RTC driver implementation (or code duplication)

- using custom interfaces (between net and RTC driver), rather than generic interfaces

- RTC could not use device configuration

- unrecognized issues, or future generic changes to the Virtio standard, may inhibit future development of the RTC device or the net device

- unclear responsibility and timeline for net device related spec and proof-of-concept

To me the main argument in favor of appending the RTC device to the net device seems to be resource consumption related, so might better be addressed in a generic way at the PCI transport level.

That said, by also changing the spec to merge the RTC readq and controlq, in my understanding all the issues which were raised about compatibility with the net spec have been addressed for now.

Last, thank you very much for your valuable remarks! I hope the resulting spec will be able to address diverse use cases, including interoperability with the net device.

Best regards,

Peter

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#7 lore raw

Hi Peter,

Quoted 12 lines

From: Peter Hilber <[email protected]>
Sent: Wednesday, September 13, 2023 2:34 PM

Quoted 1 line

Can you please explain how 64-bit better when the number of clocks per

device is not so much?

Quoted 1 line

Even with 256 or 64K clocks device wont care how the ids are issued.

64 bit width should address the potential future case where two clocks get hot- plugged and hot-unplugged at a high rate. If the device hands out a new id each time a clock is re-plugged, and if the driver suspends and resumes during this, an 8 bit id could result in the driver interpreting ids wrongly.

In the proposed spec, in a given rtc device, total number of clocks are static at the device discovery time. Clock cannot be hotplug or unplugged dynamically.
Did I understand right?

If not, and if the clocks can be added/removed dynamically, if the clock id is not available, the device will wait to hotplug the new one, until the old one is removed. Does one need 64K clocks under single device?
If so, can you please share some thoughts on example application that might find it useful?

Quoted 18 lines
Quoted 10 lines
Quoted 8 lines
Quoted 2 lines

Both the modes are in use, hence we want to support #1 and #2 both without

duplicating the interfaces.

Quoted 1 line

What I wanted to refer to in my reply are the Linux kernel /dev/ptp* interfaces, which are separate from the network device interfaces.

Sure. It can be.
When the netdevice has the RTC features supports it will create /dev/ptp*

and /sys/class/net/netdev devices.

Quoted 6 lines

All the plumbing that is done is for dual use.
So command etc definitions is for dual use as I explained before. Those can be used for virtio-net device when it has in-built rtc.

When there is only rtc device with its own device-type, same functionality

and commands are there.

Quoted 9 lines
Quoted 4 lines

Driver will create only /dev/ptp* device.

From SW point of view it is just matter of layering.. Already done by more than 13+ industry devices..

I think there was a misunderstanding. I was trying to say: When it is better to show distinct devices to userspace, would it not also be better to show distinct devices at the Virtio level?

If a rtc functionality is in the virtio-rtc only device, it will be as virtio-rtc device. If the net device has embedded this functionality, the user will still see a distinct device at the OS level. Every OS would create OS level device object to operate on for the underlying hw device.
So it will be fine to end user.

Quoted 7 lines
Quoted 1 line

PCI transport resource improvement goes just beyond PCI transport

improvement. Such as APIC, IOMMU, device registers and more.

Quoted 1 line

This is equally an overkill.

I understand it would be more effort. But to me it looks better as a generic solution, which avoids imposing restrictions on the RTC device.

There are zero restrictions imposed on the RTC device because the spec is being drafted now. May be I didn’t convey the idea well.

The whole idea is the virtio-rtc can be independent device.
A net-device can have the exact same functionality, so that virtio-rtc functionality can be embedded in it. Thereby, both model co-exists.

Quoted 8 lines
Quoted 1 line

And defining and duplicating exact same commands in net is equally sub-

optimal, specially when we know its need for last several months.

Quoted 2 lines

This is the right junction to reuse functionality on two device types without

duplication.

In my opinion it would still be preferable to have one RTC device, and have the RTC device cooperate with the net device where needed.

I read your below comments for the opinion.
More below.

Quoted 20 lines
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines

E.g., the
RTC device should also get an alarm, with an associated eventq, in the

future.

Quoted 7 lines

Sure, it is ok. it will be optional when it is RTC device.

Quoted 3 lines

Also, I wonder if the driver implementation should then work both as a standalone Virtio driver and as part of the net driver.

Driver layer will be fine. Linux stack evolved over last 3 years to support

Quoted 8 lines
Quoted 8 lines

multi-functionality device.

Quoted 2 lines

A RTC/PTP driver will be a stand alone entity that works on underlying layered

way to issue read cmd, config etc.

Quoted 1 line

As per my above reply, I think it would be better to tackle the problem at the PCI transport level, then there would be no impact on

Quoted 9 lines

individual device drivers.

Quoted 2 lines
Quoted 1 line

There is no impact by better definition usable on two devices.

My main concerns about sharing the spec with the net device are:

- the RTC device would act as a guinea pig, since no other devices share device-specific requirements so far

He he. No it should not. It is only matter of defining cmd opcode and status values that do not overlap one time.

Quoted 2 lines

- a more complex spec impacting two devices

Well it is the functionality, so once you look for slightly larger use cases, overall it is still less complexity than writing two different specs, one for net and one for rtc only device.

Quoted 3 lines

- strong coupling between net and RTC driver implementation (or code duplication)

Mostly no. it is the software layer. RTC layer operates over control VQ.
So net device will provide the needed API. Same API would be available for dedicated rtc device lower layer too. No problem.

Quoted 3 lines

- using custom interfaces (between net and RTC driver), rather than generic interfaces

Multi function devices are common in nature for functionality and scale. And OSes supports it already.

Quoted 2 lines

- RTC could not use device configuration

In your proposal all things are over cvq already.

Quoted 3 lines

- unrecognized issues, or future generic changes to the Virtio standard, may inhibit future development of the RTC device or the net device

Unlikely as the reviews eliminate such issues.

Quoted 3 lines

- unclear responsibility and timeline for net device related spec and proof-of-concept

All the responsible people of the technical committee are here.
In fact the requirements are more closely tracked by its stake owners for net compared to other devices. So if you need help to find net side owner, let me know. Others and I are interested to have it in the net side.

Quoted 29 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Maybe the above use case could also be addressed by a standalone RTC device as follows:

1) The net device exports a UUID object to which the RTC device can refer to (so the drivers can link the net device to the RTC clock).

2) If needed, the RTC device aligns the lifetime of the clock to the net

device.

Quoted 21 lines
Quoted 21 lines
Quoted 4 lines
Quoted 4 lines

Then, clock lifetimes are no longer static, and can be aligned with the net device. Through an eventq (to be added), the RTC device announces clock creation/destruction, which can be synchronized to the net

device state.

Quoted 3 lines
Quoted 1 line

All of the infrastructure, cross device synchronization, its device side

management will come with equal complexity.

Quoted 2 lines

And we will end up with sub-optimal way to using multiple PCI devices.

As per my above reply, I think it would be better to tackle the problem at the PCI transport level...

Quoted 3 lines

Net already has cvq required by RTC. So cvq commands can be common. Readq is additional, we need to define common between the two and way to

discover it.

Quoted 1 line
Quoted 9 lines
Quoted 3 lines
Quoted 2 lines

I would still try to avoid mixing net and RTC details. RTC should
keep a dedicated cvq.

Can you please explain why a dedicate cvq is needed on net device?

I think the concern about redundant queues could be addressed by merging the RTC readq and controlq. At the moment separating these queues would not yield performance benefits, so I think I will just merge the readq and controlq RTC queues in the next spec iteration.

Nice. Ok. lets have single cvq and related opcode range for RTC device and we will place them in the net side too.

Quoted 6 lines

As to why keeping the net and RTC queues separated: With a dedicated queue, the RTC driver message processing could remain decoupled from the net driver. Also, the net cvq format does not align well with the RTC messages. RTC messages have detailed status codes, while net only has VIRTIO_NET_OK and VIRTIO_NET_ERR.

We can extend the virtio net code too. If they are generic, it is likely that net will also find it useful.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Then the RTC driver could use generic Virtio functionality to send/receive rather than using an interface which is specific to the net driver cvq.

Please explain why?
We have dedicated vqs work ongoing for flow filters due to high rate of

Quoted 9 lines

requests requirement handled by hw.

Quoted 2 lines

So would like to understand if we can extend similar dedicated cvq. Given

most of the queries done via cvq for rts are not time sensitive.

Quoted 1 line

Hence net cvq seems enough.

This should have been addressed by merging the RTC readq and controlq, as per my above reply.

Ok.

Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

Actually, the cross-timestamping support can be specific to the combination of clock and hardware counter. This is explained in more detail in the context of VIRTIO_RTC_M_READ_CROSS. I will mention this

here as well.

Quoted 3 lines
Quoted 1 line

Hardware counter is a misleading term here. It is really platform (system)

Quoted 8 lines
Quoted 8 lines

specific identifier.

Quoted 1 line

Maybe with the currently defined hardware counters there is a one-to-one correspondence to platforms, but that need not be the case. E.g. it might be more convenient for a device on an Intel x86 platform to read the Always Running Timer (ART) rather than the TSC derived from the ART, and then the ART could become another HW counter

Quoted 7 lines

in the spec for x86.

Quoted 2 lines
Quoted 1 line

Maybe I am wrong, but it looks like this is platform feature where driver can

device to read ART or ARM timer without any device knowledge?

Not sure what is meant by "driver can device".

I meant to say that driver can read the clock from the system clock and device clock.
At that point, cross timestamping convergence point is driver (and not device).
Would that be sufficient?
This way cross timestamping a driver level is still less accurate, but abstracted from the underlying system.

Or such level of accuracy is insufficient?

Quoted 30 lines

What do you mean by "platform feature"? As I argued in my previous reply, the ART could be used instead of the TSC, and then would need to be distinguished from the TSC. Also, there is no guarantee that all platforms will always have the same single relevant counter. E.g. on Arm there is the physical counter view and the virtual counter view (though distinguishing those might not be so important). So device knowledge is required for reliable operation.

To me there is also no advantage in narrowing down the hardware counter terminology to "platform", when it might in principle be other counters, maybe from peripheral devices.

Quoted 1 line

Can you please explain, if the device does not have the knowledge of

platform/system timer, can it still achieve the cross timestamping at driver level?

In a simple scenario, if everything is working out, the driver might just guess the right hardware counter. But it should make sense to be able to detect error conditions, and to be able to choose among multiple hardware counters should the device support those.

Rationale: For cross-timestamping, the device sends the driver a hardware counter reading. The driver needs to know how to interpret the hardware counter reading. If the driver assumes an incompatible hardware counter (e.g. TSC instead of ART, or HPET instead of TSC), the cross-timestamp interpretation would be nonsense.

Of course, in general indicating the type of hardware counter might not be sufficient, e.g. when a heterogeneous multicore platform might not have all counters synchronized. In general, the device would need to have some extra implementation-specific knowledge about which hardware counter view the driver has.

#8 lore raw
Quoted 2 lines · Parav Pandit

Hi Peter,

Hi Parav,

I'm sorry to be a bit late with my answer, since I only received your email via the mailing list, which I do not check so often.

Quoted 7 lines
Quoted 7 lines

From: Peter Hilber <[email protected]>
Sent: Wednesday, September 13, 2023 2:34 PM

Quoted 1 line

Can you please explain how 64-bit better when the number of clocks per

device is not so much?

Quoted 1 line

Even with 256 or 64K clocks device wont care how the ids are issued.

Quoted 8 lines
Quoted 5 lines

64 bit width should address the potential future case where two clocks get hot- plugged and hot-unplugged at a high rate. If the device hands out a new id each time a clock is re-plugged, and if the driver suspends and resumes during this, an 8 bit id could result in the driver interpreting ids wrongly.

In the proposed spec, in a given rtc device, total number of clocks are static at the device discovery time. Clock cannot be hotplug or unplugged dynamically.
Did I understand right?

Yes. The 64 bit width was just to accommodate for potential future extensions. This does not force the device to use effectively 64 bit wide ids. I chose 64 bit over 8 bit because I did not expect that this would become an issue.

Quoted 4 lines

If not, and if the clocks can be added/removed dynamically, if the clock id is not available, the device will wait to hotplug the new one, until the old one is removed. Does one need 64K clocks under single device?
If so, can you please share some thoughts on example application that might find it useful?

The example scenario is the one which is quoted above: The device might not know when the driver is suspended, and continuously hand out new ids (when clocks are continuously re-plugged for some reason). If the driver later resumes at the wrong moment, by chance the same id which was in use before might now be assigned to another clock.

This and similar scenarios can be prevented by using 64 bit ids.

Quoted 24 lines
Quoted 24 lines
Quoted 11 lines
Quoted 9 lines
Quoted 3 lines

Both the modes are in use, hence we want to support #1 and #2 both without

duplicating the interfaces.

Quoted 1 line

What I wanted to refer to in my reply are the Linux kernel /dev/ptp* interfaces, which are separate from the network device interfaces.

Sure. It can be.
When the netdevice has the RTC features supports it will create /dev/ptp*

and /sys/class/net/netdev devices.

Quoted 6 lines

All the plumbing that is done is for dual use.
So command etc definitions is for dual use as I explained before. Those can be used for virtio-net device when it has in-built rtc.

When there is only rtc device with its own device-type, same functionality

and commands are there.

Quoted 4 lines

Driver will create only /dev/ptp* device.

From SW point of view it is just matter of layering.. Already done by more than 13+ industry devices..

Quoted 10 lines
Quoted 4 lines

I think there was a misunderstanding. I was trying to say: When it is better to show distinct devices to userspace, would it not also be better to show distinct devices at the Virtio level?

If a rtc functionality is in the virtio-rtc only device, it will be as virtio-rtc device. If the net device has embedded this functionality, the user will still see a distinct device at the OS level. Every OS would create OS level device object to operate on for the underlying hw device.
So it will be fine to end user.

I agree that this is irrelevant for the end user.

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

PCI transport resource improvement goes just beyond PCI transport

improvement. Such as APIC, IOMMU, device registers and more.

Quoted 1 line

This is equally an overkill.

I understand it would be more effort. But to me it looks better as a generic solution, which avoids imposing restrictions on the RTC device.

Quoted 7 lines

There are zero restrictions imposed on the RTC device because the spec is being drafted now. May be I didn’t convey the idea well.

The whole idea is the virtio-rtc can be independent device.
A net-device can have the exact same functionality, so that virtio-rtc functionality can be embedded in it. Thereby, both model co-exists.

I understand. I am just still concerned about the dual use of the spec for the reasons which I gave below.

Quoted 33 lines
Quoted 8 lines
Quoted 1 line

And defining and duplicating exact same commands in net is equally sub-

optimal, specially when we know its need for last several months.

Quoted 2 lines

This is the right junction to reuse functionality on two device types without

duplication.

In my opinion it would still be preferable to have one RTC device, and have the RTC device cooperate with the net device where needed.

I read your below comments for the opinion.
More below.

Quoted 22 lines
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines

E.g., the
RTC device should also get an alarm, with an associated eventq, in the

future.

Quoted 7 lines

Sure, it is ok. it will be optional when it is RTC device.

Quoted 3 lines

Also, I wonder if the driver implementation should then work both as a standalone Virtio driver and as part of the net driver.

Driver layer will be fine. Linux stack evolved over last 3 years to support

Quoted 8 lines
Quoted 8 lines

multi-functionality device.

Quoted 2 lines

A RTC/PTP driver will be a stand alone entity that works on underlying layered

way to issue read cmd, config etc.

Quoted 1 line

As per my above reply, I think it would be better to tackle the problem at the PCI transport level, then there would be no impact on

individual device drivers.

Quoted 1 line
Quoted 1 line
Quoted 9 lines
Quoted 7 lines
Quoted 1 line

There is no impact by better definition usable on two devices.

My main concerns about sharing the spec with the net device are:

- the RTC device would act as a guinea pig, since no other devices share device-specific requirements so far

He he. No it should not. It is only matter of defining cmd opcode and status values that do not overlap one time.

I don't know how spec sharing would be judged by the TC. At the moment, no other devices do this.

Quoted 4 lines
Quoted 2 lines

- a more complex spec impacting two devices

Well it is the functionality, so once you look for slightly larger use cases, overall it is still less complexity than writing two different specs, one for net and one for rtc only device.

Complexity-wise it would be best to have one RTC spec, and link the RTC device to other devices through UUID if needed.

Quoted 7 lines
Quoted 3 lines

- strong coupling between net and RTC driver implementation (or code duplication)

Mostly no. it is the software layer. RTC layer operates over control VQ.
So net device will provide the needed API. Same API would be available for dedicated rtc device lower layer too. No problem.

But Virtio API will usually be time-proven, while drivers certainly have opportunities to get the custom API wrong.

Quoted 6 lines
Quoted 3 lines

- using custom interfaces (between net and RTC driver), rather than generic interfaces

Multi function devices are common in nature for functionality and scale. And OSes supports it already.

Cf. above reply.

Quoted 4 lines
Quoted 2 lines

- RTC could not use device configuration

In your proposal all things are over cvq already.

But it would prevent future use of the device configuration.

Quoted 12 lines
Quoted 3 lines

- unrecognized issues, or future generic changes to the Virtio standard, may inhibit future development of the RTC device or the net device

Unlikely as the reviews eliminate such issues.

Quoted 3 lines

- unclear responsibility and timeline for net device related spec and proof-of-concept

All the responsible people of the technical committee are here.
In fact the requirements are more closely tracked by its stake owners for net compared to other devices. So if you need help to find net side owner, let me know. Others and I are interested to have it in the net side.

With "unclear responsibility" I meant to refer to the responsibility for drafting a spec extension for virtio-net, and for creating a proof-of-concept implementation for the virtio-net extension. As I wrote in my previous email:

Quoted 38 lines

I would prefer to first try to upstream the RTC spec without altering the net spec, while trying to avoid conflicts with the net spec. Maybe most of the RTC spec could then, as a second step, also be applied to the net device. The responsibility for this second step and the related proof of concept work would need to be clarified. To me it still seems better to not share the spec and address resource problems at the transport level.

Quoted 31 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Maybe the above use case could also be addressed by a standalone RTC device as follows:

1) The net device exports a UUID object to which the RTC device can refer to (so the drivers can link the net device to the RTC clock).

2) If needed, the RTC device aligns the lifetime of the clock to the net

device.

Quoted 23 lines
Quoted 23 lines
Quoted 4 lines
Quoted 4 lines

Then, clock lifetimes are no longer static, and can be aligned with the net device. Through an eventq (to be added), the RTC device announces clock creation/destruction, which can be synchronized to the net

device state.

Quoted 3 lines
Quoted 1 line

All of the infrastructure, cross device synchronization, its device side

management will come with equal complexity.

Quoted 2 lines

And we will end up with sub-optimal way to using multiple PCI devices.

As per my above reply, I think it would be better to tackle the problem at the PCI transport level...

Quoted 3 lines

Net already has cvq required by RTC. So cvq commands can be common. Readq is additional, we need to define common between the two and way to

discover it.

Quoted 1 line

I would still try to avoid mixing net and RTC details. RTC should
keep a dedicated cvq.

Quoted 9 lines
Quoted 7 lines
Quoted 1 line

Can you please explain why a dedicate cvq is needed on net device?

I think the concern about redundant queues could be addressed by merging the RTC readq and controlq. At the moment separating these queues would not yield performance benefits, so I think I will just merge the readq and controlq RTC queues in the next spec iteration.

Nice. Ok. lets have single cvq and related opcode range for RTC device and we will place them in the net side too.

OK. In the net device, this queue should still be distinct from the net controlq, which has a less predictable usage pattern than RTC control requests.

Quoted 46 lines
Quoted 6 lines

As to why keeping the net and RTC queues separated: With a dedicated queue, the RTC driver message processing could remain decoupled from the net driver. Also, the net cvq format does not align well with the RTC messages. RTC messages have detailed status codes, while net only has VIRTIO_NET_OK and VIRTIO_NET_ERR.

We can extend the virtio net code too. If they are generic, it is likely that net will also find it useful.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Then the RTC driver could use generic Virtio functionality to send/receive rather than using an interface which is specific to the net driver cvq.

Please explain why?
We have dedicated vqs work ongoing for flow filters due to high rate of

Quoted 9 lines

requests requirement handled by hw.

Quoted 2 lines

So would like to understand if we can extend similar dedicated cvq. Given

most of the queries done via cvq for rts are not time sensitive.

Quoted 1 line

Hence net cvq seems enough.

This should have been addressed by merging the RTC readq and controlq, as per my above reply.

Ok.

Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

Actually, the cross-timestamping support can be specific to the combination of clock and hardware counter. This is explained in more detail in the context of VIRTIO_RTC_M_READ_CROSS. I will mention this

here as well.

Quoted 3 lines
Quoted 1 line

Hardware counter is a misleading term here. It is really platform (system)

Quoted 8 lines
Quoted 8 lines

specific identifier.

Quoted 1 line

Maybe with the currently defined hardware counters there is a one-to-one correspondence to platforms, but that need not be the case. E.g. it might be more convenient for a device on an Intel x86 platform to read the Always Running Timer (ART) rather than the TSC derived from the ART, and then the ART could become another HW counter

in the spec for x86.

Quoted 2 lines
Quoted 1 line

Maybe I am wrong, but it looks like this is platform feature where driver can

device to read ART or ARM timer without any device knowledge?

Quoted 9 lines
Quoted 2 lines

Not sure what is meant by "driver can device".

I meant to say that driver can read the clock from the system clock and device clock.
At that point, cross timestamping convergence point is driver (and not device).
Would that be sufficient?
This way cross timestamping a driver level is still less accurate, but abstracted from the underlying system.

Or such level of accuracy is insufficient?

Depends on the use case. Also, the driver-side cross-timestamping performance will then depend a lot on the Virtio latency (while the device-side cross-timestamping is very robust to Virtio latency).

The driver-side cross-timestamping can already be done now using VIRTIO_RTC_M_READ. Arguably ioctl PTP_SYS_OFFSET_EXTENDED does such cross-timestamping already with the current Linux kernel RFC driver [1], as I described in an earlier email in this thread:

Quoted 5 lines

The Linux kernel has ioctl PTP_SYS_OFFSET_EXTENDED, which will read the system time immediately before and after each PTP clock reading, leaving it up to userspace how to interpret the clock reading. This ioctl can also be used with the RFC Linux kernel driver implementing this spec [...]

But I think from the POV of the spec the driver-side cross-timestamping would then be behavior internal to the driver, so does not need to be discussed in the spec. In this case, Virtio only passes the device clock reading via VIRTIO_RTC_M_READ.

Thank you for your comments,

Peter

[1] https://lore.kernel.org/all/[email protected]/T/#t

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#9 lore raw
Quoted 43 lines

From: Peter Hilber <[email protected]>
Sent: Saturday, September 30, 2023 12:09 AM

Quoted 2 lines · Parav Pandit

Hi Peter,

Hi Parav,

I'm sorry to be a bit late with my answer, since I only received your email via the mailing list, which I do not check so often.

Quoted 5 lines
Quoted 5 lines

From: Peter Hilber <[email protected]>
Sent: Wednesday, September 13, 2023 2:34 PM

Quoted 2 lines

Can you please explain how 64-bit better when the number of clocks per

Quoted 7 lines
Quoted 7 lines

device is not so much?

Quoted 1 line

Even with 256 or 64K clocks device wont care how the ids are issued.

64 bit width should address the potential future case where two clocks get hot- plugged and hot-unplugged at a high rate. If the device hands out a new id each time a clock is re-plugged, and if the driver suspends and resumes during this, an 8 bit id could result in the driver

interpreting ids wrongly.

Quoted 2 lines
Quoted 1 line

In the proposed spec, in a given rtc device, total number of clocks are static at

the device discovery time.

Quoted 2 lines

Clock cannot be hotplug or unplugged dynamically.
Did I understand right?

Yes. The 64 bit width was just to accommodate for potential future extensions. This does not force the device to use effectively 64 bit wide ids. I chose 64 bit over 8 bit because I did not expect that this would become an issue.

Quoted 2 lines

If not, and if the clocks can be added/removed dynamically, if the clock id is

not available, the device will wait to hotplug the new one, until the old one is removed.

Quoted 2 lines

Does one need 64K clocks under single device?
If so, can you please share some thoughts on example application that might

Quoted 8 lines

find it useful?

The example scenario is the one which is quoted above: The device might not know when the driver is suspended, and continuously hand out new ids (when clocks are continuously re-plugged for some reason). If the driver later resumes at the wrong moment, by chance the same id which was in use before might now be assigned to another clock.

If I understand correctly, inv2 we don’t have a way to dynamically hotplug and unplug the clocks of _one_ virtio-rtc device. So am I missing something?
In an arbitrary, interface, once unplug is successful, i.e. when the device exactly know that unplug is completed by the driver, It can re-use the id.

This would work with 8-bit too.
Not that I have issue with 64-bits, but the use case for large number of clocks under single virtio-rtc device is unclear. And if there is no use case, we should have reasonable range.

Quoted 2 lines

This and similar scenarios can be prevented by using 64 bit ids.

Lets first establish a use case if it exists.

Quoted 4 lines

I understand. I am just still concerned about the dual use of the spec for the reasons which I gave below.

It is not the dual use of the spec.
It is the draft that enables common functionality for using in two use cases.

Quoted 4 lines

I don't know how spec sharing would be judged by the TC. At the moment, no other devices do this.

This is the first time two use cases has such a common functionality that brings. So it is fine to introduce.
TC will evaluate and give their feedback that we can incorporate further. Since this reduces
a. dual spec
b. dual implementation

and improves better device scaling, it is logical to cover both use cases.

Quoted 12 lines
Quoted 3 lines
Quoted 2 lines

- a more complex spec impacting two devices

Well it is the functionality, so once you look for slightly larger use cases, overall

it is still less complexity than writing two different specs, one for net and one for rtc only device.

Quoted 1 line

Complexity-wise it would be best to have one RTC spec, and link the RTC device to other devices through UUID if needed.

I am looking at larger use case than just rtc device.
Hence, total complexity of net device and rtc is counted in spec, and also in implementation.

So it looks better to do one time.

Quoted 12 lines
Quoted 5 lines
Quoted 3 lines

- strong coupling between net and RTC driver implementation (or code duplication)

Mostly no. it is the software layer. RTC layer operates over control VQ. So net device will provide the needed API. Same API would be available for

dedicated rtc device lower layer too.

Quoted 2 lines

No problem.

But Virtio API will usually be time-proven, while drivers certainly have opportunities to get the custom API wrong.

I do not understand this much.
The bottom line is common functionality exists, driver maintainers can craft good APIs in their OS. There are more than 13 drivers in one OS that I inspected, which has been peer reviewed. So I wouldn’t claim that they all got it wrong.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

- using custom interfaces (between net and RTC driver), rather than generic interfaces

Multi function devices are common in nature for functionality and scale. And OSes supports it already.

Quoted 9 lines

Cf. above reply.

Quoted 4 lines
Quoted 2 lines

- RTC could not use device configuration

In your proposal all things are over cvq already.

But it would prevent future use of the device configuration.

Not really. cvq has the commands to do the device config in your patch already. Even for net device, implementing RW registers are hard.
The industry in various aspects are moving away from device specific configurations.

Quoted 10 lines
Quoted 10 lines
Quoted 3 lines

- unrecognized issues, or future generic changes to the Virtio standard, may inhibit future development of the RTC device or the net device

Unlikely as the reviews eliminate such issues.

Quoted 3 lines

- unclear responsibility and timeline for net device related spec and proof-of-concept

All the responsible people of the technical committee are here. In fact the requirements are more closely tracked by its stake owners for net

Quoted 7 lines

compared to other devices.

Quoted 1 line

So if you need help to find net side owner, let me know. Others and I are

interested to have it in the net side.

Quoted 1 line

With "unclear responsibility" I meant to refer to the responsibility for drafting a spec extension for virtio-net,

You can count me to draft the virtio-net spec for responsibility.
I am sure others would join as there is common interest across multiple device implementors to support it. We know at least 4 stake holders have interests in it in virtio-net (Alibaba, google, marvell, nvidia).

Quoted 48 lines
Quoted 7 lines

I would prefer to first try to upstream the RTC spec without altering the net spec, while trying to avoid conflicts with the net spec. Maybe most of the RTC spec could then, as a second step, also be applied to the net device. The responsibility for this second step and the related proof of concept work would need to be clarified. To me it still seems better to not share the spec and address resource problems at the

transport level.

Quoted 39 lines
Quoted 39 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Maybe the above use case could also be addressed by a standalone RTC device as follows:

1) The net device exports a UUID object to which the RTC device can refer to (so the drivers can link the net device to the RTC clock).

2) If needed, the RTC device aligns the lifetime of the clock to the net

device.

Quoted 18 lines
Quoted 18 lines
Quoted 4 lines
Quoted 4 lines

Then, clock lifetimes are no longer static, and can be aligned with the net device. Through an eventq (to be added), the RTC device announces clock creation/destruction, which can be synchronized to the net

device state.

Quoted 3 lines
Quoted 1 line

All of the infrastructure, cross device synchronization, its device side

management will come with equal complexity.

Quoted 2 lines

And we will end up with sub-optimal way to using multiple PCI devices.

As per my above reply, I think it would be better to tackle the problem at the PCI transport level...

Quoted 3 lines

Net already has cvq required by RTC. So cvq commands can be common. Readq is additional, we need to define common between the two and way to

Quoted 6 lines
Quoted 5 lines

discover it.

Quoted 1 line

I would still try to avoid mixing net and RTC details. RTC should
keep a dedicated cvq.

Can you please explain why a dedicate cvq is needed on net device?

I think the concern about redundant queues could be addressed by merging the RTC readq and controlq. At the moment separating these queues would not yield performance benefits, so I think I will just merge the readq and controlq RTC queues in the next spec iteration.

Quoted 7 lines
Quoted 1 line

Nice. Ok. lets have single cvq and related opcode range for RTC device and we

will place them in the net side too.

Quoted 1 line

OK. In the net device, this queue should still be distinct from the net controlq, which has a less predictable usage pattern than RTC control requests.

As you describe that with cross timestamping, the predictability is not important. Or did I misunderstand you?

#10 lore raw
Quoted 31 lines · Parav Pandit
Quoted 31 lines
Quoted 5 lines
Quoted 5 lines

From: Peter Hilber <[email protected]>
Sent: Wednesday, September 13, 2023 2:34 PM

Quoted 2 lines

Can you please explain how 64-bit better when the number of clocks per

Quoted 7 lines
Quoted 7 lines

device is not so much?

Quoted 1 line

Even with 256 or 64K clocks device wont care how the ids are issued.

64 bit width should address the potential future case where two clocks get hot- plugged and hot-unplugged at a high rate. If the device hands out a new id each time a clock is re-plugged, and if the driver suspends and resumes during this, an 8 bit id could result in the driver

interpreting ids wrongly.

Quoted 2 lines
Quoted 1 line

In the proposed spec, in a given rtc device, total number of clocks are static at

the device discovery time.

Quoted 2 lines

Clock cannot be hotplug or unplugged dynamically.
Did I understand right?

Yes. The 64 bit width was just to accommodate for potential future extensions. This does not force the device to use effectively 64 bit wide ids. I chose 64 bit over 8 bit because I did not expect that this would become an issue.

Quoted 2 lines

If not, and if the clocks can be added/removed dynamically, if the clock id is

not available, the device will wait to hotplug the new one, until the old one is removed.

Quoted 2 lines

Does one need 64K clocks under single device?
If so, can you please share some thoughts on example application that might

find it useful?

Quoted 7 lines
Quoted 6 lines

The example scenario is the one which is quoted above: The device might not know when the driver is suspended, and continuously hand out new ids (when clocks are continuously re-plugged for some reason). If the driver later resumes at the wrong moment, by chance the same id which was in use before might now be assigned to another clock.

If I understand correctly, inv2 we don’t have a way to dynamically hotplug and unplug the clocks of _one_ virtio-rtc device.

Hi Parav,

correct.

Quoted 3 lines

So am I missing something?
In an arbitrary, interface, once unplug is successful, i.e. when the device exactly know that unplug is completed by the driver, It can re-use the id.

But this would require the device to remember previously unplugged clocks.

Quoted 5 lines

This would work with 8-bit too.
Not that I have issue with 64-bits, but the use case for large number of clocks under single virtio-rtc device is unclear. And if there is no use case, we should have reasonable range.

Using a large id range might help in the future to avoid dealing with the whole class of id ABA problems, cf. the above example.

To me using 64 bits to avoid the potential future ABA problems looks reasonable, but if others disagree, I can reduce to 8 bits.

Quoted 4 lines
Quoted 2 lines

This and similar scenarios can be prevented by using 64 bit ids.

Lets first establish a use case if it exists.

The benefit would be to avoid potential future id ABA problems, as per above.

Quoted 92 lines
Quoted 4 lines

I understand. I am just still concerned about the dual use of the spec for the reasons which I gave below.

It is not the dual use of the spec.
It is the draft that enables common functionality for using in two use cases.

Quoted 4 lines

I don't know how spec sharing would be judged by the TC. At the moment, no other devices do this.

This is the first time two use cases has such a common functionality that brings. So it is fine to introduce.
TC will evaluate and give their feedback that we can incorporate further. Since this reduces
a. dual spec
b. dual implementation

and improves better device scaling, it is logical to cover both use cases.

Quoted 12 lines
Quoted 3 lines
Quoted 2 lines

- a more complex spec impacting two devices

Well it is the functionality, so once you look for slightly larger use cases, overall

it is still less complexity than writing two different specs, one for net and one for rtc only device.

Quoted 1 line

Complexity-wise it would be best to have one RTC spec, and link the RTC device to other devices through UUID if needed.

I am looking at larger use case than just rtc device.
Hence, total complexity of net device and rtc is counted in spec, and also in implementation.

So it looks better to do one time.

Quoted 12 lines
Quoted 5 lines
Quoted 3 lines

- strong coupling between net and RTC driver implementation (or code duplication)

Mostly no. it is the software layer. RTC layer operates over control VQ. So net device will provide the needed API. Same API would be available for

dedicated rtc device lower layer too.

Quoted 2 lines

No problem.

But Virtio API will usually be time-proven, while drivers certainly have opportunities to get the custom API wrong.

I do not understand this much.
The bottom line is common functionality exists, driver maintainers can craft good APIs in their OS. There are more than 13 drivers in one OS that I inspected, which has been peer reviewed. So I wouldn’t claim that they all got it wrong.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

- using custom interfaces (between net and RTC driver), rather than generic interfaces

Multi function devices are common in nature for functionality and scale. And OSes supports it already.

Quoted 9 lines

Cf. above reply.

Quoted 4 lines
Quoted 2 lines

- RTC could not use device configuration

In your proposal all things are over cvq already.

But it would prevent future use of the device configuration.

Not really. cvq has the commands to do the device config in your patch already. Even for net device, implementing RW registers are hard.
The industry in various aspects are moving away from device specific configurations.

Quoted 15 lines
Quoted 10 lines
Quoted 3 lines

- unrecognized issues, or future generic changes to the Virtio standard, may inhibit future development of the RTC device or the net device

Unlikely as the reviews eliminate such issues.

Quoted 3 lines

- unclear responsibility and timeline for net device related spec and proof-of-concept

All the responsible people of the technical committee are here. In fact the requirements are more closely tracked by its stake owners for net

compared to other devices.

Quoted 1 line

So if you need help to find net side owner, let me know. Others and I are

interested to have it in the net side.

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

With "unclear responsibility" I meant to refer to the responsibility for drafting a spec extension for virtio-net,

You can count me to draft the virtio-net spec for responsibility.
I am sure others would join as there is common interest across multiple device implementors to support it. We know at least 4 stake holders have interests in it in virtio-net (Alibaba, google, marvell, nvidia).

OK, thank you for the information.

I think the arguments on how to have the RTC spec interact with the net device are now on the table. Maybe the discussion could continue on a draft spec extension for the net device?

Quoted 49 lines
Quoted 49 lines
Quoted 7 lines

I would prefer to first try to upstream the RTC spec without altering the net spec, while trying to avoid conflicts with the net spec. Maybe most of the RTC spec could then, as a second step, also be applied to the net device. The responsibility for this second step and the related proof of concept work would need to be clarified. To me it still seems better to not share the spec and address resource problems at the

transport level.

Quoted 40 lines
Quoted 30 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Maybe the above use case could also be addressed by a standalone RTC device as follows:

1) The net device exports a UUID object to which the RTC device can refer to (so the drivers can link the net device to the RTC clock).

2) If needed, the RTC device aligns the lifetime of the clock to the net

device.

Quoted 21 lines
Quoted 21 lines
Quoted 4 lines
Quoted 4 lines

Then, clock lifetimes are no longer static, and can be aligned with the net device. Through an eventq (to be added), the RTC device announces clock creation/destruction, which can be synchronized to the net

device state.

Quoted 3 lines
Quoted 1 line

All of the infrastructure, cross device synchronization, its device side

management will come with equal complexity.

Quoted 2 lines

And we will end up with sub-optimal way to using multiple PCI devices.

As per my above reply, I think it would be better to tackle the problem at the PCI transport level...

Quoted 3 lines

Net already has cvq required by RTC. So cvq commands can be common. Readq is additional, we need to define common between the two and way to

discover it.

Quoted 1 line
Quoted 9 lines
Quoted 3 lines
Quoted 2 lines

I would still try to avoid mixing net and RTC details. RTC should
keep a dedicated cvq.

Can you please explain why a dedicate cvq is needed on net device?

I think the concern about redundant queues could be addressed by merging the RTC readq and controlq. At the moment separating these queues would not yield performance benefits, so I think I will just merge the readq and controlq RTC queues in the next spec iteration.

Nice. Ok. lets have single cvq and related opcode range for RTC device and we

Quoted 8 lines
Quoted 6 lines

will place them in the net side too.

Quoted 1 line

OK. In the net device, this queue should still be distinct from the net controlq, which has a less predictable usage pattern than RTC control requests.

As you describe that with cross timestamping, the predictability is not important. Or did I misunderstand you?

Cross-timestamping is very robust to response latency, if the driver timekeeping is good enough. When not using cross-timestamping, the response latency is still important.

The expected RTC control requests pattern is that the driver issues all of them during device probing. For the net device, it should not be assumed that the driver makes all net control requests during device probing.

So in some use cases sharing the net controlq with RTC non-control requests could still be problematic.

Plus, the name "controlq" isn't really appropriate for non-control clock reading requests, which the driver might issue several times per second. Existing net controlq implementations might not match such a use case well (the Linux driver is busy waiting on controlq responses).

Thank you for your comments!

Best regards,

Peter

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#11 lore raw
Quoted 9 lines

From: Peter Hilber <[email protected]>
Sent: Thursday, October 12, 2023 4:26 PM

Quoted 1 line

If I understand correctly, inv2 we don’t have a way to dynamically hotplug and

unplug the clocks of _one_ virtio-rtc device.

Hi Parav,

Quoted 9 lines

correct.

Quoted 3 lines

So am I missing something?
In an arbitrary, interface, once unplug is successful, i.e. when the device exactly know that unplug is completed by the driver, It can re-use the

id.

But this would require the device to remember previously unplugged clocks.

Not always, it can do when the clock wrap arounds.
Since this interface does not have the complexity of hotplug and unplug of the clocks, we should just keep it simple to a reasonable number of clocks. The 64-bit value is not hurting at all, I just feel it is built for unknown.

Any reason to not do hot plug/unplug of clocks now?

Quoted 9 lines
Quoted 3 lines

This would work with 8-bit too.
Not that I have issue with 64-bits, but the use case for large number of clocks

under single virtio-rtc device is unclear.

Quoted 2 lines

And if there is no use case, we should have reasonable range.

Using a large id range might help in the future to avoid dealing with the whole class of id ABA problems, cf. the above example.

In my view the clocks are expected to give precision and accuracy, so number of clocks in a system that occupies 64-bit wide sounds distant. If it is used to not_care about finding the free id, it is ok.

Is hotplug unplug the clock (instead of device) is critical that drives the clock id?

Quoted 4 lines

To me using 64 bits to avoid the potential future ABA problems looks reasonable, but if others disagree, I can reduce to 8 bits.

It is not must, I just find the 64-bit level clocks a bit too much specially at hotplug/unplug part.

Quoted 118 lines
Quoted 4 lines
Quoted 2 lines

This and similar scenarios can be prevented by using 64 bit ids.

Lets first establish a use case if it exists.

The benefit would be to avoid potential future id ABA problems, as per above.

Quoted 4 lines
Quoted 4 lines

I understand. I am just still concerned about the dual use of the spec for the reasons which I gave below.

Quoted 8 lines

It is not the dual use of the spec.
It is the draft that enables common functionality for using in two use cases.

Quoted 4 lines

I don't know how spec sharing would be judged by the TC. At the moment, no other devices do this.

This is the first time two use cases has such a common functionality that

brings.

Quoted 19 lines

So it is fine to introduce.
TC will evaluate and give their feedback that we can incorporate further. Since this reduces
a. dual spec
b. dual implementation

and improves better device scaling, it is logical to cover both use cases.

Quoted 10 lines
Quoted 4 lines
Quoted 2 lines

- a more complex spec impacting two devices

Well it is the functionality, so once you look for slightly larger use cases, overall

it is still less complexity than writing two different specs, one for net and one for rtc only device.

Quoted 1 line
Quoted 5 lines
Quoted 3 lines

Complexity-wise it would be best to have one RTC spec, and link the RTC device to other devices through UUID if needed.

I am looking at larger use case than just rtc device.
Hence, total complexity of net device and rtc is counted in spec, and also in

implementation.

Quoted 12 lines

So it looks better to do one time.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

- strong coupling between net and RTC driver implementation (or code duplication)

Mostly no. it is the software layer. RTC layer operates over control VQ. So net device will provide the needed API. Same API would be available for

dedicated rtc device lower layer too.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

No problem.

But Virtio API will usually be time-proven, while drivers certainly have opportunities to get the custom API wrong.

I do not understand this much.
The bottom line is common functionality exists, driver maintainers can craft

good APIs in their OS.

Quoted 1 line

There are more than 13 drivers in one OS that I inspected, which has been

peer reviewed.

Quoted 3 lines

So I wouldn’t claim that they all got it wrong.

Quoted 1 line
Quoted 1 line
Quoted 1 line

- using custom interfaces (between net and RTC driver), rather than

generic

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

interfaces

Multi function devices are common in nature for functionality and scale. And OSes supports it already.

Cf. above reply.

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

- RTC could not use device configuration

In your proposal all things are over cvq already.

But it would prevent future use of the device configuration.

Not really. cvq has the commands to do the device config in your patch

already.

Quoted 2 lines

Even for net device, implementing RW registers are hard. The industry in various aspects are moving away from device specific

configurations.

Quoted 16 lines
Quoted 16 lines
Quoted 5 lines
Quoted 5 lines

- unrecognized issues, or future generic changes to the Virtio standard, may inhibit future development of the RTC device or the net
device

Quoted 8 lines

Unlikely as the reviews eliminate such issues.

Quoted 3 lines

- unclear responsibility and timeline for net device related spec and proof-of-concept

All the responsible people of the technical committee are here. In fact the requirements are more closely tracked by its stake owners for net

compared to other devices.

Quoted 2 lines

So if you need help to find net side owner, let me know. Others and I are

Quoted 8 lines
Quoted 5 lines

interested to have it in the net side.

Quoted 1 line

With "unclear responsibility" I meant to refer to the responsibility for drafting a spec extension for virtio-net,

You can count me to draft the virtio-net spec for responsibility. I am sure others would join as there is common interest across multiple device

implementors to support it.

Quoted 1 line

We know at least 4 stake holders have interests in it in virtio-net (Alibaba,

Quoted 9 lines

google, marvell, nvidia).

Quoted 1 line

OK, thank you for the information.

I think the arguments on how to have the RTC spec interact with the net device are now on the table. Maybe the discussion could continue on a draft spec extension for the net device?

I didn’t follow the comment.
My suggestion is lets draft the spec _now_ that covers rtc functionality as part of virtio-net device and as independent rtc device. And if that requires me to work and draft the net part for you, I am here to draft it.

Would it work for you?

Quoted 67 lines
Quoted 27 lines
Quoted 27 lines
Quoted 8 lines

I would prefer to first try to upstream the RTC spec without altering the net spec, while trying to avoid conflicts with the net spec. Maybe most of the RTC spec could then, as a second step, also be applied to the net device. The responsibility for this second step and the related proof of concept work would need to be clarified. To me it still seems better to not share the spec and address resource problems at the

transport level.

Quoted 17 lines
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Maybe the above use case could also be addressed by a standalone RTC device as follows:

1) The net device exports a UUID object to which the RTC device can refer to (so the drivers can link the net device to the RTC clock).

2) If needed, the RTC device aligns the lifetime of the clock to the net

device.

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

Then, clock lifetimes are no longer static, and can be aligned with the net device. Through an eventq (to be added), the RTC device announces clock creation/destruction, which can be synchronized to the net

device state.

Quoted 3 lines
Quoted 1 line

All of the infrastructure, cross device synchronization, its device side

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

management will come with equal complexity.

Quoted 2 lines

And we will end up with sub-optimal way to using multiple PCI devices.

As per my above reply, I think it would be better to tackle the problem at the PCI transport level...

Quoted 1 line

Net already has cvq required by RTC. So cvq commands can be

common.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Readq is additional, we need to define common between the two and way to

discover it.

Quoted 1 line

I would still try to avoid mixing net and RTC details. RTC
should keep a dedicated cvq.

Can you please explain why a dedicate cvq is needed on net device?

I think the concern about redundant queues could be addressed by merging the RTC readq and controlq. At the moment separating these queues would not yield performance benefits, so I think I will just merge the readq and controlq RTC queues in the next spec iteration.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Nice. Ok. lets have single cvq and related opcode range for RTC device and we

will place them in the net side too.

Quoted 1 line

OK. In the net device, this queue should still be distinct from the net controlq, which has a less predictable usage pattern than RTC control

requests.

Quoted 2 lines
Quoted 1 line

As you describe that with cross timestamping, the predictability is not

important.

Quoted 1 line

Or did I misunderstand you?

Cross-timestamping is very robust to response latency, if the driver timekeeping is good enough. When not using cross-timestamping, the response latency is still important.

Quoted 7 lines

The expected RTC control requests pattern is that the driver issues all of them during device probing. For the net device, it should not be assumed that the driver makes all net control requests during device probing.

So in some use cases sharing the net controlq with RTC non-control requests could still be problematic.

Net can issue when it prefers to on the ctrlq.
Rtc can issue at driver probe time on the ctrlq.

Quoted 2 lines

Plus, the name "controlq" isn't really appropriate for non-control clock reading requests, which the driver might issue several times per second.

Net has control q for query and set. Statistics commands are added to the ctrlq as well. It is single queue symmetric interface for get and set.

Quoted 3 lines

Existing net controlq implementations might not match such a use case well (the Linux driver is busy waiting on controlq responses).

Oh, that can change surely change for the new use cases. Should not be a problem. I remember there were some patches too to improve that.

Quoted 5 lines

Thank you for your comments!

Best regards,

Peter

#12 lore raw
Quoted 5 lines · Parav Pandit
Quoted 4 lines

From: Peter Hilber <[email protected]

Sent: Thursday, October 12, 2023 4:26 PM

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

So am I missing something?
In an arbitrary, interface, once unplug is successful, i.e. when the device exactly know that unplug is completed by the driver, It can re-use the

id.

But this would require the device to remember previously unplugged clocks.

Not always, it can do when the clock wrap arounds.

Sorry, I do not understand the reply. When the clock id is wrapping around, in the discussed scenario the device would need to know which clock had assigned id 0 in the past to avoid potential misinterpretation by the driver.

Quoted 5 lines

Since this interface does not have the complexity of hotplug and unplug of the clocks, we should just keep it simple to a reasonable number of clocks. The 64-bit value is not hurting at all, I just feel it is built for unknown.

Any reason to not do hot plug/unplug of clocks now?

Current use cases do not require hotplugging, but I tried to make the spec future-proof.

Quoted 12 lines
Quoted 9 lines
Quoted 3 lines

This would work with 8-bit too.
Not that I have issue with 64-bits, but the use case for large number of clocks

under single virtio-rtc device is unclear.

Quoted 2 lines

And if there is no use case, we should have reasonable range.

Using a large id range might help in the future to avoid dealing with the whole class of id ABA problems, cf. the above example.

In my view the clocks are expected to give precision and accuracy, so number of clocks in a system that occupies 64-bit wide sounds distant. If it is used to not_care about finding the free id, it is ok.

Not caring about finding a concurrency-safe free id was the reason for 64 bits width.

Quoted 2 lines

Is hotplug unplug the clock (instead of device) is critical that drives the clock id?

Clock id width > 8 bit was only considered to be able to easily support hotplugging in the future.

Quoted 24 lines
Quoted 4 lines

To me using 64 bits to avoid the potential future ABA problems looks reasonable, but if others disagree, I can reduce to 8 bits.

It is not must, I just find the 64-bit level clocks a bit too much specially at hotplug/unplug part.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

With "unclear responsibility" I meant to refer to the responsibility for drafting a spec extension for virtio-net,

You can count me to draft the virtio-net spec for responsibility. I am sure others would join as there is common interest across multiple device

implementors to support it.

Quoted 1 line

We know at least 4 stake holders have interests in it in virtio-net (Alibaba,

Quoted 9 lines

google, marvell, nvidia).

Quoted 1 line

OK, thank you for the information.

I think the arguments on how to have the RTC spec interact with the net device are now on the table. Maybe the discussion could continue on a draft spec extension for the net device?

Quoted 6 lines

I didn’t follow the comment.
My suggestion is lets draft the spec _now_ that covers rtc functionality as part of virtio-net device and as independent rtc device. And if that requires me to work and draft the net part for you, I am here to draft it.

Would it work for you?

Of course, work on the net part could also start now.

I would prefer if the net part could be discussed as a dedicated patch, on top of the patch(es) introducing the RTC device (and referring to those).

Quoted 48 lines
Quoted 32 lines
Quoted 15 lines
Quoted 15 lines
Quoted 7 lines

I would prefer to first try to upstream the RTC spec without altering the net spec, while trying to avoid conflicts with the net spec. Maybe most of the RTC spec could then, as a second step, also be applied to the net device. The responsibility for this second step and the related proof of concept work would need to be clarified. To me it still seems better to not share the spec and address resource problems at the

transport level.

Quoted 5 lines
Quoted 5 lines

I think the concern about redundant queues could be addressed by merging the RTC readq and controlq. At the moment separating these queues would not yield performance benefits, so I think I will just merge the readq and controlq RTC queues in the next spec iteration.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Nice. Ok. lets have single cvq and related opcode range for RTC device and we

will place them in the net side too.

Quoted 1 line

OK. In the net device, this queue should still be distinct from the net controlq, which has a less predictable usage pattern than RTC control

requests.

Quoted 2 lines
Quoted 1 line

As you describe that with cross timestamping, the predictability is not

important.

Quoted 1 line

Or did I misunderstand you?

Cross-timestamping is very robust to response latency, if the driver timekeeping is good enough. When not using cross-timestamping, the response latency is still important.

Quoted 7 lines

The expected RTC control requests pattern is that the driver issues all of them during device probing. For the net device, it should not be assumed that the driver makes all net control requests during device probing.

So in some use cases sharing the net controlq with RTC non-control requests could still be problematic.

Net can issue when it prefers to on the ctrlq.
Rtc can issue at driver probe time on the ctrlq.

Quoted 2 lines

Plus, the name "controlq" isn't really appropriate for non-control clock reading requests, which the driver might issue several times per second.

Net has control q for query and set. Statistics commands are added to the ctrlq as well. It is single queue symmetric interface for get and set.

Quoted 6 lines
Quoted 3 lines

Existing net controlq implementations might not match such a use case well (the Linux driver is busy waiting on controlq responses).

Oh, that can change surely change for the new use cases. Should not be a problem. I remember there were some patches too to improve that.

OK, so I assume you still want to use the net controlq for all RTC requests.

Thanks for the reply,

Peter

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#13 lore raw
Quoted 24 lines

From: Peter Hilber <[email protected]>
Sent: Friday, October 13, 2023 9:00 PM

Quoted 9 lines · Parav Pandit
Quoted 9 lines

From: Peter Hilber <[email protected]

Sent: Thursday, October 12, 2023 4:26 PM

Quoted 4 lines

So am I missing something?
In an arbitrary, interface, once unplug is successful, i.e. when the device exactly know that unplug is completed by the driver, It can re-use the

Quoted 5 lines
Quoted 4 lines

id.

But this would require the device to remember previously unplugged clocks.

Not always, it can do when the clock wrap arounds.

Sorry, I do not understand the reply. When the clock id is wrapping around, in the discussed scenario the device would need to know which clock had assigned id 0 in the past to avoid potential misinterpretation by the driver.

Quoted 1 line

Since this interface does not have the complexity of hotplug and unplug of the

Quoted 9 lines

clocks, we should just keep it simple to a reasonable number of clocks.

Quoted 4 lines

The 64-bit value is not hurting at all, I just feel it is built for unknown.

Any reason to not do hot plug/unplug of clocks now?

Current use cases do not require hotplugging, but I tried to make the spec future-proof.

If there is no use case for hotplugging a clock, we should keep things simple to a reasonable width.

Quoted 58 lines
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines

This would work with 8-bit too.
Not that I have issue with 64-bits, but the use case for large number of clocks

under single virtio-rtc device is unclear.

Quoted 2 lines

And if there is no use case, we should have reasonable range.

Using a large id range might help in the future to avoid dealing with the whole class of id ABA problems, cf. the above example.

In my view the clocks are expected to give precision and accuracy, so number

of clocks in a system that occupies 64-bit wide sounds distant.

Quoted 2 lines

If it is used to not_care about finding the free id, it is ok.

Not caring about finding a concurrency-safe free id was the reason for 64 bits width.

Quoted 1 line

Is hotplug unplug the clock (instead of device) is critical that drives the clock

id?

Quoted 1 line

Clock id width > 8 bit was only considered to be able to easily support hotplugging in the future.

Quoted 5 lines
Quoted 4 lines

To me using 64 bits to avoid the potential future ABA problems looks reasonable, but if others disagree, I can reduce to 8 bits.

It is not must, I just find the 64-bit level clocks a bit too much specially at

hotplug/unplug part.

Quoted 15 lines
Quoted 15 lines
Quoted 9 lines
Quoted 5 lines

With "unclear responsibility" I meant to refer to the responsibility for drafting a spec extension for virtio-net,

You can count me to draft the virtio-net spec for responsibility. I am sure others would join as there is common interest across multiple device

implementors to support it.

Quoted 2 lines

We know at least 4 stake holders have interests in it in virtio-net (Alibaba,

google, marvell, nvidia).

Quoted 1 line
Quoted 8 lines
Quoted 6 lines

OK, thank you for the information.

I think the arguments on how to have the RTC spec interact with the net device are now on the table. Maybe the discussion could continue on a draft spec extension for the net device?

I didn’t follow the comment.
My suggestion is lets draft the spec _now_ that covers rtc functionality as part

of virtio-net device and as independent rtc device.

Quoted 1 line

And if that requires me to work and draft the net part for you, I am here to

draft it.

Quoted 1 line
Quoted 8 lines
Quoted 2 lines

Would it work for you?

Of course, work on the net part could also start now.

I would prefer if the net part could be discussed as a dedicated patch, on top of the patch(es) introducing the RTC device (and referring to those).

It will be separate patch (or patches) in device-type/net/description.tex.
It should be part of same patch series as rtc and net so both devices agree and reserve necessary opcode ranges etc to utilize same infra.

Quoted 56 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

I would prefer to first try to upstream the RTC spec without altering the net spec, while trying to avoid conflicts with the net spec. Maybe most of the RTC spec could then, as a second step, also be applied to the net device. The responsibility for this second step and the related proof of concept work would need to be clarified. To me it still seems better to not share the spec and address resource problems at the

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

transport level.

Quoted 4 lines
Quoted 4 lines

I think the concern about redundant queues could be addressed by merging the RTC readq and controlq. At the moment separating these queues would not yield performance benefits, so I think I will just merge the readq and controlq RTC queues in the next spec

iteration.

Quoted 16 lines
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
Quoted 1 line

Nice. Ok. lets have single cvq and related opcode range for RTC device and we

will place them in the net side too.

Quoted 1 line

OK. In the net device, this queue should still be distinct from the net controlq, which has a less predictable usage pattern than RTC control

requests.

Quoted 3 lines
Quoted 1 line

As you describe that with cross timestamping, the predictability is not

important.

Quoted 1 line

Or did I misunderstand you?

Quoted 7 lines
Quoted 7 lines

Cross-timestamping is very robust to response latency, if the driver timekeeping is good enough. When not using cross-timestamping, the response latency is still important.

The expected RTC control requests pattern is that the driver issues all of them during device probing. For the net device, it should not be assumed that the driver makes all net control requests during device

probing.

Quoted 9 lines
Quoted 4 lines

So in some use cases sharing the net controlq with RTC non-control requests could still be problematic.

Net can issue when it prefers to on the ctrlq.
Rtc can issue at driver probe time on the ctrlq.

Quoted 2 lines

Plus, the name "controlq" isn't really appropriate for non-control clock reading requests, which the driver might issue several times per

second.

Quoted 7 lines

Net has control q for query and set. Statistics commands are added to the ctrlq as well. It is single queue symmetric interface for get and set.

Quoted 3 lines

Existing net controlq implementations might not match such a use case well (the Linux driver is busy waiting on controlq responses).

Quoted 7 lines
Quoted 1 line

Oh, that can change surely change for the new use cases. Should not be a

problem.

Quoted 2 lines

I remember there were some patches too to improve that.

OK, so I assume you still want to use the net controlq for all RTC requests.

Yes.

#14 lore raw
Quoted 27 lines · Parav Pandit
Quoted 27 lines

From: Peter Hilber <[email protected]>
Sent: Friday, October 13, 2023 9:00 PM

Quoted 9 lines · Parav Pandit
Quoted 9 lines

From: Peter Hilber <[email protected]

Sent: Thursday, October 12, 2023 4:26 PM

Quoted 4 lines

So am I missing something?
In an arbitrary, interface, once unplug is successful, i.e. when the device exactly know that unplug is completed by the driver, It can re-use the

Quoted 5 lines
Quoted 4 lines

id.

But this would require the device to remember previously unplugged clocks.

Not always, it can do when the clock wrap arounds.

Sorry, I do not understand the reply. When the clock id is wrapping around, in the discussed scenario the device would need to know which clock had assigned id 0 in the past to avoid potential misinterpretation by the driver.

Quoted 1 line

Since this interface does not have the complexity of hotplug and unplug of the

clocks, we should just keep it simple to a reasonable number of clocks.

Quoted 2 lines

The 64-bit value is not hurting at all, I just feel it is built for unknown.

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

Any reason to not do hot plug/unplug of clocks now?

Current use cases do not require hotplugging, but I tried to make the spec future-proof.

If there is no use case for hotplugging a clock, we should keep things simple to a reasonable width.

I will change the width to 8 bit, although I don't think that makes things simpler.

Quoted 61 lines
Quoted 61 lines
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines

This would work with 8-bit too.
Not that I have issue with 64-bits, but the use case for large number of clocks

under single virtio-rtc device is unclear.

Quoted 2 lines

And if there is no use case, we should have reasonable range.

Using a large id range might help in the future to avoid dealing with the whole class of id ABA problems, cf. the above example.

In my view the clocks are expected to give precision and accuracy, so number

of clocks in a system that occupies 64-bit wide sounds distant.

Quoted 2 lines

If it is used to not_care about finding the free id, it is ok.

Not caring about finding a concurrency-safe free id was the reason for 64 bits width.

Quoted 1 line

Is hotplug unplug the clock (instead of device) is critical that drives the clock

id?

Quoted 1 line

Clock id width > 8 bit was only considered to be able to easily support hotplugging in the future.

Quoted 5 lines
Quoted 4 lines

To me using 64 bits to avoid the potential future ABA problems looks reasonable, but if others disagree, I can reduce to 8 bits.

It is not must, I just find the 64-bit level clocks a bit too much specially at

hotplug/unplug part.

Quoted 15 lines
Quoted 15 lines
Quoted 9 lines
Quoted 5 lines

With "unclear responsibility" I meant to refer to the responsibility for drafting a spec extension for virtio-net,

You can count me to draft the virtio-net spec for responsibility. I am sure others would join as there is common interest across multiple device

implementors to support it.

Quoted 2 lines

We know at least 4 stake holders have interests in it in virtio-net (Alibaba,

google, marvell, nvidia).

Quoted 1 line
Quoted 8 lines
Quoted 6 lines

OK, thank you for the information.

I think the arguments on how to have the RTC spec interact with the net device are now on the table. Maybe the discussion could continue on a draft spec extension for the net device?

I didn’t follow the comment.
My suggestion is lets draft the spec _now_ that covers rtc functionality as part

of virtio-net device and as independent rtc device.

Quoted 1 line

And if that requires me to work and draft the net part for you, I am here to

draft it.

Quoted 3 lines

Would it work for you?

Quoted 8 lines
Quoted 5 lines

Of course, work on the net part could also start now.

I would prefer if the net part could be discussed as a dedicated patch, on top of the patch(es) introducing the RTC device (and referring to those).

It will be separate patch (or patches) in device-type/net/description.tex.
It should be part of same patch series as rtc and net so both devices agree and reserve necessary opcode ranges etc to utilize same infra.

I plan to send out RTC spec RFC v3, which should contain all the agreed changes from the v2 discussion, in November. Maybe the net related changes could then be put on top of that (and become part of a common RFC v4)?

Best regards,

Peter

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#15 lore raw
Quoted 45 lines

From: Peter Hilber <[email protected]>
Sent: Tuesday, October 17, 2023 9:36 PM
To: Parav Pandit <[email protected]>; [email protected]
Cc: Cornelia Huck <[email protected]>
Subject: Re: [virtio-comment] [RFC PATCH v2] virtio-rtc: Add device specification.

Quoted 9 lines · Parav Pandit
Quoted 9 lines

From: Peter Hilber <[email protected]>
Sent: Friday, October 13, 2023 9:00 PM

Quoted 5 lines · Parav Pandit
Quoted 4 lines

From: Peter Hilber <[email protected]

Sent: Thursday, October 12, 2023 4:26 PM

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

So am I missing something?
In an arbitrary, interface, once unplug is successful, i.e. when the device exactly know that unplug is completed by the driver, It can re-use the

id.

But this would require the device to remember previously unplugged

clocks.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 1 line

Not always, it can do when the clock wrap arounds.

Sorry, I do not understand the reply. When the clock id is wrapping around, in the discussed scenario the device would need to know which clock had assigned id 0 in the past to avoid potential misinterpretation by the

driver.

Quoted 12 lines
Quoted 12 lines
Quoted 3 lines

Since this interface does not have the complexity of hotplug and unplug of the

clocks, we should just keep it simple to a reasonable number of clocks.

Quoted 4 lines

The 64-bit value is not hurting at all, I just feel it is built for unknown.

Any reason to not do hot plug/unplug of clocks now?

Current use cases do not require hotplugging, but I tried to make the spec future-proof.

Quoted 6 lines
Quoted 1 line

If there is no use case for hotplugging a clock, we should keep things simple

to a reasonable width.

Quoted 2 lines

I will change the width to 8 bit, although I don't think that makes things simpler.

If there are no hotplug clocks, than 8-bit or 16-bit is just enough right for reasonable number of clocks. I don’t have strong opinion for bit width.
64-bit width without hot plug case seemed very large.
Please the practical range that you find suitable.

Quoted 73 lines
Quoted 64 lines
Quoted 64 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

This would work with 8-bit too.
Not that I have issue with 64-bits, but the use case for large number of clocks

Quoted 8 lines
Quoted 6 lines

under single virtio-rtc device is unclear.

Quoted 2 lines

And if there is no use case, we should have reasonable range.

Using a large id range might help in the future to avoid dealing with the whole class of id ABA problems, cf. the above example.

In my view the clocks are expected to give precision and accuracy, so number

of clocks in a system that occupies 64-bit wide sounds distant.

Quoted 2 lines

If it is used to not_care about finding the free id, it is ok.

Not caring about finding a concurrency-safe free id was the reason for 64 bits width.

Quoted 2 lines

Is hotplug unplug the clock (instead of device) is critical that drives the clock

id?

Quoted 1 line

Clock id width > 8 bit was only considered to be able to easily support hotplugging in the future.

Quoted 6 lines
Quoted 4 lines

To me using 64 bits to avoid the potential future ABA problems looks reasonable, but if others disagree, I can reduce to 8 bits.

It is not must, I just find the 64-bit level clocks a bit too much specially at

hotplug/unplug part.

Quoted 17 lines
Quoted 17 lines
Quoted 9 lines
Quoted 5 lines

With "unclear responsibility" I meant to refer to the responsibility for drafting a spec extension for virtio-net,

You can count me to draft the virtio-net spec for responsibility. I am sure others would join as there is common interest across multiple device

implementors to support it.

Quoted 2 lines

We know at least 4 stake holders have interests in it in virtio-net (Alibaba,

google, marvell, nvidia).

Quoted 1 line

OK, thank you for the information.

Quoted 7 lines
Quoted 4 lines

I think the arguments on how to have the RTC spec interact with the net device are now on the table. Maybe the discussion could continue on a draft spec extension for the net device?

I didn’t follow the comment.
My suggestion is lets draft the spec _now_ that covers rtc functionality as part

of virtio-net device and as independent rtc device.

Quoted 2 lines

And if that requires me to work and draft the net part for you, I am here to

draft it.

Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Would it work for you?

Of course, work on the net part could also start now.

I would prefer if the net part could be discussed as a dedicated patch, on top of the patch(es) introducing the RTC device (and referring to

those).

Quoted 1 line
Quoted 1 line
Quoted 9 lines
Quoted 2 lines

It will be separate patch (or patches) in device-type/net/description.tex. It should be part of same patch series as rtc and net so both devices agree

and reserve necessary opcode ranges etc to utilize same infra.

Quoted 1 line

I plan to send out RTC spec RFC v3, which should contain all the agreed changes from the v2 discussion, in November. Maybe the net related changes could then be put on top of that (and become part of a common RFC v4)?

Sure, we can possibly converge before posting v3 itself so that v4 does not require rewrite due to net. This can be done by drafting v3 in such a way that it fits the v4 scenario.

#16 lore raw
Quoted 50 lines · Parav Pandit
Quoted 50 lines

From: Peter Hilber <[email protected]>
Sent: Tuesday, October 17, 2023 9:36 PM
To: Parav Pandit <[email protected]>; [email protected]
Cc: Cornelia Huck <[email protected]>
Subject: Re: [virtio-comment] [RFC PATCH v2] virtio-rtc: Add device specification.

Quoted 9 lines · Parav Pandit
Quoted 9 lines

From: Peter Hilber <[email protected]>
Sent: Friday, October 13, 2023 9:00 PM

Quoted 5 lines · Parav Pandit
Quoted 4 lines

From: Peter Hilber <[email protected]

Sent: Thursday, October 12, 2023 4:26 PM

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

So am I missing something?
In an arbitrary, interface, once unplug is successful, i.e. when the device exactly know that unplug is completed by the driver, It can re-use the

id.

But this would require the device to remember previously unplugged

clocks.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 1 line

Not always, it can do when the clock wrap arounds.

Sorry, I do not understand the reply. When the clock id is wrapping around, in the discussed scenario the device would need to know which clock had assigned id 0 in the past to avoid potential misinterpretation by the

driver.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

Since this interface does not have the complexity of hotplug and unplug of the

clocks, we should just keep it simple to a reasonable number of clocks.

Quoted 2 lines

The 64-bit value is not hurting at all, I just feel it is built for unknown.

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

Any reason to not do hot plug/unplug of clocks now?

Current use cases do not require hotplugging, but I tried to make the spec future-proof.

If there is no use case for hotplugging a clock, we should keep things simple

to a reasonable width.

Quoted 2 lines
Quoted 6 lines
Quoted 1 line

I will change the width to 8 bit, although I don't think that makes things simpler.

If there are no hotplug clocks, than 8-bit or 16-bit is just enough right for reasonable number of clocks. I don’t have strong opinion for bit width.
64-bit width without hot plug case seemed very large.
Please the practical range that you find suitable.

OK, I will use 16 bits, this should in the context of the current spec satisfy all imaginable use cases.

Quoted 78 lines
Quoted 78 lines
Quoted 64 lines
Quoted 64 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

This would work with 8-bit too.
Not that I have issue with 64-bits, but the use case for large number of clocks

Quoted 8 lines
Quoted 6 lines

under single virtio-rtc device is unclear.

Quoted 2 lines

And if there is no use case, we should have reasonable range.

Using a large id range might help in the future to avoid dealing with the whole class of id ABA problems, cf. the above example.

In my view the clocks are expected to give precision and accuracy, so number

of clocks in a system that occupies 64-bit wide sounds distant.

Quoted 2 lines

If it is used to not_care about finding the free id, it is ok.

Not caring about finding a concurrency-safe free id was the reason for 64 bits width.

Quoted 2 lines

Is hotplug unplug the clock (instead of device) is critical that drives the clock

id?

Quoted 1 line

Clock id width > 8 bit was only considered to be able to easily support hotplugging in the future.

Quoted 6 lines
Quoted 4 lines

To me using 64 bits to avoid the potential future ABA problems looks reasonable, but if others disagree, I can reduce to 8 bits.

It is not must, I just find the 64-bit level clocks a bit too much specially at

hotplug/unplug part.

Quoted 17 lines
Quoted 17 lines
Quoted 9 lines
Quoted 5 lines

With "unclear responsibility" I meant to refer to the responsibility for drafting a spec extension for virtio-net,

You can count me to draft the virtio-net spec for responsibility. I am sure others would join as there is common interest across multiple device

implementors to support it.

Quoted 2 lines

We know at least 4 stake holders have interests in it in virtio-net (Alibaba,

google, marvell, nvidia).

Quoted 1 line

OK, thank you for the information.

Quoted 7 lines
Quoted 4 lines

I think the arguments on how to have the RTC spec interact with the net device are now on the table. Maybe the discussion could continue on a draft spec extension for the net device?

I didn’t follow the comment.
My suggestion is lets draft the spec _now_ that covers rtc functionality as part

of virtio-net device and as independent rtc device.

Quoted 2 lines

And if that requires me to work and draft the net part for you, I am here to

draft it.

Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Would it work for you?

Of course, work on the net part could also start now.

I would prefer if the net part could be discussed as a dedicated patch, on top of the patch(es) introducing the RTC device (and referring to

those).

Quoted 3 lines
Quoted 1 line

It will be separate patch (or patches) in device-type/net/description.tex. It should be part of same patch series as rtc and net so both devices agree

and reserve necessary opcode ranges etc to utilize same infra.

Quoted 1 line
Quoted 6 lines
Quoted 4 lines

I plan to send out RTC spec RFC v3, which should contain all the agreed changes from the v2 discussion, in November. Maybe the net related changes could then be put on top of that (and become part of a common RFC v4)?

Sure, we can possibly converge before posting v3 itself so that v4 does not require rewrite due to net. This can be done by drafting v3 in such a way that it fits the v4 scenario.

AFAIU the only interoperability issue which was brought up in this discussion so far, for which there was no explicit agreement on resolution, is how to encapsulate RTC messages in the net controlq.

For this I propose to encapsulate the RTC requests resp. responses in the net controlq pseudo-fields command-specific-data resp. command-specific-result. (command-specific-result is being introduced through [1].)

This handles natural alignment of the payload in a way which is compatible with

- existing virtio-net controlq commands (including those being introduced by [1]), and

- with the standalone virtio-rtc messages.

Best regards,

Peter

[1] https://lore.kernel.org/virtio-comment/[email protected]/#iZ31device-types:net:description.tex

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#17 lore raw
Quoted 170 lines

From: Peter Hilber <[email protected]>
Sent: Wednesday, October 18, 2023 9:44 PM

Quoted 48 lines · Parav Pandit
Quoted 48 lines

From: Peter Hilber <[email protected]>
Sent: Tuesday, October 17, 2023 9:36 PM
To: Parav Pandit <[email protected]>; [email protected]
Cc: Cornelia Huck <[email protected]>
Subject: Re: [virtio-comment] [RFC PATCH v2] virtio-rtc: Add device specification.

Quoted 9 lines · Parav Pandit
Quoted 9 lines

From: Peter Hilber <[email protected]>
Sent: Friday, October 13, 2023 9:00 PM

Quoted 5 lines · Parav Pandit
Quoted 4 lines

From: Peter Hilber <[email protected]

Sent: Thursday, October 12, 2023 4:26 PM

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

So am I missing something?
In an arbitrary, interface, once unplug is successful, i.e. when the device exactly know that unplug is completed by the driver, It can re-use the

id.

But this would require the device to remember previously unplugged

clocks.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 1 line

Not always, it can do when the clock wrap arounds.

Sorry, I do not understand the reply. When the clock id is wrapping around, in the discussed scenario the device would need to know which clock had assigned id 0 in the past to avoid potential misinterpretation by the

driver.

Quoted 12 lines
Quoted 12 lines
Quoted 3 lines

Since this interface does not have the complexity of hotplug and unplug of the

clocks, we should just keep it simple to a reasonable number of clocks.

Quoted 4 lines

The 64-bit value is not hurting at all, I just feel it is built for unknown.

Any reason to not do hot plug/unplug of clocks now?

Current use cases do not require hotplugging, but I tried to make the spec future-proof.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

If there is no use case for hotplugging a clock, we should keep things simple

to a reasonable width.

Quoted 2 lines

I will change the width to 8 bit, although I don't think that makes things

simpler.

Quoted 1 line

If there are no hotplug clocks, than 8-bit or 16-bit is just enough right for

reasonable number of clocks.

Quoted 4 lines

I don’t have strong opinion for bit width. 64-bit width without hot plug case seemed very large. Please the practical range that you find suitable.

OK, I will use 16 bits, this should in the context of the current spec satisfy all imaginable use cases.

Quoted 77 lines
Quoted 77 lines
Quoted 64 lines
Quoted 64 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

This would work with 8-bit too.
Not that I have issue with 64-bits, but the use case for large number of clocks

Quoted 8 lines
Quoted 6 lines

under single virtio-rtc device is unclear.

Quoted 2 lines

And if there is no use case, we should have reasonable range.

Using a large id range might help in the future to avoid dealing with the whole class of id ABA problems, cf. the above example.

In my view the clocks are expected to give precision and accuracy, so number

of clocks in a system that occupies 64-bit wide sounds distant.

Quoted 2 lines

If it is used to not_care about finding the free id, it is ok.

Not caring about finding a concurrency-safe free id was the reason for 64 bits width.

Quoted 2 lines

Is hotplug unplug the clock (instead of device) is critical that drives the clock

id?

Quoted 1 line

Clock id width > 8 bit was only considered to be able to easily support hotplugging in the future.

Quoted 6 lines
Quoted 4 lines

To me using 64 bits to avoid the potential future ABA problems looks reasonable, but if others disagree, I can reduce to 8 bits.

It is not must, I just find the 64-bit level clocks a bit too much specially at

hotplug/unplug part.

Quoted 17 lines
Quoted 17 lines
Quoted 9 lines
Quoted 5 lines

With "unclear responsibility" I meant to refer to the responsibility for drafting a spec extension for virtio-net,

You can count me to draft the virtio-net spec for responsibility. I am sure others would join as there is common interest across multiple device

implementors to support it.

Quoted 2 lines

We know at least 4 stake holders have interests in it in virtio-net (Alibaba,

google, marvell, nvidia).

Quoted 1 line

OK, thank you for the information.

Quoted 7 lines
Quoted 4 lines

I think the arguments on how to have the RTC spec interact with the net device are now on the table. Maybe the discussion could continue on a draft spec extension for the net device?

I didn’t follow the comment.
My suggestion is lets draft the spec _now_ that covers rtc functionality as part

of virtio-net device and as independent rtc device.

Quoted 2 lines

And if that requires me to work and draft the net part for you, I am here to

draft it.

Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

Would it work for you?

Of course, work on the net part could also start now.

I would prefer if the net part could be discussed as a dedicated patch, on top of the patch(es) introducing the RTC device (and referring to

those).

Quoted 4 lines
Quoted 1 line

It will be separate patch (or patches) in device-type/net/description.tex. It should be part of same patch series as rtc and net so both devices agree

Quoted 6 lines
Quoted 6 lines

and reserve necessary opcode ranges etc to utilize same infra.

Quoted 1 line

I plan to send out RTC spec RFC v3, which should contain all the agreed changes from the v2 discussion, in November. Maybe the net related changes could then be put on top of that (and become part of a

common RFC v4)?

Quoted 2 lines
Quoted 1 line

Sure, we can possibly converge before posting v3 itself so that v4 does not

require rewrite due to net.

Quoted 1 line

This can be done by drafting v3 in such a way that it fits the v4 scenario.

AFAIU the only interoperability issue which was brought up in this discussion so far, for which there was no explicit agreement on resolution, is how to encapsulate RTC messages in the net controlq.

For this I propose to encapsulate the RTC requests resp. responses in the net controlq pseudo-fields command-specific-data resp. command-specific-result. (command-specific-result is being introduced through [1].)

Quoted 6 lines

This handles natural alignment of the payload in a way which is compatible with

- existing virtio-net controlq commands (including those being introduced by [1]), and

- with the standalone virtio-rtc messages.

I will need to re-read the rtc v2, but the generic approach I had in mind is to take the ctrl vq opcode being common between net and rtc. And follow the net's cvq command response format (don’t recall if fullfils rtc or not).
And that’s it. Rtc and net gets both same command, same response.
Net will have feature bit to indicate this functionality exists, for rtc it is implicit as its being and rtc device.

Quoted 2 lines

[1] https://lore.kernel.org/virtio-comment/20231009013135.55890-1-
[email protected]/#iZ31device-types:net:description.tex

#18 lore raw
Quoted 72 lines · Parav Pandit
Quoted 72 lines

From: Peter Hilber <[email protected]>
Sent: Wednesday, October 18, 2023 9:44 PM

Quoted 40 lines · Parav Pandit
Quoted 40 lines
Quoted 27 lines
Quoted 27 lines
Quoted 15 lines
Quoted 15 lines
Quoted 7 lines
Quoted 3 lines

With "unclear responsibility" I meant to refer to the responsibility for drafting a spec extension for virtio-net,

You can count me to draft the virtio-net spec for responsibility. I am sure others would join as there is common interest across multiple device

implementors to support it.

Quoted 2 lines

We know at least 4 stake holders have interests in it in virtio-net (Alibaba,

google, marvell, nvidia).

Quoted 1 line

OK, thank you for the information.

Quoted 7 lines
Quoted 4 lines

I think the arguments on how to have the RTC spec interact with the net device are now on the table. Maybe the discussion could continue on a draft spec extension for the net device?

I didn’t follow the comment.
My suggestion is lets draft the spec _now_ that covers rtc functionality as part

of virtio-net device and as independent rtc device.

Quoted 2 lines

And if that requires me to work and draft the net part for you, I am here to

draft it.

Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

Would it work for you?

Of course, work on the net part could also start now.

I would prefer if the net part could be discussed as a dedicated patch, on top of the patch(es) introducing the RTC device (and referring to

those).

Quoted 4 lines
Quoted 1 line

It will be separate patch (or patches) in device-type/net/description.tex. It should be part of same patch series as rtc and net so both devices agree

Quoted 6 lines
Quoted 6 lines

and reserve necessary opcode ranges etc to utilize same infra.

Quoted 1 line

I plan to send out RTC spec RFC v3, which should contain all the agreed changes from the v2 discussion, in November. Maybe the net related changes could then be put on top of that (and become part of a

common RFC v4)?

Quoted 2 lines
Quoted 1 line

Sure, we can possibly converge before posting v3 itself so that v4 does not

require rewrite due to net.

Quoted 1 line

This can be done by drafting v3 in such a way that it fits the v4 scenario.

AFAIU the only interoperability issue which was brought up in this discussion so far, for which there was no explicit agreement on resolution, is how to encapsulate RTC messages in the net controlq.

For this I propose to encapsulate the RTC requests resp. responses in the net controlq pseudo-fields command-specific-data resp. command-specific-result. (command-specific-result is being introduced through [1].)

This handles natural alignment of the payload in a way which is compatible with

- existing virtio-net controlq commands (including those being introduced by [1]), and

Quoted 7 lines
Quoted 1 line

- with the standalone virtio-rtc messages.

I will need to re-read the rtc v2, but the generic approach I had in mind is to take the ctrl vq opcode being common between net and rtc. And follow the net's cvq command response format (don’t recall if fullfils rtc or not).
And that’s it. Rtc and net gets both same command, same response.
Net will have feature bit to indicate this functionality exists, for rtc it is implicit as its being and rtc device.

But that would force virtio-rtc to use the not padded request/reply headers which the virtio-net controlq has, implying dedicated descriptors for class/command and ack to still have natural alignment in command-specific-data and command-specific-result.

	struct virtio_net_ctrl {
		u8 class;
		u8 command;
		u8 command-specific-data[];
		u8 ack;
		u8 command-specific-result[];
	};

To me, this looks like a net legacy issue, so it should be addressed in net by encapsulating the aligned RTC request/response.

Thanks for your feedback,

Peter

Quoted 2 lines
Quoted 2 lines

[1] https://lore.kernel.org/virtio-comment/20231009013135.55890-1-
[email protected]/#iZ31device-types:net:description.tex

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#19 lore raw
Quoted 28 lines

From: [email protected] <[email protected] open.org> On Behalf Of Peter Hilber
Sent: Wednesday, October 18, 2023 10:30 PM

Quoted 1 line

I will need to re-read the rtc v2, but the generic approach I had in mind is to

take the ctrl vq opcode being common between net and rtc.

Quoted 1 line

And follow the net's cvq command response format (don’t recall if fullfils rtc

or not).

Quoted 2 lines

And that’s it. Rtc and net gets both same command, same response.
Net will have feature bit to indicate this functionality exists, for rtc it is implicit

as its being and rtc device.

Quoted 1 line

But that would force virtio-rtc to use the not padded request/reply headers which the virtio-net controlq has, implying dedicated descriptors for class/command and ack to still have natural alignment in command-specific- data and command-specific-result.

	struct virtio_net_ctrl {
		u8 class;
		u8 command;
		u8 command-specific-data[];
		u8 ack;
		u8 command-specific-result[];
	};
Quoted 3 lines

To me, this looks like a net legacy issue, so it should be addressed in net by encapsulating the aligned RTC request/response.

Right. It makes sense to have rtc request as command-specific-data and rtc response as command-specific-result.