Skip to contentSkip to conversation

virtio-net: support device stats

PATCH Superseded Xuan Zhuo 4 participants 18 messages Jan 11, 2022 – Jan 18, 2022

v8 of 12 Superseded Latest is v12 →

lore
Share & tools
lore

Xuan Zhuo

author
#0 lore raw

PATCH v8 1/1 virtio-net: support device stats

This patch allows the driver to obtain some statistics from the device.

In the back-end implementation, we can count a lot of such information, which can be used for debugging and judging the running status of the back-end. We hope to directly display it to the user through ethtool.

---

v8:
1. Modified based on comments by Cornelia Huck

v7:
1. add rx_reset, tx_reset
2. add device normative and dirver normative
3. add comments for *_packets, *_bytres

v6:
1. correct the names and descriptions of some stats items

v5:
1. add VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
2. more item for virtio_net_ctrl_reply_stats_queue_pair

v4:
1. remove dev_stats_num, {rx|tx}_stats_num
2. Use two commands to get the stats of queue pair and dev respectively

v3 changes:
1. add dev_version
2. use queue_pair_index replace rx_num, tx_num
3. Explain the processing when the device and driver support different numbers of stats

 conformance.tex |   2 +
 content.tex     | 133 +++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 134 insertions(+), 1 deletion(-)
conformance.tex +2 -0
index 42f8537..950924e 100644
@@ -142,6 +142,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
 \end{itemize}
 
 \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / Driver Conformance / Block Driver Conformance}
@@ -401,6 +402,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Gratuitous Packet Sending}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
 \end{itemize}
 
 \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / Device Conformance / Block Device Conformance}
content.tex +132 -1

163 lines not on this page · raw message

--
2.31.0

#1 lore raw
Quoted 35 lines · Xuan Zhuo

This patch allows the driver to obtain some statistics from the device.

In the back-end implementation, we can count a lot of such information, which can be used for debugging and judging the running status of the back-end. We hope to directly display it to the user through ethtool.

---

v8:
1. Modified based on comments by Cornelia Huck

v7:
1. add rx_reset, tx_reset
2. add device normative and dirver normative
3. add comments for *_packets, *_bytres

v6:
1. correct the names and descriptions of some stats items

v5:
1. add VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
2. more item for virtio_net_ctrl_reply_stats_queue_pair

v4:
1. remove dev_stats_num, {rx|tx}_stats_num
2. Use two commands to get the stats of queue pair and dev respectively

v3 changes:
1. add dev_version
2. use queue_pair_index replace rx_num, tx_num
3. Explain the processing when the device and driver support different numbers of stats

Quoted 3 lines
 conformance.tex |   2 +
 content.tex     | 133 +++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 134 insertions(+), 1 deletion(-)

LGTM.

Unless someone else spots a problem, I think this is ready to vote on, so please open a github issue and link it here.

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

author
#2 lore raw
Quoted 37 lines · Cornelia Huck
Quoted 35 lines · Xuan Zhuo

This patch allows the driver to obtain some statistics from the device.

In the back-end implementation, we can count a lot of such information, which can be used for debugging and judging the running status of the back-end. We hope to directly display it to the user through ethtool.

---

v8:
1. Modified based on comments by Cornelia Huck

v7:
1. add rx_reset, tx_reset
2. add device normative and dirver normative
3. add comments for *_packets, *_bytres

v6:
1. correct the names and descriptions of some stats items

v5:
1. add VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
2. more item for virtio_net_ctrl_reply_stats_queue_pair

v4:
1. remove dev_stats_num, {rx|tx}_stats_num
2. Use two commands to get the stats of queue pair and dev respectively

v3 changes:
1. add dev_version
2. use queue_pair_index replace rx_num, tx_num
3. Explain the processing when the device and driver support different numbers of stats

Quoted 8 lines
Quoted 3 lines
 conformance.tex |   2 +
 content.tex     | 133 +++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 134 insertions(+), 1 deletion(-)

LGTM.

Unless someone else spots a problem, I think this is ready to vote on, so please open a github issue and link it here.

Quoted 1 line

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#3 lore raw
In reply to Xuan Zhuo
Quoted 82 lines · Xuan Zhuo

This patch allows the driver to obtain some statistics from the device.

In the back-end implementation, we can count a lot of such information, which can be used for debugging and judging the running status of the back-end. We hope to directly display it to the user through ethtool.

---

v8:
1. Modified based on comments by Cornelia Huck

v7:
1. add rx_reset, tx_reset
2. add device normative and dirver normative
3. add comments for *_packets, *_bytres

v6:
1. correct the names and descriptions of some stats items

v5:
1. add VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
2. more item for virtio_net_ctrl_reply_stats_queue_pair

v4:
1. remove dev_stats_num, {rx|tx}_stats_num
2. Use two commands to get the stats of queue pair and dev respectively

v3 changes:
1. add dev_version
2. use queue_pair_index replace rx_num, tx_num
3. Explain the processing when the device and driver support different numbers of stats

 conformance.tex |   2 +
 content.tex     | 133 +++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 134 insertions(+), 1 deletion(-)
conformance.tex +2 -0
index 42f8537..950924e 100644
@@ -142,6 +142,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
 \end{itemize}
 
 \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / Driver Conformance / Block Driver Conformance}
@@ -401,6 +402,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Gratuitous Packet Sending}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
 \end{itemize}
 
 \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / Device Conformance / Block Device Conformance}
content.tex +4 -0
index cf20570..fa385e2 100644
@@ -3092,6 +3092,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
     channel.
 
+\item[VIRTIO_NET_F_CTRL_STATS(55)] Device can provide device-level statistics
+    to the driver through the control channel.
+
 \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO
  (fragmenting the packet) the USO splits large UDP packet
  to several segments when each of these smaller packets has UDP header.
@@ -3137,6 +3140,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
 \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_CTRL_STATS] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
 \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
 \end{description}

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4025

Quoted 7 lines
content.tex +1 -0
@@ -4015,6 +4019,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
         u8 command;
         u8 command-specific-data[];
         u8 ack;
+        u8 command-specific-data-reply[];
 };
 
 /* ack values */

we now need to document which commands include the reply, otherwise reader will be confused.

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4033

Quoted 8 lines
content.tex +2 -1
@@ -4023,7 +4028,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 \end{lstlisting}
 
 The \field{class}, \field{command} and command-specific-data are set by the
-driver, and the device sets the \field{ack} byte. There is little it can
+driver, and the device sets the \field{ack} byte and optionally
+\field{command-specific-data-reply}. There is little it can
 do

Now "it" refers to device where it used to refer to the driver.

Quoted 23 lines

except issue a diagnostic if \field{ack} is not
VIRTIO_NET_OK.

content.tex +16 -0
@@ -4471,6 +4477,131 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 according to the native endian of the guest rather than
 (necessarily when not using the legacy interface) little-endian.
 
+\paragraph{Device stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+If the VIRTIO_NET_F_CTRL_STATS feature is negotiated, the driver can
+get device stats from the device in \field{command-specific-data-reply}.
+
+To get the stats, the following definitions are used:
+\begin{lstlisting}
+#define VIRTIO_NET_CTRL_STATS                  6
+#define VIRTIO_NET_CTRL_STATS_GET_DEV          0
+#define VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ      1
+#define VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR   2
+\end{lstlisting}
+
+The following layout structures are used:
+
+\field{command-specific-data}

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4505

Quoted 10 lines
content.tex +10 -0
+\begin{lstlisting}
+struct virtio_net_ctrl_stats_queue_pair {
+    le64 queue_pair_index;
+}
+\end{lstlisting}
+
+\field{command-specific-data-reply}
+\begin{lstlisting}
+struct virtio_net_ctrl_reply_stats_dev {
+    le64 dev_reset;         // The number of device resets.

while we already have some places using // comments, they are not used consistently. Better to use /* text */ for now.

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4506

Quoted 1 line
content.tex +1 -0
+}

I am still worrying about this one. I think it's a bit vague. Some types of reset will zero this counter, others will not. Also, this looks more like a generic capability than anything specific to net. And this raises questions about .

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4509

Quoted 3 lines
content.tex +3 -0
+
+struct virtio_net_ctrl_reply_stats_cvq {
+    le64 request_num; // The number of requests.

We call them commands elsewhere. It is however unclear whether the request for the stats itself is included in the counter. Also, do these counters reset on device reset? I would assume so but given dev_reset does not reset it's hard to be sure.

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4511

Quoted 2 lines
content.tex +2 -0
+    le64 ok_num;      // The number of ok acks.
+    le64 err_num;     // The number of err acks.

A bit more precise pls: e.g. "the number of commands where ack was VIRTIO_NET_OK"?

Quoted 7 lines
content.tex +7 -0
+
+    le64 req_rx_promisc;         // The number of requests with command VIRTIO_NET_CTRL_RX_PROMISC.
+    le64 req_rx_allmulti;        // The number of requests with command VIRTIO_NET_CTRL_RX_ALLMULTI.
+    le64 req_rx_alluni;          // The number of requests with command VIRTIO_NET_CTRL_RX_ALLUNI.
+    le64 req_rx_nomulti;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOMULTI.
+    le64 req_rx_nouni;           // The number of requests with command VIRTIO_NET_CTRL_RX_NOUNI.
+    le64 req_rx_nobcast;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOBCAST.

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4528

Quoted 10 lines
content.tex +10 -0
+    le64 req_mac_table_set;      // The number of requests with command VIRTIO_NET_CTRL_MAC_TABLE_SET.
+    le64 req_mac_addr_set;       // The number of requests with command VIRTIO_NET_CTRL_MAC_ADDR_SET.
+    le64 req_vlan_add;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_ADD.
+    le64 req_vlan_del;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_DEL.
+    le64 req_announce_ack;       // The number of requests with command VIRTIO_NET_CTRL_ANNOUNCE_ACK.
+    le64 req_mq_vq_pairs_set;    // The number of requests with command VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.
+    le64 req_mq_rss_config;      // The number of requests with command VIRTIO_NET_CTRL_MQ_RSS_CONFIG.
+    le64 req_mq_hash_config;     // The number of requests with command VIRTIO_NET_CTRL_MQ_HASH_CONFIG.
+    le64 req_guest_offloads_set; // The number of requests with command VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET.
+}

I just had an idea:
how about the query passes in the command we are debugging instead? this way we do not need to extend this each time.

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4533

Quoted 5 lines
content.tex +5 -0
+
+struct virtio_net_ctrl_reply_stats_queue_pair {
+    /* rx stats */
+    le64 rx_packets;          // The number of packets recived by device, including the dropped packets by device.
+    le64 rx_bytes;            // The number of bytes recived by device, including the dropped packets by device.

typos: received

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4540

Quoted 7 lines
content.tex +7 -0
+
+    le64 rx_notification;     // The number of notifications from driver.
+    le64 rx_interrupt;        // The number of interrupts generated by device.
+
+    le64 rx_drop;             // The number of packets dropped by the rx queue. Contains all kinds of packet drop.
+    le64 rx_drop_overruns;    // The number of packets dropped by the rx queue when no more descriptors were available.
+    le64 rx_drop_oversize;    // The number of oversized packets received by the rx queue.

What does oversized mean in this context? does it apply to mergeable buffers?

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4545

Quoted 5 lines
content.tex +5 -0
+
+    le64 rx_csum_valid;       // The number of packets with VIRTIO_NET_HDR_F_DATA_VALID.
+    le64 rx_csum_partial;     // The number of packets with VIRTIO_NET_HDR_F_NEEDS_CSUM.
+    le64 rx_csum_bad;         // The number of packets with abnormal csum.
+    le64 rx_csum_none;        // The number of packets without hardware csum.

I assume they are only incremented with VIRTIO_NET_F_GUEST_CSUM?

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4547

Quoted 2 lines
content.tex +2 -0
+
+    le64 rx_gso_packets;      // The number of gso packets received by rx.

let's spell it out: I think this means any packet put into VQ and with gso_type != VIRTIO_NET_HDR_GSO_NONE

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4548

Quoted 1 line
content.tex +1 -0
+    le64 rx_gso_bytes;        // The number of gso bytes received by rx.

this one is a bit vague: I assume these are bytes written into the RX VQ buffers? the difference is that multiple packets are combined into a
single gso packet.
and does this include the virtio net header?

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4553

Quoted 5 lines
content.tex +5 -0
+    le64 rx_reset;            // The number of queue resets.
+
+    /* tx stats */
+    le64 tx_packets;          // The number of packets sent by device, excluding the dropped packets by device.
+    le64 tx_bytes;            // The number of bytes sent by device, excluding the dropped packets by device.

same question: are we talking about bytes sent by device? or received from guest for transmission?

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4559

Quoted 6 lines
content.tex +6 -0
+
+    le64 tx_notification;     // The number of notifications from driver.
+    le64 tx_interrupt;        // The number of interrupts generated by device.
+
+    le64 tx_drop;             // The number of packets dropped by the tx queue. Contains all kinds of packet drop.
+    le64 tx_drop_desc_err;    // The number of packets dropped when the descriptor is in an error state.

what is a descriptor in error state?

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4565

Quoted 6 lines
content.tex +6 -0
+
+    le64 tx_csum_none;        // The number of packets that doesn't require hardware csum.
+    le64 tx_csum_partial;     // The number of packets that requires hardware csum.
+
+    le64 tx_gso_packets;      // The number of gso packets transmitted.
+    le64 tx_gso_bytes;        // The number of gso bytes transmitted.

point is, gso packets are not transmitted. they are received from guest. is this what this means? gso bytes probably somehow means bytes in gso packets, but again this is a bit vague.

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4566

Quoted 1 line
content.tex +1 -0
+    le64 tx_reset;            // The number of queue resets.

queue reset counter sounds like a useful generic capability. is this with the new features jason added recently? can we add it to all devices?

Quoted 48 lines
content.tex +38 -1
+}
+\end{lstlisting}
+
+All device stats are divided into three categories:
+\begin{itemize}
+    \item the stats of the device.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_DEV
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_dev
+        \end{itemize}
+
+    \item the stats of the controlq.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_cvq
+        \end{itemize}
+
+    \item the stats of the queue pair. This contains the stats of rx and tx.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR
+            \item command-specific-data structure: virtio_net_ctrl_stats_queue_pair
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_queue_pair
+        \end{itemize}
+\end{itemize}
+
+\devicenormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+If \field{queue_pair_index} is out of range for a
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR command, the device MUST set \field{ack} in
+virtio_net_ctrl to VIRTIO_NET_ERR.
+
+If VIRTIO_NET_F_CTRL_STATS has been negotiated, the device MUST support the
+VIRTIO_NET_CTRL_STATS_GET_DEV, VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ and
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR commands.
+
+\drivernormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+\field{command-specific-data} MUST be empty for VIRTIO_NET_CTRL_STATS_GET_DEV
+and VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ.
 
 \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
 Types / Network Device / Legacy Interface: Framing Requirements}
--

2.31.0

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

author
#4 lore raw
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Xuan Zhuo

This patch allows the driver to obtain some statistics from the device.

In the back-end implementation, we can count a lot of such information, which can be used for debugging and judging the running status of the back-end. We hope to directly display it to the user through ethtool.

---

v8:
1. Modified based on comments by Cornelia Huck

v7:
1. add rx_reset, tx_reset
2. add device normative and dirver normative
3. add comments for *_packets, *_bytres

v6:
1. correct the names and descriptions of some stats items

v5:
1. add VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
2. more item for virtio_net_ctrl_reply_stats_queue_pair

v4:
1. remove dev_stats_num, {rx|tx}_stats_num
2. Use two commands to get the stats of queue pair and dev respectively

v3 changes:
1. add dev_version
2. use queue_pair_index replace rx_num, tx_num
3. Explain the processing when the device and driver support different numbers of stats

 conformance.tex |   2 +
 content.tex     | 133 +++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 134 insertions(+), 1 deletion(-)
conformance.tex +2 -0
index 42f8537..950924e 100644
@@ -142,6 +142,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
 \end{itemize}
 
 \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / Driver Conformance / Block Driver Conformance}
@@ -401,6 +402,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Gratuitous Packet Sending}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
 \end{itemize}
 
 \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / Device Conformance / Block Device Conformance}
content.tex +5 -0
index cf20570..fa385e2 100644
@@ -3092,6 +3092,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
     channel.
 
+\item[VIRTIO_NET_F_CTRL_STATS(55)] Device can provide device-level statistics
+    to the driver through the control channel.
+
 \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO
  (fragmenting the packet) the USO splits large UDP packet
  to several segments when each of these smaller packets has UDP header.
@@ -3137,6 +3140,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
 \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_CTRL_STATS] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
 \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
 \end{description}
@@ -4015,6 +4019,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
         u8 command;
         u8 command-specific-data[];
         u8 ack;
+        u8 command-specific-data-reply[];
 };
 
Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4025

Quoted 1 line
content.tex +0 -0
 /* ack values */

we now need to document which commands include the reply, otherwise reader will be confused.

OK.

Quoted 11 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4033

Quoted 9 lines
content.tex +2 -1
@@ -4023,7 +4028,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 \end{lstlisting}
 
 The \field{class}, \field{command} and command-specific-data are set by the
-driver, and the device sets the \field{ack} byte. There is little it can
+driver, and the device sets the \field{ack} byte and optionally
+\field{command-specific-data-reply}. There is little it can
 do

Now "it" refers to device where it used to refer to the driver.

There is little it can except issue a diagnostic if \field{ack} is not VIRTIO_NET_OK.

I don't feel like this has changed or affected anything. Can you help me change it?

Quoted 29 lines
Quoted 29 lines

except issue a diagnostic if \field{ack} is not
VIRTIO_NET_OK.

content.tex +21 -0
@@ -4471,6 +4477,131 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 according to the native endian of the guest rather than
 (necessarily when not using the legacy interface) little-endian.
 
+\paragraph{Device stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+If the VIRTIO_NET_F_CTRL_STATS feature is negotiated, the driver can
+get device stats from the device in \field{command-specific-data-reply}.
+
+To get the stats, the following definitions are used:
+\begin{lstlisting}
+#define VIRTIO_NET_CTRL_STATS                  6
+#define VIRTIO_NET_CTRL_STATS_GET_DEV          0
+#define VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ      1
+#define VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR   2
+\end{lstlisting}
+
+The following layout structures are used:
+
+\field{command-specific-data}
+\begin{lstlisting}
+struct virtio_net_ctrl_stats_queue_pair {
+    le64 queue_pair_index;
+}
+\end{lstlisting}
Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4505

Quoted 5 lines
content.tex +5 -0
+
+\field{command-specific-data-reply}
+\begin{lstlisting}
+struct virtio_net_ctrl_reply_stats_dev {
+    le64 dev_reset;         // The number of device resets.

while we already have some places using // comments, they are not used consistently. Better to use /* text */ for now.

OK.

Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4506

Quoted 2 lines
content.tex +1 -0
+}

I am still worrying about this one. I think it's a bit vague. Some types of reset will zero this counter, others will not. Also, this looks more like a generic capability than anything specific to net. And this raises questions about .

I think, I should define that the device will not set this variable to 0 during reset.

The purpose of adding this is to count the number of times the user deletes/adds the network card. The virtio-net device may be deleted/added by the user multiple times, and this is reflected in the device layer as the number of resets.

Quoted 12 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4509

Quoted 6 lines
content.tex +3 -0
+
+struct virtio_net_ctrl_reply_stats_cvq {
+    le64 request_num; // The number of requests.

We call them commands elsewhere. It is however unclear whether the request for the stats itself is included in the counter. Also, do these counters reset on device reset? I would assume so but given dev_reset does not reset it's hard to be sure.

I will change request -> command.

We should really define that the device should not reset these statistics during reset

Quoted 6 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4511

Quoted 3 lines
content.tex +2 -0
+    le64 ok_num;      // The number of ok acks.
+    le64 err_num;     // The number of err acks.

A bit more precise pls: e.g. "the number of commands where ack was VIRTIO_NET_OK"?

OK

Quoted 16 lines
Quoted 16 lines
content.tex +14 -0
+
+    le64 req_rx_promisc;         // The number of requests with command VIRTIO_NET_CTRL_RX_PROMISC.
+    le64 req_rx_allmulti;        // The number of requests with command VIRTIO_NET_CTRL_RX_ALLMULTI.
+    le64 req_rx_alluni;          // The number of requests with command VIRTIO_NET_CTRL_RX_ALLUNI.
+    le64 req_rx_nomulti;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOMULTI.
+    le64 req_rx_nouni;           // The number of requests with command VIRTIO_NET_CTRL_RX_NOUNI.
+    le64 req_rx_nobcast;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOBCAST.
+    le64 req_mac_table_set;      // The number of requests with command VIRTIO_NET_CTRL_MAC_TABLE_SET.
+    le64 req_mac_addr_set;       // The number of requests with command VIRTIO_NET_CTRL_MAC_ADDR_SET.
+    le64 req_vlan_add;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_ADD.
+    le64 req_vlan_del;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_DEL.
+    le64 req_announce_ack;       // The number of requests with command VIRTIO_NET_CTRL_ANNOUNCE_ACK.
+    le64 req_mq_vq_pairs_set;    // The number of requests with command VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.
+    le64 req_mq_rss_config;      // The number of requests with command VIRTIO_NET_CTRL_MQ_RSS_CONFIG.
Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4528

Quoted 3 lines
content.tex +3 -0
+    le64 req_mq_hash_config;     // The number of requests with command VIRTIO_NET_CTRL_MQ_HASH_CONFIG.
+    le64 req_guest_offloads_set; // The number of requests with command VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET.
+}

I just had an idea:
how about the query passes in the command we are debugging instead? this way we do not need to extend this each time.

Do you mean command-specific-data uses a structure like this:

{
	le64 class;
	le64 command;
}

This is indeed a really great idea.

Quoted 12 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4533

Quoted 8 lines
content.tex +5 -0
+
+struct virtio_net_ctrl_reply_stats_queue_pair {
+    /* rx stats */
+    le64 rx_packets;          // The number of packets recived by device, including the dropped packets by device.
+    le64 rx_bytes;            // The number of bytes recived by device, including the dropped packets by device.

typos: received

Quoted 10 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4540

Quoted 7 lines
content.tex +7 -0
+
+    le64 rx_notification;     // The number of notifications from driver.
+    le64 rx_interrupt;        // The number of interrupts generated by device.
+
+    le64 rx_drop;             // The number of packets dropped by the rx queue. Contains all kinds of packet drop.
+    le64 rx_drop_overruns;    // The number of packets dropped by the rx queue when no more descriptors were available.
+    le64 rx_drop_oversize;    // The number of oversized packets received by the rx queue.

What does oversized mean in this context? does it apply to mergeable buffers?

For mergeable buffers this statistic is meaningless.

Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4545

Quoted 6 lines
content.tex +5 -0
+
+    le64 rx_csum_valid;       // The number of packets with VIRTIO_NET_HDR_F_DATA_VALID.
+    le64 rx_csum_partial;     // The number of packets with VIRTIO_NET_HDR_F_NEEDS_CSUM.
+    le64 rx_csum_bad;         // The number of packets with abnormal csum.
+    le64 rx_csum_none;        // The number of packets without hardware csum.

I assume they are only incremented with VIRTIO_NET_F_GUEST_CSUM?

Yes

Quoted 6 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4547

Quoted 3 lines
content.tex +2 -0
+
+    le64 rx_gso_packets;      // The number of gso packets received by rx.

let's spell it out: I think this means any packet put into VQ and with gso_type != VIRTIO_NET_HDR_GSO_NONE

Yes

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4548

Quoted 2 lines
content.tex +1 -0
+    le64 rx_gso_bytes;        // The number of gso bytes received by rx.

this one is a bit vague: I assume these are bytes written into the RX VQ buffers? the difference is that multiple packets are combined into a
single gso packet.
and does this include the virtio net header?

I think all *_bytes statistics do not include virtio net header, the packet bytes we care about does not include virtio net header.

Multiple packets are combined into one, and some network headers, such as ip/tcp, etc., are also removed. The final count is the bytes that are finalized to the RV VQ buffers.

Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4553

Quoted 6 lines
content.tex +5 -0
+    le64 rx_reset;            // The number of queue resets.
+
+    /* tx stats */
+    le64 tx_packets;          // The number of packets sent by device, excluding the dropped packets by device.
+    le64 tx_bytes;            // The number of bytes sent by device, excluding the dropped packets by device.

same question: are we talking about bytes sent by device? or received from guest for transmission?

The number of bytes sent by device.

Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4559

Quoted 7 lines
content.tex +6 -0
+
+    le64 tx_notification;     // The number of notifications from driver.
+    le64 tx_interrupt;        // The number of interrupts generated by device.
+
+    le64 tx_drop;             // The number of packets dropped by the tx queue. Contains all kinds of packet drop.
+    le64 tx_drop_desc_err;    // The number of packets dropped when the descriptor is in an error state.

what is a descriptor in error state?

such as len == 0.

Quoted 10 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4565

Quoted 7 lines
content.tex +6 -0
+
+    le64 tx_csum_none;        // The number of packets that doesn't require hardware csum.
+    le64 tx_csum_partial;     // The number of packets that requires hardware csum.
+
+    le64 tx_gso_packets;      // The number of gso packets transmitted.
+    le64 tx_gso_bytes;        // The number of gso bytes transmitted.

point is, gso packets are not transmitted. they are received from guest. is this what this means?

Yes.

Quoted 2 lines

gso bytes probably somehow
means bytes in gso packets, but again this is a bit vague.

It should be defined, I think it should be defined as the number of bytes from the guest.

Quoted 6 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4566

Quoted 2 lines
content.tex +1 -0
+    le64 tx_reset;            // The number of queue resets.

queue reset counter sounds like a useful generic capability. is this with the new features jason added recently? can we add it to all devices?

It's the one I submitted earlier.

	https://github.com/oasis-tcs/virtio-spec/issues/124

The purpose of adding this is that the network device will disable/enable the queue. The statistics of the corresponding device is queue_reset.

Thank you very much.

Quoted 50 lines
Quoted 49 lines
content.tex +38 -1
+}
+\end{lstlisting}
+
+All device stats are divided into three categories:
+\begin{itemize}
+    \item the stats of the device.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_DEV
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_dev
+        \end{itemize}
+
+    \item the stats of the controlq.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_cvq
+        \end{itemize}
+
+    \item the stats of the queue pair. This contains the stats of rx and tx.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR
+            \item command-specific-data structure: virtio_net_ctrl_stats_queue_pair
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_queue_pair
+        \end{itemize}
+\end{itemize}
+
+\devicenormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+If \field{queue_pair_index} is out of range for a
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR command, the device MUST set \field{ack} in
+virtio_net_ctrl to VIRTIO_NET_ERR.
+
+If VIRTIO_NET_F_CTRL_STATS has been negotiated, the device MUST support the
+VIRTIO_NET_CTRL_STATS_GET_DEV, VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ and
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR commands.
+
+\drivernormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+\field{command-specific-data} MUST be empty for VIRTIO_NET_CTRL_STATS_GET_DEV
+and VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ.
 
 \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
 Types / Network Device / Legacy Interface: Framing Requirements}
--

2.31.0

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#5 lore raw
Quoted 210 lines · Xuan Zhuo
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Xuan Zhuo

This patch allows the driver to obtain some statistics from the device.

In the back-end implementation, we can count a lot of such information, which can be used for debugging and judging the running status of the back-end. We hope to directly display it to the user through ethtool.

---

v8:
1. Modified based on comments by Cornelia Huck

v7:
1. add rx_reset, tx_reset
2. add device normative and dirver normative
3. add comments for *_packets, *_bytres

v6:
1. correct the names and descriptions of some stats items

v5:
1. add VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
2. more item for virtio_net_ctrl_reply_stats_queue_pair

v4:
1. remove dev_stats_num, {rx|tx}_stats_num
2. Use two commands to get the stats of queue pair and dev respectively

v3 changes:
1. add dev_version
2. use queue_pair_index replace rx_num, tx_num
3. Explain the processing when the device and driver support different numbers of stats

  conformance.tex |   2 +
  content.tex     | 133 +++++++++++++++++++++++++++++++++++++++++++++++-
  2 files changed, 134 insertions(+), 1 deletion(-)
conformance.tex +2 -0
index 42f8537..950924e 100644
@@ -142,6 +142,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
  \end{itemize}
 
  \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / Driver Conformance / Block Driver Conformance}
@@ -401,6 +402,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Gratuitous Packet Sending}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
  \end{itemize}
 
  \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / Device Conformance / Block Device Conformance}
content.tex +5 -0
index cf20570..fa385e2 100644
@@ -3092,6 +3092,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
      channel.
 
+\item[VIRTIO_NET_F_CTRL_STATS(55)] Device can provide device-level statistics
+    to the driver through the control channel.
+
  \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO
   (fragmenting the packet) the USO splits large UDP packet
   to several segments when each of these smaller packets has UDP header.
@@ -3137,6 +3140,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
  \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_CTRL_STATS] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}
@@ -4015,6 +4019,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
          u8 command;
          u8 command-specific-data[];
          u8 ack;
+        u8 command-specific-data-reply[];
  };
 
Quoted 3 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4025

Quoted 1 line
content.tex +0 -0
  /* ack values */

we now need to document which commands include the reply, otherwise reader will be confused.

OK.

Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4033

Quoted 8 lines
content.tex +2 -1
@@ -4023,7 +4028,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  \end{lstlisting}
 
  The \field{class}, \field{command} and command-specific-data are set by the
-driver, and the device sets the \field{ack} byte. There is little it can
+driver, and the device sets the \field{ack} byte and optionally
+\field{command-specific-data-reply}. There is little it can
  do

Now "it" refers to device where it used to refer to the driver.

There is little it can except issue a diagnostic if \field{ack} is not VIRTIO_NET_OK.

I don't feel like this has changed or affected anything. Can you help me change it?

Quoted 27 lines
Quoted 27 lines

except issue a diagnostic if \field{ack} is not
VIRTIO_NET_OK.

content.tex +20 -0
@@ -4471,6 +4477,131 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  according to the native endian of the guest rather than
  (necessarily when not using the legacy interface) little-endian.
 
+\paragraph{Device stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+If the VIRTIO_NET_F_CTRL_STATS feature is negotiated, the driver can
+get device stats from the device in \field{command-specific-data-reply}.
+
+To get the stats, the following definitions are used:
+\begin{lstlisting}
+#define VIRTIO_NET_CTRL_STATS                  6
+#define VIRTIO_NET_CTRL_STATS_GET_DEV          0
+#define VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ      1
+#define VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR   2
+\end{lstlisting}
+
+The following layout structures are used:
+
+\field{command-specific-data}
+\begin{lstlisting}
+struct virtio_net_ctrl_stats_queue_pair {
+    le64 queue_pair_index;
+}
Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4505

Quoted 6 lines
content.tex +6 -0
+\end{lstlisting}
+
+\field{command-specific-data-reply}
+\begin{lstlisting}
+struct virtio_net_ctrl_reply_stats_dev {
+    le64 dev_reset;         // The number of device resets.

while we already have some places using // comments, they are not used consistently. Better to use /* text */ for now.

OK.

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4506

Quoted 1 line
content.tex +1 -0
+}

I am still worrying about this one. I think it's a bit vague. Some types of reset will zero this counter, others will not. Also, this looks more like a generic capability than anything specific to net. And this raises questions about .

I think, I should define that the device will not set this variable to 0 during reset.

The purpose of adding this is to count the number of times the user deletes/adds the network card. The virtio-net device may be deleted/added by the user multiple times, and this is reflected in the device layer as the number of resets.

Quoted 10 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4509

Quoted 5 lines
content.tex +3 -0
+
+struct virtio_net_ctrl_reply_stats_cvq {
+    le64 request_num; // The number of requests.

We call them commands elsewhere. It is however unclear whether the request for the stats itself is included in the counter. Also, do these counters reset on device reset? I would assume so but given dev_reset does not reset it's hard to be sure.

I will change request -> command.

We should really define that the device should not reset these statistics during reset

Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4511

Quoted 2 lines
content.tex +2 -0
+    le64 ok_num;      // The number of ok acks.
+    le64 err_num;     // The number of err acks.

A bit more precise pls: e.g. "the number of commands where ack was VIRTIO_NET_OK"?

OK

Quoted 14 lines
Quoted 14 lines
content.tex +13 -0
+
+    le64 req_rx_promisc;         // The number of requests with command VIRTIO_NET_CTRL_RX_PROMISC.
+    le64 req_rx_allmulti;        // The number of requests with command VIRTIO_NET_CTRL_RX_ALLMULTI.
+    le64 req_rx_alluni;          // The number of requests with command VIRTIO_NET_CTRL_RX_ALLUNI.
+    le64 req_rx_nomulti;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOMULTI.
+    le64 req_rx_nouni;           // The number of requests with command VIRTIO_NET_CTRL_RX_NOUNI.
+    le64 req_rx_nobcast;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOBCAST.
+    le64 req_mac_table_set;      // The number of requests with command VIRTIO_NET_CTRL_MAC_TABLE_SET.
+    le64 req_mac_addr_set;       // The number of requests with command VIRTIO_NET_CTRL_MAC_ADDR_SET.
+    le64 req_vlan_add;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_ADD.
+    le64 req_vlan_del;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_DEL.
+    le64 req_announce_ack;       // The number of requests with command VIRTIO_NET_CTRL_ANNOUNCE_ACK.
+    le64 req_mq_vq_pairs_set;    // The number of requests with command VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.
Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4528

Quoted 4 lines
content.tex +4 -0
+    le64 req_mq_rss_config;      // The number of requests with command VIRTIO_NET_CTRL_MQ_RSS_CONFIG.
+    le64 req_mq_hash_config;     // The number of requests with command VIRTIO_NET_CTRL_MQ_HASH_CONFIG.
+    le64 req_guest_offloads_set; // The number of requests with command VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET.
+}

I just had an idea:
how about the query passes in the command we are debugging instead? this way we do not need to extend this each time.

Quoted 8 lines

Do you mean command-specific-data uses a structure like this:

{
	le64 class;
	le64 command;
}

This is indeed a really great idea.

This probably only work if we mandate the dependency to stats for any new added control vq features.

Quoted 21 lines
Quoted 12 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4533

Quoted 9 lines
content.tex +5 -0
+
+struct virtio_net_ctrl_reply_stats_queue_pair {
+    /* rx stats */
+    le64 rx_packets;          // The number of packets recived by device, including the dropped packets by device.
+    le64 rx_bytes;            // The number of bytes recived by device, including the dropped packets by device.

typos: received

Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4540

Quoted 7 lines
content.tex +7 -0
+
+    le64 rx_notification;     // The number of notifications from driver.
+    le64 rx_interrupt;        // The number of interrupts generated by device.
+
+    le64 rx_drop;             // The number of packets dropped by the rx queue. Contains all kinds of packet drop.
+    le64 rx_drop_overruns;    // The number of packets dropped by the rx queue when no more descriptors were available.
+    le64 rx_drop_oversize;    // The number of oversized packets received by the rx queue.

What does oversized mean in this context? does it apply to mergeable buffers?

Quoted 1 line

For mergeable buffers this statistic is meaningless.

I guess not, we may run out of descriptors even in this case.

Quoted 56 lines
Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4545

Quoted 6 lines
content.tex +5 -0
+
+    le64 rx_csum_valid;       // The number of packets with VIRTIO_NET_HDR_F_DATA_VALID.
+    le64 rx_csum_partial;     // The number of packets with VIRTIO_NET_HDR_F_NEEDS_CSUM.
+    le64 rx_csum_bad;         // The number of packets with abnormal csum.
+    le64 rx_csum_none;        // The number of packets without hardware csum.

I assume they are only incremented with VIRTIO_NET_F_GUEST_CSUM?

Yes

Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4547

Quoted 2 lines
content.tex +2 -0
+
+    le64 rx_gso_packets;      // The number of gso packets received by rx.

let's spell it out: I think this means any packet put into VQ and with gso_type != VIRTIO_NET_HDR_GSO_NONE

Yes

Quoted 5 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4548

Quoted 1 line
content.tex +1 -0
+    le64 rx_gso_bytes;        // The number of gso bytes received by rx.

this one is a bit vague: I assume these are bytes written into the RX VQ buffers? the difference is that multiple packets are combined into a
single gso packet.
and does this include the virtio net header?

I think all *_bytes statistics do not include virtio net header, the packet bytes we care about does not include virtio net header.

Multiple packets are combined into one, and some network headers, such as ip/tcp, etc., are also removed. The final count is the bytes that are finalized to the RV VQ buffers.

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4553

Quoted 5 lines
content.tex +5 -0
+    le64 rx_reset;            // The number of queue resets.
+
+    /* tx stats */
+    le64 tx_packets;          // The number of packets sent by device, excluding the dropped packets by device.
+    le64 tx_bytes;            // The number of bytes sent by device, excluding the dropped packets by device.

same question: are we talking about bytes sent by device? or received from guest for transmission?

The number of bytes sent by device.

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4559

Quoted 6 lines
content.tex +6 -0
+
+    le64 tx_notification;     // The number of notifications from driver.
+    le64 tx_interrupt;        // The number of interrupts generated by device.
+
+    le64 tx_drop;             // The number of packets dropped by the tx queue. Contains all kinds of packet drop.
+    le64 tx_drop_desc_err;    // The number of packets dropped when the descriptor is in an error state.

what is a descriptor in error state?

such as len == 0.

Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4565

Quoted 6 lines
content.tex +6 -0
+
+    le64 tx_csum_none;        // The number of packets that doesn't require hardware csum.
+    le64 tx_csum_partial;     // The number of packets that requires hardware csum.
+
+    le64 tx_gso_packets;      // The number of gso packets transmitted.
+    le64 tx_gso_bytes;        // The number of gso bytes transmitted.

point is, gso packets are not transmitted. they are received from guest. is this what this means?

Quoted 1 line

Yes.

Or it could be case of TSO in the case of TX?

Thanks

Quoted 68 lines
Quoted 3 lines

gso bytes probably somehow
means bytes in gso packets, but again this is a bit vague.

It should be defined, I think it should be defined as the number of bytes from the guest.

Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4566

Quoted 1 line
content.tex +1 -0
+    le64 tx_reset;            // The number of queue resets.

queue reset counter sounds like a useful generic capability. is this with the new features jason added recently? can we add it to all devices?

It's the one I submitted earlier.

	https://github.com/oasis-tcs/virtio-spec/issues/124

The purpose of adding this is that the network device will disable/enable the queue. The statistics of the corresponding device is queue_reset.

Thank you very much.

Quoted 48 lines
Quoted 48 lines
content.tex +38 -1
+}
+\end{lstlisting}
+
+All device stats are divided into three categories:
+\begin{itemize}
+    \item the stats of the device.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_DEV
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_dev
+        \end{itemize}
+
+    \item the stats of the controlq.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_cvq
+        \end{itemize}
+
+    \item the stats of the queue pair. This contains the stats of rx and tx.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR
+            \item command-specific-data structure: virtio_net_ctrl_stats_queue_pair
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_queue_pair
+        \end{itemize}
+\end{itemize}
+
+\devicenormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+If \field{queue_pair_index} is out of range for a
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR command, the device MUST set \field{ack} in
+virtio_net_ctrl to VIRTIO_NET_ERR.
+
+If VIRTIO_NET_F_CTRL_STATS has been negotiated, the device MUST support the
+VIRTIO_NET_CTRL_STATS_GET_DEV, VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ and
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR commands.
+
+\drivernormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+\field{command-specific-data} MUST be empty for VIRTIO_NET_CTRL_STATS_GET_DEV
+and VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ.
 
  \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
  Types / Network Device / Legacy Interface: Framing Requirements}
--

2.31.0

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

author
#6 lore raw
Quoted 219 lines · Jason Wang
Quoted 217 lines · Xuan Zhuo
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Xuan Zhuo

This patch allows the driver to obtain some statistics from the device.

In the back-end implementation, we can count a lot of such information, which can be used for debugging and judging the running status of the back-end. We hope to directly display it to the user through ethtool.

---

v8:
1. Modified based on comments by Cornelia Huck

v7:
1. add rx_reset, tx_reset
2. add device normative and dirver normative
3. add comments for *_packets, *_bytres

v6:
1. correct the names and descriptions of some stats items

v5:
1. add VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
2. more item for virtio_net_ctrl_reply_stats_queue_pair

v4:
1. remove dev_stats_num, {rx|tx}_stats_num
2. Use two commands to get the stats of queue pair and dev respectively

v3 changes:
1. add dev_version
2. use queue_pair_index replace rx_num, tx_num
3. Explain the processing when the device and driver support different numbers of stats

  conformance.tex |   2 +
  content.tex     | 133 +++++++++++++++++++++++++++++++++++++++++++++++-
  2 files changed, 134 insertions(+), 1 deletion(-)
conformance.tex +2 -0
index 42f8537..950924e 100644
@@ -142,6 +142,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
  \end{itemize}
 
  \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / Driver Conformance / Block Driver Conformance}
@@ -401,6 +402,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Gratuitous Packet Sending}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
  \end{itemize}
 
  \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / Device Conformance / Block Device Conformance}
content.tex +5 -0
index cf20570..fa385e2 100644
@@ -3092,6 +3092,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
      channel.
 
+\item[VIRTIO_NET_F_CTRL_STATS(55)] Device can provide device-level statistics
+    to the driver through the control channel.
+
  \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO
   (fragmenting the packet) the USO splits large UDP packet
   to several segments when each of these smaller packets has UDP header.
@@ -3137,6 +3140,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
  \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_CTRL_STATS] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}
@@ -4015,6 +4019,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
          u8 command;
          u8 command-specific-data[];
          u8 ack;
+        u8 command-specific-data-reply[];
  };
 
Quoted 3 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4025

Quoted 1 line
content.tex +0 -0
  /* ack values */

we now need to document which commands include the reply, otherwise reader will be confused.

OK.

Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4033

Quoted 8 lines
content.tex +2 -1
@@ -4023,7 +4028,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  \end{lstlisting}
 
  The \field{class}, \field{command} and command-specific-data are set by the
-driver, and the device sets the \field{ack} byte. There is little it can
+driver, and the device sets the \field{ack} byte and optionally
+\field{command-specific-data-reply}. There is little it can
  do

Now "it" refers to device where it used to refer to the driver.

There is little it can except issue a diagnostic if \field{ack} is not VIRTIO_NET_OK.

I don't feel like this has changed or affected anything. Can you help me change it?

Quoted 27 lines
Quoted 27 lines

except issue a diagnostic if \field{ack} is not
VIRTIO_NET_OK.

content.tex +20 -0
@@ -4471,6 +4477,131 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  according to the native endian of the guest rather than
  (necessarily when not using the legacy interface) little-endian.
 
+\paragraph{Device stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+If the VIRTIO_NET_F_CTRL_STATS feature is negotiated, the driver can
+get device stats from the device in \field{command-specific-data-reply}.
+
+To get the stats, the following definitions are used:
+\begin{lstlisting}
+#define VIRTIO_NET_CTRL_STATS                  6
+#define VIRTIO_NET_CTRL_STATS_GET_DEV          0
+#define VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ      1
+#define VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR   2
+\end{lstlisting}
+
+The following layout structures are used:
+
+\field{command-specific-data}
+\begin{lstlisting}
+struct virtio_net_ctrl_stats_queue_pair {
+    le64 queue_pair_index;
+}
Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4505

Quoted 6 lines
content.tex +6 -0
+\end{lstlisting}
+
+\field{command-specific-data-reply}
+\begin{lstlisting}
+struct virtio_net_ctrl_reply_stats_dev {
+    le64 dev_reset;         // The number of device resets.

while we already have some places using // comments, they are not used consistently. Better to use /* text */ for now.

OK.

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4506

Quoted 1 line
content.tex +1 -0
+}

I am still worrying about this one. I think it's a bit vague. Some types of reset will zero this counter, others will not. Also, this looks more like a generic capability than anything specific to net. And this raises questions about .

I think, I should define that the device will not set this variable to 0 during reset.

The purpose of adding this is to count the number of times the user deletes/adds the network card. The virtio-net device may be deleted/added by the user multiple times, and this is reflected in the device layer as the number of resets.

Quoted 10 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4509

Quoted 5 lines
content.tex +3 -0
+
+struct virtio_net_ctrl_reply_stats_cvq {
+    le64 request_num; // The number of requests.

We call them commands elsewhere. It is however unclear whether the request for the stats itself is included in the counter. Also, do these counters reset on device reset? I would assume so but given dev_reset does not reset it's hard to be sure.

I will change request -> command.

We should really define that the device should not reset these statistics during reset

Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4511

Quoted 2 lines
content.tex +2 -0
+    le64 ok_num;      // The number of ok acks.
+    le64 err_num;     // The number of err acks.

A bit more precise pls: e.g. "the number of commands where ack was VIRTIO_NET_OK"?

OK

Quoted 14 lines
Quoted 14 lines
content.tex +13 -0
+
+    le64 req_rx_promisc;         // The number of requests with command VIRTIO_NET_CTRL_RX_PROMISC.
+    le64 req_rx_allmulti;        // The number of requests with command VIRTIO_NET_CTRL_RX_ALLMULTI.
+    le64 req_rx_alluni;          // The number of requests with command VIRTIO_NET_CTRL_RX_ALLUNI.
+    le64 req_rx_nomulti;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOMULTI.
+    le64 req_rx_nouni;           // The number of requests with command VIRTIO_NET_CTRL_RX_NOUNI.
+    le64 req_rx_nobcast;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOBCAST.
+    le64 req_mac_table_set;      // The number of requests with command VIRTIO_NET_CTRL_MAC_TABLE_SET.
+    le64 req_mac_addr_set;       // The number of requests with command VIRTIO_NET_CTRL_MAC_ADDR_SET.
+    le64 req_vlan_add;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_ADD.
+    le64 req_vlan_del;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_DEL.
+    le64 req_announce_ack;       // The number of requests with command VIRTIO_NET_CTRL_ANNOUNCE_ACK.
+    le64 req_mq_vq_pairs_set;    // The number of requests with command VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.
Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4528

Quoted 4 lines
content.tex +4 -0
+    le64 req_mq_rss_config;      // The number of requests with command VIRTIO_NET_CTRL_MQ_RSS_CONFIG.
+    le64 req_mq_hash_config;     // The number of requests with command VIRTIO_NET_CTRL_MQ_HASH_CONFIG.
+    le64 req_guest_offloads_set; // The number of requests with command VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET.
+}

I just had an idea:
how about the query passes in the command we are debugging instead? this way we do not need to extend this each time.

Do you mean command-specific-data uses a structure like this:

{
	le64 class;
	le64 command;
}
Quoted 6 lines
Quoted 1 line

This is indeed a really great idea.

This probably only work if we mandate the dependency to stats for any new added control vq features.

For new commands, if the device does not support it, then doing so will definitely get an ERR. If the device supports it, we don't need to modify the spec. But this seems to be a problem for migration.

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4533

Quoted 10 lines
content.tex +5 -0
+
+struct virtio_net_ctrl_reply_stats_queue_pair {
+    /* rx stats */
+    le64 rx_packets;          // The number of packets recived by device, including the dropped packets by device.
+    le64 rx_bytes;            // The number of bytes recived by device, including the dropped packets by device.

typos: received

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4540

Quoted 7 lines
content.tex +7 -0
+
+    le64 rx_notification;     // The number of notifications from driver.
+    le64 rx_interrupt;        // The number of interrupts generated by device.
+
+    le64 rx_drop;             // The number of packets dropped by the rx queue. Contains all kinds of packet drop.
+    le64 rx_drop_overruns;    // The number of packets dropped by the rx queue when no more descriptors were available.
+    le64 rx_drop_oversize;    // The number of oversized packets received by the rx queue.

What does oversized mean in this context? does it apply to mergeable buffers?

Quoted 5 lines
Quoted 1 line

For mergeable buffers this statistic is meaningless.

I guess not, we may run out of descriptors even in this case.

I guess, you may not have noticed, that case should be rx_drop_overruns.

Quoted 57 lines
Quoted 57 lines
Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4545

Quoted 7 lines
content.tex +5 -0
+
+    le64 rx_csum_valid;       // The number of packets with VIRTIO_NET_HDR_F_DATA_VALID.
+    le64 rx_csum_partial;     // The number of packets with VIRTIO_NET_HDR_F_NEEDS_CSUM.
+    le64 rx_csum_bad;         // The number of packets with abnormal csum.
+    le64 rx_csum_none;        // The number of packets without hardware csum.

I assume they are only incremented with VIRTIO_NET_F_GUEST_CSUM?

Yes

Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4547

Quoted 2 lines
content.tex +2 -0
+
+    le64 rx_gso_packets;      // The number of gso packets received by rx.

let's spell it out: I think this means any packet put into VQ and with gso_type != VIRTIO_NET_HDR_GSO_NONE

Yes

Quoted 5 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4548

Quoted 1 line
content.tex +1 -0
+    le64 rx_gso_bytes;        // The number of gso bytes received by rx.

this one is a bit vague: I assume these are bytes written into the RX VQ buffers? the difference is that multiple packets are combined into a
single gso packet.
and does this include the virtio net header?

I think all *_bytes statistics do not include virtio net header, the packet bytes we care about does not include virtio net header.

Multiple packets are combined into one, and some network headers, such as ip/tcp, etc., are also removed. The final count is the bytes that are finalized to the RV VQ buffers.

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4553

Quoted 5 lines
content.tex +5 -0
+    le64 rx_reset;            // The number of queue resets.
+
+    /* tx stats */
+    le64 tx_packets;          // The number of packets sent by device, excluding the dropped packets by device.
+    le64 tx_bytes;            // The number of bytes sent by device, excluding the dropped packets by device.

same question: are we talking about bytes sent by device? or received from guest for transmission?

The number of bytes sent by device.

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4559

Quoted 6 lines
content.tex +6 -0
+
+    le64 tx_notification;     // The number of notifications from driver.
+    le64 tx_interrupt;        // The number of interrupts generated by device.
+
+    le64 tx_drop;             // The number of packets dropped by the tx queue. Contains all kinds of packet drop.
+    le64 tx_drop_desc_err;    // The number of packets dropped when the descriptor is in an error state.

what is a descriptor in error state?

such as len == 0.

Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4565

Quoted 6 lines
content.tex +6 -0
+
+    le64 tx_csum_none;        // The number of packets that doesn't require hardware csum.
+    le64 tx_csum_partial;     // The number of packets that requires hardware csum.
+
+    le64 tx_gso_packets;      // The number of gso packets transmitted.
+    le64 tx_gso_bytes;        // The number of gso bytes transmitted.

point is, gso packets are not transmitted. they are received from guest. is this what this means?

Quoted 4 lines
Quoted 1 line

Yes.

Or it could be case of TSO in the case of TX?

Will TSO be anything special?

Thanks.

Quoted 73 lines

Thanks

Quoted 68 lines
Quoted 3 lines

gso bytes probably somehow
means bytes in gso packets, but again this is a bit vague.

It should be defined, I think it should be defined as the number of bytes from the guest.

Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4566

Quoted 1 line
content.tex +1 -0
+    le64 tx_reset;            // The number of queue resets.

queue reset counter sounds like a useful generic capability. is this with the new features jason added recently? can we add it to all devices?

It's the one I submitted earlier.

	https://github.com/oasis-tcs/virtio-spec/issues/124

The purpose of adding this is that the network device will disable/enable the queue. The statistics of the corresponding device is queue_reset.

Thank you very much.

Quoted 48 lines
Quoted 48 lines
content.tex +38 -1
+}
+\end{lstlisting}
+
+All device stats are divided into three categories:
+\begin{itemize}
+    \item the stats of the device.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_DEV
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_dev
+        \end{itemize}
+
+    \item the stats of the controlq.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_cvq
+        \end{itemize}
+
+    \item the stats of the queue pair. This contains the stats of rx and tx.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR
+            \item command-specific-data structure: virtio_net_ctrl_stats_queue_pair
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_queue_pair
+        \end{itemize}
+\end{itemize}
+
+\devicenormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+If \field{queue_pair_index} is out of range for a
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR command, the device MUST set \field{ack} in
+virtio_net_ctrl to VIRTIO_NET_ERR.
+
+If VIRTIO_NET_F_CTRL_STATS has been negotiated, the device MUST support the
+VIRTIO_NET_CTRL_STATS_GET_DEV, VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ and
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR commands.
+
+\drivernormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+\field{command-specific-data} MUST be empty for VIRTIO_NET_CTRL_STATS_GET_DEV
+and VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ.
 
  \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
  Types / Network Device / Legacy Interface: Framing Requirements}
--

2.31.0

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#7 lore raw
Quoted 224 lines · Xuan Zhuo
Quoted 222 lines · Jason Wang
Quoted 218 lines · Xuan Zhuo
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Xuan Zhuo

This patch allows the driver to obtain some statistics from the device.

In the back-end implementation, we can count a lot of such information, which can be used for debugging and judging the running status of the back-end. We hope to directly display it to the user through ethtool.

---

v8:
1. Modified based on comments by Cornelia Huck

v7:
1. add rx_reset, tx_reset
2. add device normative and dirver normative
3. add comments for *_packets, *_bytres

v6:
1. correct the names and descriptions of some stats items

v5:
1. add VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
2. more item for virtio_net_ctrl_reply_stats_queue_pair

v4:
1. remove dev_stats_num, {rx|tx}_stats_num
2. Use two commands to get the stats of queue pair and dev respectively

v3 changes:
1. add dev_version
2. use queue_pair_index replace rx_num, tx_num
3. Explain the processing when the device and driver support different numbers of stats

  conformance.tex |   2 +
  content.tex     | 133 +++++++++++++++++++++++++++++++++++++++++++++++-
  2 files changed, 134 insertions(+), 1 deletion(-)
conformance.tex +2 -0
index 42f8537..950924e 100644
@@ -142,6 +142,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
  \end{itemize}
 
  \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / Driver Conformance / Block Driver Conformance}
@@ -401,6 +402,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Gratuitous Packet Sending}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
  \end{itemize}
 
  \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / Device Conformance / Block Device Conformance}
content.tex +5 -0
index cf20570..fa385e2 100644
@@ -3092,6 +3092,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
      channel.
 
+\item[VIRTIO_NET_F_CTRL_STATS(55)] Device can provide device-level statistics
+    to the driver through the control channel.
+
  \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO
   (fragmenting the packet) the USO splits large UDP packet
   to several segments when each of these smaller packets has UDP header.
@@ -3137,6 +3140,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
  \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_CTRL_STATS] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}
@@ -4015,6 +4019,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
          u8 command;
          u8 command-specific-data[];
          u8 ack;
+        u8 command-specific-data-reply[];
  };
 
Quoted 3 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4025

Quoted 1 line
content.tex +0 -0
  /* ack values */

we now need to document which commands include the reply, otherwise reader will be confused.

OK.

Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4033

Quoted 8 lines
content.tex +2 -1
@@ -4023,7 +4028,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  \end{lstlisting}
 
  The \field{class}, \field{command} and command-specific-data are set by the
-driver, and the device sets the \field{ack} byte. There is little it can
+driver, and the device sets the \field{ack} byte and optionally
+\field{command-specific-data-reply}. There is little it can
  do

Now "it" refers to device where it used to refer to the driver.

There is little it can except issue a diagnostic if \field{ack} is not VIRTIO_NET_OK.

I don't feel like this has changed or affected anything. Can you help me change it?

Quoted 27 lines
Quoted 27 lines

except issue a diagnostic if \field{ack} is not
VIRTIO_NET_OK.

content.tex +20 -0
@@ -4471,6 +4477,131 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  according to the native endian of the guest rather than
  (necessarily when not using the legacy interface) little-endian.
 
+\paragraph{Device stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+If the VIRTIO_NET_F_CTRL_STATS feature is negotiated, the driver can
+get device stats from the device in \field{command-specific-data-reply}.
+
+To get the stats, the following definitions are used:
+\begin{lstlisting}
+#define VIRTIO_NET_CTRL_STATS                  6
+#define VIRTIO_NET_CTRL_STATS_GET_DEV          0
+#define VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ      1
+#define VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR   2
+\end{lstlisting}
+
+The following layout structures are used:
+
+\field{command-specific-data}
+\begin{lstlisting}
+struct virtio_net_ctrl_stats_queue_pair {
+    le64 queue_pair_index;
+}
Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4505

Quoted 6 lines
content.tex +6 -0
+\end{lstlisting}
+
+\field{command-specific-data-reply}
+\begin{lstlisting}
+struct virtio_net_ctrl_reply_stats_dev {
+    le64 dev_reset;         // The number of device resets.

while we already have some places using // comments, they are not used consistently. Better to use /* text */ for now.

OK.

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4506

Quoted 1 line
content.tex +1 -0
+}

I am still worrying about this one. I think it's a bit vague. Some types of reset will zero this counter, others will not. Also, this looks more like a generic capability than anything specific to net. And this raises questions about .

I think, I should define that the device will not set this variable to 0 during reset.

The purpose of adding this is to count the number of times the user deletes/adds the network card. The virtio-net device may be deleted/added by the user multiple times, and this is reflected in the device layer as the number of resets.

Quoted 10 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4509

Quoted 5 lines
content.tex +3 -0
+
+struct virtio_net_ctrl_reply_stats_cvq {
+    le64 request_num; // The number of requests.

We call them commands elsewhere. It is however unclear whether the request for the stats itself is included in the counter. Also, do these counters reset on device reset? I would assume so but given dev_reset does not reset it's hard to be sure.

I will change request -> command.

We should really define that the device should not reset these statistics during reset

Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4511

Quoted 2 lines
content.tex +2 -0
+    le64 ok_num;      // The number of ok acks.
+    le64 err_num;     // The number of err acks.

A bit more precise pls: e.g. "the number of commands where ack was VIRTIO_NET_OK"?

OK

Quoted 14 lines
Quoted 14 lines
content.tex +13 -0
+
+    le64 req_rx_promisc;         // The number of requests with command VIRTIO_NET_CTRL_RX_PROMISC.
+    le64 req_rx_allmulti;        // The number of requests with command VIRTIO_NET_CTRL_RX_ALLMULTI.
+    le64 req_rx_alluni;          // The number of requests with command VIRTIO_NET_CTRL_RX_ALLUNI.
+    le64 req_rx_nomulti;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOMULTI.
+    le64 req_rx_nouni;           // The number of requests with command VIRTIO_NET_CTRL_RX_NOUNI.
+    le64 req_rx_nobcast;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOBCAST.
+    le64 req_mac_table_set;      // The number of requests with command VIRTIO_NET_CTRL_MAC_TABLE_SET.
+    le64 req_mac_addr_set;       // The number of requests with command VIRTIO_NET_CTRL_MAC_ADDR_SET.
+    le64 req_vlan_add;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_ADD.
+    le64 req_vlan_del;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_DEL.
+    le64 req_announce_ack;       // The number of requests with command VIRTIO_NET_CTRL_ANNOUNCE_ACK.
+    le64 req_mq_vq_pairs_set;    // The number of requests with command VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.
Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4528

Quoted 4 lines
content.tex +4 -0
+    le64 req_mq_rss_config;      // The number of requests with command VIRTIO_NET_CTRL_MQ_RSS_CONFIG.
+    le64 req_mq_hash_config;     // The number of requests with command VIRTIO_NET_CTRL_MQ_HASH_CONFIG.
+    le64 req_guest_offloads_set; // The number of requests with command VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET.
+}

I just had an idea:
how about the query passes in the command we are debugging instead? this way we do not need to extend this each time.

Do you mean command-specific-data uses a structure like this:

{
    le64 class;
    le64 command;
}

This is indeed a really great idea.

Quoted 7 lines
Quoted 3 lines

This probably only work if we mandate the dependency to stats for any new added control vq features.

For new commands, if the device does not support it, then doing so will definitely get an ERR. If the device supports it, we don't need to modify the spec. But this seems to be a problem for migration.

Yes, as discussed, we need to keep migration compatibility here.

Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4533

Quoted 12 lines
content.tex +5 -0
+
+struct virtio_net_ctrl_reply_stats_queue_pair {
+    /* rx stats */
+    le64 rx_packets;          // The number of packets recived by device, including the dropped packets by device.
+    le64 rx_bytes;            // The number of bytes recived by device, including the dropped packets by device.

typos: received

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4540

Quoted 7 lines
content.tex +7 -0
+
+    le64 rx_notification;     // The number of notifications from driver.
+    le64 rx_interrupt;        // The number of interrupts generated by device.
+
+    le64 rx_drop;             // The number of packets dropped by the rx queue. Contains all kinds of packet drop.
+    le64 rx_drop_overruns;    // The number of packets dropped by the rx queue when no more descriptors were available.
+    le64 rx_drop_oversize;    // The number of oversized packets received by the rx queue.

What does oversized mean in this context? does it apply to mergeable buffers?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

For mergeable buffers this statistic is meaningless.

I guess not, we may run out of descriptors even in this case.

I guess, you may not have noticed, that case should be rx_drop_overruns.

Probably, but I wonder how much value it would be for maintaining two separate counters. Can we unify them?

Quoted 58 lines
Quoted 58 lines
Quoted 58 lines
Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4545

Quoted 8 lines
content.tex +5 -0
+
+    le64 rx_csum_valid;       // The number of packets with VIRTIO_NET_HDR_F_DATA_VALID.
+    le64 rx_csum_partial;     // The number of packets with VIRTIO_NET_HDR_F_NEEDS_CSUM.
+    le64 rx_csum_bad;         // The number of packets with abnormal csum.
+    le64 rx_csum_none;        // The number of packets without hardware csum.

I assume they are only incremented with VIRTIO_NET_F_GUEST_CSUM?

Yes

Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4547

Quoted 2 lines
content.tex +2 -0
+
+    le64 rx_gso_packets;      // The number of gso packets received by rx.

let's spell it out: I think this means any packet put into VQ and with gso_type != VIRTIO_NET_HDR_GSO_NONE

Yes

Quoted 5 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4548

Quoted 1 line
content.tex +1 -0
+    le64 rx_gso_bytes;        // The number of gso bytes received by rx.

this one is a bit vague: I assume these are bytes written into the RX VQ buffers? the difference is that multiple packets are combined into a
single gso packet.
and does this include the virtio net header?

I think all *_bytes statistics do not include virtio net header, the packet bytes we care about does not include virtio net header.

Multiple packets are combined into one, and some network headers, such as ip/tcp, etc., are also removed. The final count is the bytes that are finalized to the RV VQ buffers.

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4553

Quoted 5 lines
content.tex +5 -0
+    le64 rx_reset;            // The number of queue resets.
+
+    /* tx stats */
+    le64 tx_packets;          // The number of packets sent by device, excluding the dropped packets by device.
+    le64 tx_bytes;            // The number of bytes sent by device, excluding the dropped packets by device.

same question: are we talking about bytes sent by device? or received from guest for transmission?

The number of bytes sent by device.

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4559

Quoted 6 lines
content.tex +6 -0
+
+    le64 tx_notification;     // The number of notifications from driver.
+    le64 tx_interrupt;        // The number of interrupts generated by device.
+
+    le64 tx_drop;             // The number of packets dropped by the tx queue. Contains all kinds of packet drop.
+    le64 tx_drop_desc_err;    // The number of packets dropped when the descriptor is in an error state.

what is a descriptor in error state?

such as len == 0.

Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4565

Quoted 6 lines
content.tex +6 -0
+
+    le64 tx_csum_none;        // The number of packets that doesn't require hardware csum.
+    le64 tx_csum_partial;     // The number of packets that requires hardware csum.
+
+    le64 tx_gso_packets;      // The number of gso packets transmitted.
+    le64 tx_gso_bytes;        // The number of gso bytes transmitted.

point is, gso packets are not transmitted. they are received from guest. is this what this means?

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Yes.

Or it could be case of TSO in the case of TX?

Will TSO be anything special?

For real hardware: it won't transmit GSO packets, but it can do hardware segmentation. I meant we could cover this in this counter. For software devices: it can transmit GSO packets.

Thanks

Quoted 81 lines

Thanks.

Quoted 69 lines

Thanks

Quoted 65 lines
Quoted 3 lines

gso bytes probably somehow
means bytes in gso packets, but again this is a bit vague.

It should be defined, I think it should be defined as the number of bytes from the guest.

Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4566

Quoted 1 line
content.tex +1 -0
+    le64 tx_reset;            // The number of queue resets.

queue reset counter sounds like a useful generic capability. is this with the new features jason added recently? can we add it to all devices?

It's the one I submitted earlier.

    https://github.com/oasis-tcs/virtio-spec/issues/124

The purpose of adding this is that the network device will disable/enable the queue. The statistics of the corresponding device is queue_reset.

Thank you very much.

Quoted 45 lines
Quoted 45 lines
content.tex +38 -1
+}
+\end{lstlisting}
+
+All device stats are divided into three categories:
+\begin{itemize}
+    \item the stats of the device.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_DEV
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_dev
+        \end{itemize}
+
+    \item the stats of the controlq.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_cvq
+        \end{itemize}
+
+    \item the stats of the queue pair. This contains the stats of rx and tx.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR
+            \item command-specific-data structure: virtio_net_ctrl_stats_queue_pair
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_queue_pair
+        \end{itemize}
+\end{itemize}
+
+\devicenormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+If \field{queue_pair_index} is out of range for a
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR command, the device MUST set \field{ack} in
+virtio_net_ctrl to VIRTIO_NET_ERR.
+
+If VIRTIO_NET_F_CTRL_STATS has been negotiated, the device MUST support the
+VIRTIO_NET_CTRL_STATS_GET_DEV, VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ and
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR commands.
+
+\drivernormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+\field{command-specific-data} MUST be empty for VIRTIO_NET_CTRL_STATS_GET_DEV
+and VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ.
 
  \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
  Types / Network Device / Legacy Interface: Framing Requirements}
--

2.31.0

Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

author
#8 lore raw
Quoted 262 lines · Jason Wang
Quoted 224 lines · Xuan Zhuo
Quoted 222 lines · Jason Wang
Quoted 218 lines · Xuan Zhuo
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Xuan Zhuo

This patch allows the driver to obtain some statistics from the device.

In the back-end implementation, we can count a lot of such information, which can be used for debugging and judging the running status of the back-end. We hope to directly display it to the user through ethtool.

---

v8:
1. Modified based on comments by Cornelia Huck

v7:
1. add rx_reset, tx_reset
2. add device normative and dirver normative
3. add comments for *_packets, *_bytres

v6:
1. correct the names and descriptions of some stats items

v5:
1. add VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
2. more item for virtio_net_ctrl_reply_stats_queue_pair

v4:
1. remove dev_stats_num, {rx|tx}_stats_num
2. Use two commands to get the stats of queue pair and dev respectively

v3 changes:
1. add dev_version
2. use queue_pair_index replace rx_num, tx_num
3. Explain the processing when the device and driver support different numbers of stats

  conformance.tex |   2 +
  content.tex     | 133 +++++++++++++++++++++++++++++++++++++++++++++++-
  2 files changed, 134 insertions(+), 1 deletion(-)
conformance.tex +2 -0
index 42f8537..950924e 100644
@@ -142,6 +142,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
  \end{itemize}
 
  \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / Driver Conformance / Block Driver Conformance}
@@ -401,6 +402,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Gratuitous Packet Sending}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
  \end{itemize}
 
  \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / Device Conformance / Block Device Conformance}
content.tex +5 -0
index cf20570..fa385e2 100644
@@ -3092,6 +3092,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
      channel.
 
+\item[VIRTIO_NET_F_CTRL_STATS(55)] Device can provide device-level statistics
+    to the driver through the control channel.
+
  \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO
   (fragmenting the packet) the USO splits large UDP packet
   to several segments when each of these smaller packets has UDP header.
@@ -3137,6 +3140,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
  \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_CTRL_STATS] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}
@@ -4015,6 +4019,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
          u8 command;
          u8 command-specific-data[];
          u8 ack;
+        u8 command-specific-data-reply[];
  };
 
Quoted 3 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4025

Quoted 1 line
content.tex +0 -0
  /* ack values */

we now need to document which commands include the reply, otherwise reader will be confused.

OK.

Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4033

Quoted 8 lines
content.tex +2 -1
@@ -4023,7 +4028,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  \end{lstlisting}
 
  The \field{class}, \field{command} and command-specific-data are set by the
-driver, and the device sets the \field{ack} byte. There is little it can
+driver, and the device sets the \field{ack} byte and optionally
+\field{command-specific-data-reply}. There is little it can
  do

Now "it" refers to device where it used to refer to the driver.

There is little it can except issue a diagnostic if \field{ack} is not VIRTIO_NET_OK.

I don't feel like this has changed or affected anything. Can you help me change it?

Quoted 27 lines
Quoted 27 lines

except issue a diagnostic if \field{ack} is not
VIRTIO_NET_OK.

content.tex +20 -0
@@ -4471,6 +4477,131 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  according to the native endian of the guest rather than
  (necessarily when not using the legacy interface) little-endian.
 
+\paragraph{Device stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+If the VIRTIO_NET_F_CTRL_STATS feature is negotiated, the driver can
+get device stats from the device in \field{command-specific-data-reply}.
+
+To get the stats, the following definitions are used:
+\begin{lstlisting}
+#define VIRTIO_NET_CTRL_STATS                  6
+#define VIRTIO_NET_CTRL_STATS_GET_DEV          0
+#define VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ      1
+#define VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR   2
+\end{lstlisting}
+
+The following layout structures are used:
+
+\field{command-specific-data}
+\begin{lstlisting}
+struct virtio_net_ctrl_stats_queue_pair {
+    le64 queue_pair_index;
+}
Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4505

Quoted 6 lines
content.tex +6 -0
+\end{lstlisting}
+
+\field{command-specific-data-reply}
+\begin{lstlisting}
+struct virtio_net_ctrl_reply_stats_dev {
+    le64 dev_reset;         // The number of device resets.

while we already have some places using // comments, they are not used consistently. Better to use /* text */ for now.

OK.

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4506

Quoted 1 line
content.tex +1 -0
+}

I am still worrying about this one. I think it's a bit vague. Some types of reset will zero this counter, others will not. Also, this looks more like a generic capability than anything specific to net. And this raises questions about .

I think, I should define that the device will not set this variable to 0 during reset.

The purpose of adding this is to count the number of times the user deletes/adds the network card. The virtio-net device may be deleted/added by the user multiple times, and this is reflected in the device layer as the number of resets.

Quoted 10 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4509

Quoted 5 lines
content.tex +3 -0
+
+struct virtio_net_ctrl_reply_stats_cvq {
+    le64 request_num; // The number of requests.

We call them commands elsewhere. It is however unclear whether the request for the stats itself is included in the counter. Also, do these counters reset on device reset? I would assume so but given dev_reset does not reset it's hard to be sure.

I will change request -> command.

We should really define that the device should not reset these statistics during reset

Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4511

Quoted 2 lines
content.tex +2 -0
+    le64 ok_num;      // The number of ok acks.
+    le64 err_num;     // The number of err acks.

A bit more precise pls: e.g. "the number of commands where ack was VIRTIO_NET_OK"?

OK

Quoted 14 lines
Quoted 14 lines
content.tex +13 -0
+
+    le64 req_rx_promisc;         // The number of requests with command VIRTIO_NET_CTRL_RX_PROMISC.
+    le64 req_rx_allmulti;        // The number of requests with command VIRTIO_NET_CTRL_RX_ALLMULTI.
+    le64 req_rx_alluni;          // The number of requests with command VIRTIO_NET_CTRL_RX_ALLUNI.
+    le64 req_rx_nomulti;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOMULTI.
+    le64 req_rx_nouni;           // The number of requests with command VIRTIO_NET_CTRL_RX_NOUNI.
+    le64 req_rx_nobcast;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOBCAST.
+    le64 req_mac_table_set;      // The number of requests with command VIRTIO_NET_CTRL_MAC_TABLE_SET.
+    le64 req_mac_addr_set;       // The number of requests with command VIRTIO_NET_CTRL_MAC_ADDR_SET.
+    le64 req_vlan_add;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_ADD.
+    le64 req_vlan_del;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_DEL.
+    le64 req_announce_ack;       // The number of requests with command VIRTIO_NET_CTRL_ANNOUNCE_ACK.
+    le64 req_mq_vq_pairs_set;    // The number of requests with command VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.
Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4528

Quoted 4 lines
content.tex +4 -0
+    le64 req_mq_rss_config;      // The number of requests with command VIRTIO_NET_CTRL_MQ_RSS_CONFIG.
+    le64 req_mq_hash_config;     // The number of requests with command VIRTIO_NET_CTRL_MQ_HASH_CONFIG.
+    le64 req_guest_offloads_set; // The number of requests with command VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET.
+}

I just had an idea:
how about the query passes in the command we are debugging instead? this way we do not need to extend this each time.

Do you mean command-specific-data uses a structure like this:

{
    le64 class;
    le64 command;
}

This is indeed a really great idea.

Quoted 7 lines
Quoted 3 lines

This probably only work if we mandate the dependency to stats for any new added control vq features.

For new commands, if the device does not support it, then doing so will definitely get an ERR. If the device supports it, we don't need to modify the spec. But this seems to be a problem for migration.

Yes, as discussed, we need to keep migration compatibility here.

Quoted 27 lines
Quoted 27 lines
Quoted 25 lines
Quoted 15 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4533

Quoted 12 lines
content.tex +5 -0
+
+struct virtio_net_ctrl_reply_stats_queue_pair {
+    /* rx stats */
+    le64 rx_packets;          // The number of packets recived by device, including the dropped packets by device.
+    le64 rx_bytes;            // The number of bytes recived by device, including the dropped packets by device.

typos: received

Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4540

Quoted 7 lines
content.tex +7 -0
+
+    le64 rx_notification;     // The number of notifications from driver.
+    le64 rx_interrupt;        // The number of interrupts generated by device.
+
+    le64 rx_drop;             // The number of packets dropped by the rx queue. Contains all kinds of packet drop.
+    le64 rx_drop_overruns;    // The number of packets dropped by the rx queue when no more descriptors were available.
+    le64 rx_drop_oversize;    // The number of oversized packets received by the rx queue.

What does oversized mean in this context? does it apply to mergeable buffers?

For mergeable buffers this statistic is meaningless.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

I guess not, we may run out of descriptors even in this case.

I guess, you may not have noticed, that case should be rx_drop_overruns.

Probably, but I wonder how much value it would be for maintaining two separate counters. Can we unify them?

rx_drop_overruns: means the backend is busy.
rx_drop_oversize: means received an oversized package.

So, I think we should keep these two.

Quoted 62 lines
Quoted 62 lines
Quoted 62 lines
Quoted 60 lines
Quoted 10 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4545

Quoted 9 lines
content.tex +5 -0
+
+    le64 rx_csum_valid;       // The number of packets with VIRTIO_NET_HDR_F_DATA_VALID.
+    le64 rx_csum_partial;     // The number of packets with VIRTIO_NET_HDR_F_NEEDS_CSUM.
+    le64 rx_csum_bad;         // The number of packets with abnormal csum.
+    le64 rx_csum_none;        // The number of packets without hardware csum.

I assume they are only incremented with VIRTIO_NET_F_GUEST_CSUM?

Yes

Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4547

Quoted 2 lines
content.tex +2 -0
+
+    le64 rx_gso_packets;      // The number of gso packets received by rx.

let's spell it out: I think this means any packet put into VQ and with gso_type != VIRTIO_NET_HDR_GSO_NONE

Yes

Quoted 5 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4548

Quoted 1 line
content.tex +1 -0
+    le64 rx_gso_bytes;        // The number of gso bytes received by rx.

this one is a bit vague: I assume these are bytes written into the RX VQ buffers? the difference is that multiple packets are combined into a
single gso packet.
and does this include the virtio net header?

I think all *_bytes statistics do not include virtio net header, the packet bytes we care about does not include virtio net header.

Multiple packets are combined into one, and some network headers, such as ip/tcp, etc., are also removed. The final count is the bytes that are finalized to the RV VQ buffers.

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4553

Quoted 5 lines
content.tex +5 -0
+    le64 rx_reset;            // The number of queue resets.
+
+    /* tx stats */
+    le64 tx_packets;          // The number of packets sent by device, excluding the dropped packets by device.
+    le64 tx_bytes;            // The number of bytes sent by device, excluding the dropped packets by device.

same question: are we talking about bytes sent by device? or received from guest for transmission?

The number of bytes sent by device.

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4559

Quoted 6 lines
content.tex +6 -0
+
+    le64 tx_notification;     // The number of notifications from driver.
+    le64 tx_interrupt;        // The number of interrupts generated by device.
+
+    le64 tx_drop;             // The number of packets dropped by the tx queue. Contains all kinds of packet drop.
+    le64 tx_drop_desc_err;    // The number of packets dropped when the descriptor is in an error state.

what is a descriptor in error state?

such as len == 0.

Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4565

Quoted 6 lines
content.tex +6 -0
+
+    le64 tx_csum_none;        // The number of packets that doesn't require hardware csum.
+    le64 tx_csum_partial;     // The number of packets that requires hardware csum.
+
+    le64 tx_gso_packets;      // The number of gso packets transmitted.
+    le64 tx_gso_bytes;        // The number of gso bytes transmitted.

point is, gso packets are not transmitted. they are received from guest. is this what this means?

Yes.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Or it could be case of TSO in the case of TX?

Will TSO be anything special?

For real hardware: it won't transmit GSO packets, but it can do hardware segmentation. I meant we could cover this in this counter. For software devices: it can transmit GSO packets.

Oh thanks, I forgot about this situation.

tx_gso_* of course includes the case of TSO. I will explain this in the next version.

Thanks.

Quoted 85 lines

Thanks

Quoted 81 lines

Thanks.

Quoted 69 lines

Thanks

Quoted 65 lines
Quoted 3 lines

gso bytes probably somehow
means bytes in gso packets, but again this is a bit vague.

It should be defined, I think it should be defined as the number of bytes from the guest.

Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4566

Quoted 1 line
content.tex +1 -0
+    le64 tx_reset;            // The number of queue resets.

queue reset counter sounds like a useful generic capability. is this with the new features jason added recently? can we add it to all devices?

It's the one I submitted earlier.

    https://github.com/oasis-tcs/virtio-spec/issues/124

The purpose of adding this is that the network device will disable/enable the queue. The statistics of the corresponding device is queue_reset.

Thank you very much.

Quoted 45 lines
Quoted 45 lines
content.tex +38 -1
+}
+\end{lstlisting}
+
+All device stats are divided into three categories:
+\begin{itemize}
+    \item the stats of the device.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_DEV
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_dev
+        \end{itemize}
+
+    \item the stats of the controlq.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_cvq
+        \end{itemize}
+
+    \item the stats of the queue pair. This contains the stats of rx and tx.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR
+            \item command-specific-data structure: virtio_net_ctrl_stats_queue_pair
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_queue_pair
+        \end{itemize}
+\end{itemize}
+
+\devicenormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+If \field{queue_pair_index} is out of range for a
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR command, the device MUST set \field{ack} in
+virtio_net_ctrl to VIRTIO_NET_ERR.
+
+If VIRTIO_NET_F_CTRL_STATS has been negotiated, the device MUST support the
+VIRTIO_NET_CTRL_STATS_GET_DEV, VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ and
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR commands.
+
+\drivernormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+\field{command-specific-data} MUST be empty for VIRTIO_NET_CTRL_STATS_GET_DEV
+and VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ.
 
  \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
  Types / Network Device / Legacy Interface: Framing Requirements}
--

2.31.0

Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#9 lore raw
Quoted 268 lines · Xuan Zhuo
Quoted 267 lines · Jason Wang
Quoted 224 lines · Xuan Zhuo
Quoted 222 lines · Jason Wang
Quoted 218 lines · Xuan Zhuo
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Xuan Zhuo

This patch allows the driver to obtain some statistics from the device.

In the back-end implementation, we can count a lot of such information, which can be used for debugging and judging the running status of the back-end. We hope to directly display it to the user through ethtool.

---

v8:
1. Modified based on comments by Cornelia Huck

v7:
1. add rx_reset, tx_reset
2. add device normative and dirver normative
3. add comments for *_packets, *_bytres

v6:
1. correct the names and descriptions of some stats items

v5:
1. add VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
2. more item for virtio_net_ctrl_reply_stats_queue_pair

v4:
1. remove dev_stats_num, {rx|tx}_stats_num
2. Use two commands to get the stats of queue pair and dev respectively

v3 changes:
1. add dev_version
2. use queue_pair_index replace rx_num, tx_num
3. Explain the processing when the device and driver support different numbers of stats

  conformance.tex |   2 +
  content.tex     | 133 +++++++++++++++++++++++++++++++++++++++++++++++-
  2 files changed, 134 insertions(+), 1 deletion(-)
conformance.tex +2 -0
index 42f8537..950924e 100644
@@ -142,6 +142,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
  \end{itemize}
 
  \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / Driver Conformance / Block Driver Conformance}
@@ -401,6 +402,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Gratuitous Packet Sending}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
  \end{itemize}
 
  \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / Device Conformance / Block Device Conformance}
content.tex +5 -0
index cf20570..fa385e2 100644
@@ -3092,6 +3092,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
      channel.
 
+\item[VIRTIO_NET_F_CTRL_STATS(55)] Device can provide device-level statistics
+    to the driver through the control channel.
+
  \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO
   (fragmenting the packet) the USO splits large UDP packet
   to several segments when each of these smaller packets has UDP header.
@@ -3137,6 +3140,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
  \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_CTRL_STATS] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}
@@ -4015,6 +4019,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
          u8 command;
          u8 command-specific-data[];
          u8 ack;
+        u8 command-specific-data-reply[];
  };
 
Quoted 3 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4025

Quoted 1 line
content.tex +0 -0
  /* ack values */

we now need to document which commands include the reply, otherwise reader will be confused.

OK.

Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4033

Quoted 8 lines
content.tex +2 -1
@@ -4023,7 +4028,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  \end{lstlisting}
 
  The \field{class}, \field{command} and command-specific-data are set by the
-driver, and the device sets the \field{ack} byte. There is little it can
+driver, and the device sets the \field{ack} byte and optionally
+\field{command-specific-data-reply}. There is little it can
  do

Now "it" refers to device where it used to refer to the driver.

There is little it can except issue a diagnostic if \field{ack} is not VIRTIO_NET_OK.

I don't feel like this has changed or affected anything. Can you help me change it?

Quoted 27 lines
Quoted 27 lines

except issue a diagnostic if \field{ack} is not
VIRTIO_NET_OK.

content.tex +20 -0
@@ -4471,6 +4477,131 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  according to the native endian of the guest rather than
  (necessarily when not using the legacy interface) little-endian.
 
+\paragraph{Device stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+If the VIRTIO_NET_F_CTRL_STATS feature is negotiated, the driver can
+get device stats from the device in \field{command-specific-data-reply}.
+
+To get the stats, the following definitions are used:
+\begin{lstlisting}
+#define VIRTIO_NET_CTRL_STATS                  6
+#define VIRTIO_NET_CTRL_STATS_GET_DEV          0
+#define VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ      1
+#define VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR   2
+\end{lstlisting}
+
+The following layout structures are used:
+
+\field{command-specific-data}
+\begin{lstlisting}
+struct virtio_net_ctrl_stats_queue_pair {
+    le64 queue_pair_index;
+}
Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4505

Quoted 6 lines
content.tex +6 -0
+\end{lstlisting}
+
+\field{command-specific-data-reply}
+\begin{lstlisting}
+struct virtio_net_ctrl_reply_stats_dev {
+    le64 dev_reset;         // The number of device resets.

while we already have some places using // comments, they are not used consistently. Better to use /* text */ for now.

OK.

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4506

Quoted 1 line
content.tex +1 -0
+}

I am still worrying about this one. I think it's a bit vague. Some types of reset will zero this counter, others will not. Also, this looks more like a generic capability than anything specific to net. And this raises questions about .

I think, I should define that the device will not set this variable to 0 during reset.

The purpose of adding this is to count the number of times the user deletes/adds the network card. The virtio-net device may be deleted/added by the user multiple times, and this is reflected in the device layer as the number of resets.

Quoted 10 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4509

Quoted 5 lines
content.tex +3 -0
+
+struct virtio_net_ctrl_reply_stats_cvq {
+    le64 request_num; // The number of requests.

We call them commands elsewhere. It is however unclear whether the request for the stats itself is included in the counter. Also, do these counters reset on device reset? I would assume so but given dev_reset does not reset it's hard to be sure.

I will change request -> command.

We should really define that the device should not reset these statistics during reset

Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4511

Quoted 2 lines
content.tex +2 -0
+    le64 ok_num;      // The number of ok acks.
+    le64 err_num;     // The number of err acks.

A bit more precise pls: e.g. "the number of commands where ack was VIRTIO_NET_OK"?

OK

Quoted 14 lines
Quoted 14 lines
content.tex +13 -0
+
+    le64 req_rx_promisc;         // The number of requests with command VIRTIO_NET_CTRL_RX_PROMISC.
+    le64 req_rx_allmulti;        // The number of requests with command VIRTIO_NET_CTRL_RX_ALLMULTI.
+    le64 req_rx_alluni;          // The number of requests with command VIRTIO_NET_CTRL_RX_ALLUNI.
+    le64 req_rx_nomulti;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOMULTI.
+    le64 req_rx_nouni;           // The number of requests with command VIRTIO_NET_CTRL_RX_NOUNI.
+    le64 req_rx_nobcast;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOBCAST.
+    le64 req_mac_table_set;      // The number of requests with command VIRTIO_NET_CTRL_MAC_TABLE_SET.
+    le64 req_mac_addr_set;       // The number of requests with command VIRTIO_NET_CTRL_MAC_ADDR_SET.
+    le64 req_vlan_add;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_ADD.
+    le64 req_vlan_del;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_DEL.
+    le64 req_announce_ack;       // The number of requests with command VIRTIO_NET_CTRL_ANNOUNCE_ACK.
+    le64 req_mq_vq_pairs_set;    // The number of requests with command VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.
Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4528

Quoted 4 lines
content.tex +4 -0
+    le64 req_mq_rss_config;      // The number of requests with command VIRTIO_NET_CTRL_MQ_RSS_CONFIG.
+    le64 req_mq_hash_config;     // The number of requests with command VIRTIO_NET_CTRL_MQ_HASH_CONFIG.
+    le64 req_guest_offloads_set; // The number of requests with command VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET.
+}

I just had an idea:
how about the query passes in the command we are debugging instead? this way we do not need to extend this each time.

Do you mean command-specific-data uses a structure like this:

{
    le64 class;
    le64 command;
}

This is indeed a really great idea.

Quoted 7 lines
Quoted 3 lines

This probably only work if we mandate the dependency to stats for any new added control vq features.

For new commands, if the device does not support it, then doing so will definitely get an ERR. If the device supports it, we don't need to modify the spec. But this seems to be a problem for migration.

Yes, as discussed, we need to keep migration compatibility here.

Quoted 31 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4533

Quoted 12 lines
content.tex +5 -0
+
+struct virtio_net_ctrl_reply_stats_queue_pair {
+    /* rx stats */
+    le64 rx_packets;          // The number of packets recived by device, including the dropped packets by device.
+    le64 rx_bytes;            // The number of bytes recived by device, including the dropped packets by device.

typos: received

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4540

Quoted 7 lines
content.tex +7 -0
+
+    le64 rx_notification;     // The number of notifications from driver.
+    le64 rx_interrupt;        // The number of interrupts generated by device.
+
+    le64 rx_drop;             // The number of packets dropped by the rx queue. Contains all kinds of packet drop.
+    le64 rx_drop_overruns;    // The number of packets dropped by the rx queue when no more descriptors were available.
+    le64 rx_drop_oversize;    // The number of oversized packets received by the rx queue.

What does oversized mean in this context? does it apply to mergeable buffers?

Quoted 5 lines
Quoted 1 line

For mergeable buffers this statistic is meaningless.

I guess not, we may run out of descriptors even in this case.

I guess, you may not have noticed, that case should be rx_drop_overruns.

Quoted 7 lines
Quoted 2 lines

Probably, but I wonder how much value it would be for maintaining two separate counters. Can we unify them?

rx_drop_overruns: means the backend is busy.
rx_drop_oversize: means received an oversized package.

So, I think we should keep these two.

Given it's only without mergeable, will this be ever set in practice? I don't like wasting a counter on something never set.

Quoted 164 lines
Quoted 62 lines
Quoted 62 lines
Quoted 62 lines
Quoted 60 lines
Quoted 10 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4545

Quoted 9 lines
content.tex +5 -0
+
+    le64 rx_csum_valid;       // The number of packets with VIRTIO_NET_HDR_F_DATA_VALID.
+    le64 rx_csum_partial;     // The number of packets with VIRTIO_NET_HDR_F_NEEDS_CSUM.
+    le64 rx_csum_bad;         // The number of packets with abnormal csum.
+    le64 rx_csum_none;        // The number of packets without hardware csum.

I assume they are only incremented with VIRTIO_NET_F_GUEST_CSUM?

Yes

Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4547

Quoted 2 lines
content.tex +2 -0
+
+    le64 rx_gso_packets;      // The number of gso packets received by rx.

let's spell it out: I think this means any packet put into VQ and with gso_type != VIRTIO_NET_HDR_GSO_NONE

Yes

Quoted 5 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4548

Quoted 1 line
content.tex +1 -0
+    le64 rx_gso_bytes;        // The number of gso bytes received by rx.

this one is a bit vague: I assume these are bytes written into the RX VQ buffers? the difference is that multiple packets are combined into a
single gso packet.
and does this include the virtio net header?

I think all *_bytes statistics do not include virtio net header, the packet bytes we care about does not include virtio net header.

Multiple packets are combined into one, and some network headers, such as ip/tcp, etc., are also removed. The final count is the bytes that are finalized to the RV VQ buffers.

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4553

Quoted 5 lines
content.tex +5 -0
+    le64 rx_reset;            // The number of queue resets.
+
+    /* tx stats */
+    le64 tx_packets;          // The number of packets sent by device, excluding the dropped packets by device.
+    le64 tx_bytes;            // The number of bytes sent by device, excluding the dropped packets by device.

same question: are we talking about bytes sent by device? or received from guest for transmission?

The number of bytes sent by device.

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4559

Quoted 6 lines
content.tex +6 -0
+
+    le64 tx_notification;     // The number of notifications from driver.
+    le64 tx_interrupt;        // The number of interrupts generated by device.
+
+    le64 tx_drop;             // The number of packets dropped by the tx queue. Contains all kinds of packet drop.
+    le64 tx_drop_desc_err;    // The number of packets dropped when the descriptor is in an error state.

what is a descriptor in error state?

such as len == 0.

Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4565

Quoted 6 lines
content.tex +6 -0
+
+    le64 tx_csum_none;        // The number of packets that doesn't require hardware csum.
+    le64 tx_csum_partial;     // The number of packets that requires hardware csum.
+
+    le64 tx_gso_packets;      // The number of gso packets transmitted.
+    le64 tx_gso_bytes;        // The number of gso bytes transmitted.

point is, gso packets are not transmitted. they are received from guest. is this what this means?

Yes.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Or it could be case of TSO in the case of TX?

Will TSO be anything special?

For real hardware: it won't transmit GSO packets, but it can do hardware segmentation. I meant we could cover this in this counter. For software devices: it can transmit GSO packets.

Oh thanks, I forgot about this situation.

tx_gso_* of course includes the case of TSO. I will explain this in the next version.

Thanks.

Quoted 85 lines

Thanks

Quoted 81 lines

Thanks.

Quoted 69 lines

Thanks

Quoted 65 lines
Quoted 3 lines

gso bytes probably somehow
means bytes in gso packets, but again this is a bit vague.

It should be defined, I think it should be defined as the number of bytes from the guest.

Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4566

Quoted 1 line
content.tex +1 -0
+    le64 tx_reset;            // The number of queue resets.

queue reset counter sounds like a useful generic capability. is this with the new features jason added recently? can we add it to all devices?

It's the one I submitted earlier.

    https://github.com/oasis-tcs/virtio-spec/issues/124

The purpose of adding this is that the network device will disable/enable the queue. The statistics of the corresponding device is queue_reset.

Thank you very much.

Quoted 45 lines
Quoted 45 lines
content.tex +38 -1
+}
+\end{lstlisting}
+
+All device stats are divided into three categories:
+\begin{itemize}
+    \item the stats of the device.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_DEV
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_dev
+        \end{itemize}
+
+    \item the stats of the controlq.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_cvq
+        \end{itemize}
+
+    \item the stats of the queue pair. This contains the stats of rx and tx.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR
+            \item command-specific-data structure: virtio_net_ctrl_stats_queue_pair
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_queue_pair
+        \end{itemize}
+\end{itemize}
+
+\devicenormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+If \field{queue_pair_index} is out of range for a
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR command, the device MUST set \field{ack} in
+virtio_net_ctrl to VIRTIO_NET_ERR.
+
+If VIRTIO_NET_F_CTRL_STATS has been negotiated, the device MUST support the
+VIRTIO_NET_CTRL_STATS_GET_DEV, VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ and
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR commands.
+
+\drivernormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+\field{command-specific-data} MUST be empty for VIRTIO_NET_CTRL_STATS_GET_DEV
+and VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ.
 
  \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
  Types / Network Device / Legacy Interface: Framing Requirements}
--

2.31.0

Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

author
#10 lore raw
Quoted 272 lines · Michael S. Tsirkin
Quoted 271 lines · Xuan Zhuo
Quoted 269 lines · Jason Wang
Quoted 224 lines · Xuan Zhuo
Quoted 222 lines · Jason Wang
Quoted 218 lines · Xuan Zhuo
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Xuan Zhuo

This patch allows the driver to obtain some statistics from the device.

In the back-end implementation, we can count a lot of such information, which can be used for debugging and judging the running status of the back-end. We hope to directly display it to the user through ethtool.

---

v8:
1. Modified based on comments by Cornelia Huck

v7:
1. add rx_reset, tx_reset
2. add device normative and dirver normative
3. add comments for *_packets, *_bytres

v6:
1. correct the names and descriptions of some stats items

v5:
1. add VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
2. more item for virtio_net_ctrl_reply_stats_queue_pair

v4:
1. remove dev_stats_num, {rx|tx}_stats_num
2. Use two commands to get the stats of queue pair and dev respectively

v3 changes:
1. add dev_version
2. use queue_pair_index replace rx_num, tx_num
3. Explain the processing when the device and driver support different numbers of stats

  conformance.tex |   2 +
  content.tex     | 133 +++++++++++++++++++++++++++++++++++++++++++++++-
  2 files changed, 134 insertions(+), 1 deletion(-)
conformance.tex +2 -0
index 42f8537..950924e 100644
@@ -142,6 +142,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
  \end{itemize}
 
  \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / Driver Conformance / Block Driver Conformance}
@@ -401,6 +402,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Gratuitous Packet Sending}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
  \end{itemize}
 
  \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / Device Conformance / Block Device Conformance}
content.tex +5 -0
index cf20570..fa385e2 100644
@@ -3092,6 +3092,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
      channel.
 
+\item[VIRTIO_NET_F_CTRL_STATS(55)] Device can provide device-level statistics
+    to the driver through the control channel.
+
  \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO
   (fragmenting the packet) the USO splits large UDP packet
   to several segments when each of these smaller packets has UDP header.
@@ -3137,6 +3140,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
  \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_CTRL_STATS] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}
@@ -4015,6 +4019,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
          u8 command;
          u8 command-specific-data[];
          u8 ack;
+        u8 command-specific-data-reply[];
  };
 
Quoted 3 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4025

Quoted 1 line
content.tex +0 -0
  /* ack values */

we now need to document which commands include the reply, otherwise reader will be confused.

OK.

Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4033

Quoted 8 lines
content.tex +2 -1
@@ -4023,7 +4028,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  \end{lstlisting}
 
  The \field{class}, \field{command} and command-specific-data are set by the
-driver, and the device sets the \field{ack} byte. There is little it can
+driver, and the device sets the \field{ack} byte and optionally
+\field{command-specific-data-reply}. There is little it can
  do

Now "it" refers to device where it used to refer to the driver.

There is little it can except issue a diagnostic if \field{ack} is not VIRTIO_NET_OK.

I don't feel like this has changed or affected anything. Can you help me change it?

Quoted 27 lines
Quoted 27 lines

except issue a diagnostic if \field{ack} is not
VIRTIO_NET_OK.

content.tex +20 -0
@@ -4471,6 +4477,131 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  according to the native endian of the guest rather than
  (necessarily when not using the legacy interface) little-endian.
 
+\paragraph{Device stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+If the VIRTIO_NET_F_CTRL_STATS feature is negotiated, the driver can
+get device stats from the device in \field{command-specific-data-reply}.
+
+To get the stats, the following definitions are used:
+\begin{lstlisting}
+#define VIRTIO_NET_CTRL_STATS                  6
+#define VIRTIO_NET_CTRL_STATS_GET_DEV          0
+#define VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ      1
+#define VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR   2
+\end{lstlisting}
+
+The following layout structures are used:
+
+\field{command-specific-data}
+\begin{lstlisting}
+struct virtio_net_ctrl_stats_queue_pair {
+    le64 queue_pair_index;
+}
Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4505

Quoted 6 lines
content.tex +6 -0
+\end{lstlisting}
+
+\field{command-specific-data-reply}
+\begin{lstlisting}
+struct virtio_net_ctrl_reply_stats_dev {
+    le64 dev_reset;         // The number of device resets.

while we already have some places using // comments, they are not used consistently. Better to use /* text */ for now.

OK.

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4506

Quoted 1 line
content.tex +1 -0
+}

I am still worrying about this one. I think it's a bit vague. Some types of reset will zero this counter, others will not. Also, this looks more like a generic capability than anything specific to net. And this raises questions about .

I think, I should define that the device will not set this variable to 0 during reset.

The purpose of adding this is to count the number of times the user deletes/adds the network card. The virtio-net device may be deleted/added by the user multiple times, and this is reflected in the device layer as the number of resets.

Quoted 10 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4509

Quoted 5 lines
content.tex +3 -0
+
+struct virtio_net_ctrl_reply_stats_cvq {
+    le64 request_num; // The number of requests.

We call them commands elsewhere. It is however unclear whether the request for the stats itself is included in the counter. Also, do these counters reset on device reset? I would assume so but given dev_reset does not reset it's hard to be sure.

I will change request -> command.

We should really define that the device should not reset these statistics during reset

Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4511

Quoted 2 lines
content.tex +2 -0
+    le64 ok_num;      // The number of ok acks.
+    le64 err_num;     // The number of err acks.

A bit more precise pls: e.g. "the number of commands where ack was VIRTIO_NET_OK"?

OK

Quoted 14 lines
Quoted 14 lines
content.tex +13 -0
+
+    le64 req_rx_promisc;         // The number of requests with command VIRTIO_NET_CTRL_RX_PROMISC.
+    le64 req_rx_allmulti;        // The number of requests with command VIRTIO_NET_CTRL_RX_ALLMULTI.
+    le64 req_rx_alluni;          // The number of requests with command VIRTIO_NET_CTRL_RX_ALLUNI.
+    le64 req_rx_nomulti;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOMULTI.
+    le64 req_rx_nouni;           // The number of requests with command VIRTIO_NET_CTRL_RX_NOUNI.
+    le64 req_rx_nobcast;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOBCAST.
+    le64 req_mac_table_set;      // The number of requests with command VIRTIO_NET_CTRL_MAC_TABLE_SET.
+    le64 req_mac_addr_set;       // The number of requests with command VIRTIO_NET_CTRL_MAC_ADDR_SET.
+    le64 req_vlan_add;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_ADD.
+    le64 req_vlan_del;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_DEL.
+    le64 req_announce_ack;       // The number of requests with command VIRTIO_NET_CTRL_ANNOUNCE_ACK.
+    le64 req_mq_vq_pairs_set;    // The number of requests with command VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.
Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4528

Quoted 4 lines
content.tex +4 -0
+    le64 req_mq_rss_config;      // The number of requests with command VIRTIO_NET_CTRL_MQ_RSS_CONFIG.
+    le64 req_mq_hash_config;     // The number of requests with command VIRTIO_NET_CTRL_MQ_HASH_CONFIG.
+    le64 req_guest_offloads_set; // The number of requests with command VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET.
+}

I just had an idea:
how about the query passes in the command we are debugging instead? this way we do not need to extend this each time.

Do you mean command-specific-data uses a structure like this:

{
    le64 class;
    le64 command;
}

This is indeed a really great idea.

Quoted 7 lines
Quoted 3 lines

This probably only work if we mandate the dependency to stats for any new added control vq features.

For new commands, if the device does not support it, then doing so will definitely get an ERR. If the device supports it, we don't need to modify the spec. But this seems to be a problem for migration.

Yes, as discussed, we need to keep migration compatibility here.

Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4533

Quoted 12 lines
content.tex +5 -0
+
+struct virtio_net_ctrl_reply_stats_queue_pair {
+    /* rx stats */
+    le64 rx_packets;          // The number of packets recived by device, including the dropped packets by device.
+    le64 rx_bytes;            // The number of bytes recived by device, including the dropped packets by device.

typos: received

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4540

Quoted 7 lines
content.tex +7 -0
+
+    le64 rx_notification;     // The number of notifications from driver.
+    le64 rx_interrupt;        // The number of interrupts generated by device.
+
+    le64 rx_drop;             // The number of packets dropped by the rx queue. Contains all kinds of packet drop.
+    le64 rx_drop_overruns;    // The number of packets dropped by the rx queue when no more descriptors were available.
+    le64 rx_drop_oversize;    // The number of oversized packets received by the rx queue.

What does oversized mean in this context? does it apply to mergeable buffers?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

For mergeable buffers this statistic is meaningless.

I guess not, we may run out of descriptors even in this case.

I guess, you may not have noticed, that case should be rx_drop_overruns.

Probably, but I wonder how much value it would be for maintaining two separate counters. Can we unify them?

Quoted 7 lines
Quoted 4 lines

rx_drop_overruns: means the backend is busy.
rx_drop_oversize: means received an oversized package.

So, I think we should keep these two.

Given it's only without mergeable, will this be ever set in practice? I don't like wasting a counter on something never set.

OK, I see.

Thanks.

Quoted 167 lines
Quoted 166 lines
Quoted 64 lines
Quoted 64 lines
Quoted 64 lines
Quoted 62 lines
Quoted 6 lines
Quoted 6 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4545

Quoted 5 lines
content.tex +5 -0
+
+    le64 rx_csum_valid;       // The number of packets with VIRTIO_NET_HDR_F_DATA_VALID.
+    le64 rx_csum_partial;     // The number of packets with VIRTIO_NET_HDR_F_NEEDS_CSUM.
+    le64 rx_csum_bad;         // The number of packets with abnormal csum.
+    le64 rx_csum_none;        // The number of packets without hardware csum.

I assume they are only incremented with VIRTIO_NET_F_GUEST_CSUM?

Yes

Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4547

Quoted 2 lines
content.tex +2 -0
+
+    le64 rx_gso_packets;      // The number of gso packets received by rx.

let's spell it out: I think this means any packet put into VQ and with gso_type != VIRTIO_NET_HDR_GSO_NONE

Yes

Quoted 5 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4548

Quoted 1 line
content.tex +1 -0
+    le64 rx_gso_bytes;        // The number of gso bytes received by rx.

this one is a bit vague: I assume these are bytes written into the RX VQ buffers? the difference is that multiple packets are combined into a
single gso packet.
and does this include the virtio net header?

I think all *_bytes statistics do not include virtio net header, the packet bytes we care about does not include virtio net header.

Multiple packets are combined into one, and some network headers, such as ip/tcp, etc., are also removed. The final count is the bytes that are finalized to the RV VQ buffers.

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4553

Quoted 5 lines
content.tex +5 -0
+    le64 rx_reset;            // The number of queue resets.
+
+    /* tx stats */
+    le64 tx_packets;          // The number of packets sent by device, excluding the dropped packets by device.
+    le64 tx_bytes;            // The number of bytes sent by device, excluding the dropped packets by device.

same question: are we talking about bytes sent by device? or received from guest for transmission?

The number of bytes sent by device.

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4559

Quoted 6 lines
content.tex +6 -0
+
+    le64 tx_notification;     // The number of notifications from driver.
+    le64 tx_interrupt;        // The number of interrupts generated by device.
+
+    le64 tx_drop;             // The number of packets dropped by the tx queue. Contains all kinds of packet drop.
+    le64 tx_drop_desc_err;    // The number of packets dropped when the descriptor is in an error state.

what is a descriptor in error state?

such as len == 0.

Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4565

Quoted 6 lines
content.tex +6 -0
+
+    le64 tx_csum_none;        // The number of packets that doesn't require hardware csum.
+    le64 tx_csum_partial;     // The number of packets that requires hardware csum.
+
+    le64 tx_gso_packets;      // The number of gso packets transmitted.
+    le64 tx_gso_bytes;        // The number of gso bytes transmitted.

point is, gso packets are not transmitted. they are received from guest. is this what this means?

Yes.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Or it could be case of TSO in the case of TX?

Will TSO be anything special?

For real hardware: it won't transmit GSO packets, but it can do hardware segmentation. I meant we could cover this in this counter. For software devices: it can transmit GSO packets.

Oh thanks, I forgot about this situation.

tx_gso_* of course includes the case of TSO. I will explain this in the next version.

Thanks.

Quoted 85 lines

Thanks

Quoted 81 lines

Thanks.

Quoted 69 lines

Thanks

Quoted 65 lines
Quoted 3 lines

gso bytes probably somehow
means bytes in gso packets, but again this is a bit vague.

It should be defined, I think it should be defined as the number of bytes from the guest.

Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4566

Quoted 1 line
content.tex +1 -0
+    le64 tx_reset;            // The number of queue resets.

queue reset counter sounds like a useful generic capability. is this with the new features jason added recently? can we add it to all devices?

It's the one I submitted earlier.

    https://github.com/oasis-tcs/virtio-spec/issues/124

The purpose of adding this is that the network device will disable/enable the queue. The statistics of the corresponding device is queue_reset.

Thank you very much.

Quoted 45 lines
Quoted 45 lines
content.tex +38 -1
+}
+\end{lstlisting}
+
+All device stats are divided into three categories:
+\begin{itemize}
+    \item the stats of the device.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_DEV
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_dev
+        \end{itemize}
+
+    \item the stats of the controlq.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_cvq
+        \end{itemize}
+
+    \item the stats of the queue pair. This contains the stats of rx and tx.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR
+            \item command-specific-data structure: virtio_net_ctrl_stats_queue_pair
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_queue_pair
+        \end{itemize}
+\end{itemize}
+
+\devicenormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+If \field{queue_pair_index} is out of range for a
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR command, the device MUST set \field{ack} in
+virtio_net_ctrl to VIRTIO_NET_ERR.
+
+If VIRTIO_NET_F_CTRL_STATS has been negotiated, the device MUST support the
+VIRTIO_NET_CTRL_STATS_GET_DEV, VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ and
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR commands.
+
+\drivernormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+\field{command-specific-data} MUST be empty for VIRTIO_NET_CTRL_STATS_GET_DEV
+and VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ.
 
  \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
  Types / Network Device / Legacy Interface: Framing Requirements}
--

2.31.0

Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#11 lore raw
In reply to Xuan Zhuo
Quoted 109 lines · Xuan Zhuo
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Xuan Zhuo

This patch allows the driver to obtain some statistics from the device.

In the back-end implementation, we can count a lot of such information, which can be used for debugging and judging the running status of the back-end. We hope to directly display it to the user through ethtool.

---

v8:
1. Modified based on comments by Cornelia Huck

v7:
1. add rx_reset, tx_reset
2. add device normative and dirver normative
3. add comments for *_packets, *_bytres

v6:
1. correct the names and descriptions of some stats items

v5:
1. add VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
2. more item for virtio_net_ctrl_reply_stats_queue_pair

v4:
1. remove dev_stats_num, {rx|tx}_stats_num
2. Use two commands to get the stats of queue pair and dev respectively

v3 changes:
1. add dev_version
2. use queue_pair_index replace rx_num, tx_num
3. Explain the processing when the device and driver support different numbers of stats

 conformance.tex |   2 +
 content.tex     | 133 +++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 134 insertions(+), 1 deletion(-)
conformance.tex +2 -0
index 42f8537..950924e 100644
@@ -142,6 +142,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
 \end{itemize}
 
 \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / Driver Conformance / Block Driver Conformance}
@@ -401,6 +402,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Gratuitous Packet Sending}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
 \end{itemize}
 
 \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / Device Conformance / Block Device Conformance}
content.tex +5 -0
index cf20570..fa385e2 100644
@@ -3092,6 +3092,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
     channel.
 
+\item[VIRTIO_NET_F_CTRL_STATS(55)] Device can provide device-level statistics
+    to the driver through the control channel.
+
 \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO
  (fragmenting the packet) the USO splits large UDP packet
  to several segments when each of these smaller packets has UDP header.
@@ -3137,6 +3140,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
 \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_CTRL_STATS] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
 \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
 \end{description}
@@ -4015,6 +4019,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
         u8 command;
         u8 command-specific-data[];
         u8 ack;
+        u8 command-specific-data-reply[];
 };
 
Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4025

Quoted 1 line
content.tex +0 -0
 /* ack values */

we now need to document which commands include the reply, otherwise reader will be confused.

OK.

Quoted 11 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4033

Quoted 9 lines
content.tex +2 -1
@@ -4023,7 +4028,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 \end{lstlisting}
 
 The \field{class}, \field{command} and command-specific-data are set by the
-driver, and the device sets the \field{ack} byte. There is little it can
+driver, and the device sets the \field{ack} byte and optionally
+\field{command-specific-data-reply}. There is little it can
 do

Now "it" refers to device where it used to refer to the driver.

Quoted 3 lines

There is little it can except issue a diagnostic if \field{ack} is not VIRTIO_NET_OK.

I don't feel like this has changed or affected anything.

Well we used to mention just driver. next sentence refers to driver. now you mention driver and device. what does "it" refer to"? it is now ambigous.

Quoted 1 line

Can you help me change it?

Replace "it" with "driver"?

Quoted 46 lines
Quoted 30 lines
Quoted 30 lines

except issue a diagnostic if \field{ack} is not
VIRTIO_NET_OK.

content.tex +21 -0
@@ -4471,6 +4477,131 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 according to the native endian of the guest rather than
 (necessarily when not using the legacy interface) little-endian.
 
+\paragraph{Device stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+If the VIRTIO_NET_F_CTRL_STATS feature is negotiated, the driver can
+get device stats from the device in \field{command-specific-data-reply}.
+
+To get the stats, the following definitions are used:
+\begin{lstlisting}
+#define VIRTIO_NET_CTRL_STATS                  6
+#define VIRTIO_NET_CTRL_STATS_GET_DEV          0
+#define VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ      1
+#define VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR   2
+\end{lstlisting}
+
+The following layout structures are used:
+
+\field{command-specific-data}
+\begin{lstlisting}
+struct virtio_net_ctrl_stats_queue_pair {
+    le64 queue_pair_index;
+}
+\end{lstlisting}
Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4505

Quoted 5 lines
content.tex +5 -0
+
+\field{command-specific-data-reply}
+\begin{lstlisting}
+struct virtio_net_ctrl_reply_stats_dev {
+    le64 dev_reset;         // The number of device resets.

while we already have some places using // comments, they are not used consistently. Better to use /* text */ for now.

OK.

Quoted 5 lines
Quoted 2 lines
content.tex +1 -0
+}
Quoted 8 lines
Quoted 4 lines

I am still worrying about this one. I think it's a bit vague. Some types of reset will zero this counter, others will not. Also, this looks more like a generic capability than anything specific to net. And this raises questions about .

I think, I should define that the device will not set this variable to 0 during reset.

again there's reset and there's reset. if you power cycle the system is device expected to retain this? the bus? what about pcie link level reset? etc

Quoted 3 lines

The purpose of adding this is to count the number of times the user deletes/adds the network card. The virtio-net device may be deleted/added by the user multiple times,

I don't get it really ... deletes/adds card how?

Quoted 2 lines

and this is reflected in the device layer as the number of resets.

Well your text does not explain it exactly.

Quoted 14 lines
Quoted 12 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4509

Quoted 6 lines
content.tex +3 -0
+
+struct virtio_net_ctrl_reply_stats_cvq {
+    le64 request_num; // The number of requests.

We call them commands elsewhere. It is however unclear whether the request for the stats itself is included in the counter. Also, do these counters reset on device reset? I would assume so but given dev_reset does not reset it's hard to be sure.

Quoted 4 lines

I will change request -> command.

We should really define that the device should not reset these statistics during reset

I don't think I agree. new device - new stats.. physical cards seem to reset these things.

Quoted 37 lines
Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4511

Quoted 4 lines
content.tex +2 -0
+    le64 ok_num;      // The number of ok acks.
+    le64 err_num;     // The number of err acks.

A bit more precise pls: e.g. "the number of commands where ack was VIRTIO_NET_OK"?

OK

Quoted 16 lines
Quoted 16 lines
content.tex +14 -0
+
+    le64 req_rx_promisc;         // The number of requests with command VIRTIO_NET_CTRL_RX_PROMISC.
+    le64 req_rx_allmulti;        // The number of requests with command VIRTIO_NET_CTRL_RX_ALLMULTI.
+    le64 req_rx_alluni;          // The number of requests with command VIRTIO_NET_CTRL_RX_ALLUNI.
+    le64 req_rx_nomulti;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOMULTI.
+    le64 req_rx_nouni;           // The number of requests with command VIRTIO_NET_CTRL_RX_NOUNI.
+    le64 req_rx_nobcast;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOBCAST.
+    le64 req_mac_table_set;      // The number of requests with command VIRTIO_NET_CTRL_MAC_TABLE_SET.
+    le64 req_mac_addr_set;       // The number of requests with command VIRTIO_NET_CTRL_MAC_ADDR_SET.
+    le64 req_vlan_add;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_ADD.
+    le64 req_vlan_del;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_DEL.
+    le64 req_announce_ack;       // The number of requests with command VIRTIO_NET_CTRL_ANNOUNCE_ACK.
+    le64 req_mq_vq_pairs_set;    // The number of requests with command VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.
+    le64 req_mq_rss_config;      // The number of requests with command VIRTIO_NET_CTRL_MQ_RSS_CONFIG.
Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4528

Quoted 3 lines
content.tex +3 -0
+    le64 req_mq_hash_config;     // The number of requests with command VIRTIO_NET_CTRL_MQ_HASH_CONFIG.
+    le64 req_guest_offloads_set; // The number of requests with command VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET.
+}

I just had an idea:
how about the query passes in the command we are debugging instead? this way we do not need to extend this each time.

Quoted 8 lines

Do you mean command-specific-data uses a structure like this:

{
	le64 class;
	le64 command;
}

This is indeed a really great idea.

can't say what you mean exactly here. just pass the command you want stats about to the card, is my idea.

Quoted 24 lines
Quoted 13 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4533

Quoted 9 lines
content.tex +5 -0
+
+struct virtio_net_ctrl_reply_stats_queue_pair {
+    /* rx stats */
+    le64 rx_packets;          // The number of packets recived by device, including the dropped packets by device.
+    le64 rx_bytes;            // The number of bytes recived by device, including the dropped packets by device.

typos: received

Quoted 10 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4540

Quoted 7 lines
content.tex +7 -0
+
+    le64 rx_notification;     // The number of notifications from driver.
+    le64 rx_interrupt;        // The number of interrupts generated by device.
+
+    le64 rx_drop;             // The number of packets dropped by the rx queue. Contains all kinds of packet drop.
+    le64 rx_drop_overruns;    // The number of packets dropped by the rx queue when no more descriptors were available.
+    le64 rx_drop_oversize;    // The number of oversized packets received by the rx queue.

What does oversized mean in this context? does it apply to mergeable buffers?

Quoted 1 line

For mergeable buffers this statistic is meaningless.

something to mention then. or should we bother at all?

Quoted 29 lines
Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4545

Quoted 6 lines
content.tex +5 -0
+
+    le64 rx_csum_valid;       // The number of packets with VIRTIO_NET_HDR_F_DATA_VALID.
+    le64 rx_csum_partial;     // The number of packets with VIRTIO_NET_HDR_F_NEEDS_CSUM.
+    le64 rx_csum_bad;         // The number of packets with abnormal csum.
+    le64 rx_csum_none;        // The number of packets without hardware csum.

I assume they are only incremented with VIRTIO_NET_F_GUEST_CSUM?

Yes

Quoted 6 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4547

Quoted 3 lines
content.tex +2 -0
+
+    le64 rx_gso_packets;      // The number of gso packets received by rx.

let's spell it out: I think this means any packet put into VQ and with gso_type != VIRTIO_NET_HDR_GSO_NONE

Yes

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4548

Quoted 2 lines
content.tex +1 -0
+    le64 rx_gso_bytes;        // The number of gso bytes received by rx.

this one is a bit vague: I assume these are bytes written into the RX VQ buffers? the difference is that multiple packets are combined into a
single gso packet.
and does this include the virtio net header?

Quoted 6 lines

I think all *_bytes statistics do not include virtio net header, the packet bytes we care about does not include virtio net header.

Multiple packets are combined into one, and some network headers, such as ip/tcp, etc., are also removed. The final count is the bytes that are finalized to the RV VQ buffers.

sounds quite tricky to document. but go ahead and try if you like.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
content.tex +5 -0
+    le64 rx_reset;            // The number of queue resets.
+
+    /* tx stats */
+    le64 tx_packets;          // The number of packets sent by device, excluding the dropped packets by device.
+    le64 tx_bytes;            // The number of bytes sent by device, excluding the dropped packets by device.
Quoted 4 lines
Quoted 2 lines

same question: are we talking about bytes sent by device? or received from guest for transmission?

The number of bytes sent by device.

not symmetrical with rx then in case of gso.

Quoted 11 lines
Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4559

Quoted 7 lines
content.tex +6 -0
+
+    le64 tx_notification;     // The number of notifications from driver.
+    le64 tx_interrupt;        // The number of interrupts generated by device.
+
+    le64 tx_drop;             // The number of packets dropped by the tx queue. Contains all kinds of packet drop.
+    le64 tx_drop_desc_err;    // The number of packets dropped when the descriptor is in an error state.

what is a descriptor in error state?

such as len == 0.

We never really documented what does it do.
maybe malformed packets would be better?
buffer too short?
something more specific.

Quoted 12 lines
Quoted 11 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4565

Quoted 8 lines
content.tex +6 -0
+
+    le64 tx_csum_none;        // The number of packets that doesn't require hardware csum.
+    le64 tx_csum_partial;     // The number of packets that requires hardware csum.
+
+    le64 tx_gso_packets;      // The number of gso packets transmitted.
+    le64 tx_gso_bytes;        // The number of gso bytes transmitted.

point is, gso packets are not transmitted. they are received from guest. is this what this means?

Quoted 8 lines

Yes.

Quoted 2 lines

gso bytes probably somehow
means bytes in gso packets, but again this is a bit vague.

It should be defined, I think it should be defined as the number of bytes from the guest.

above you said bytes sent.

Quoted 7 lines
Quoted 6 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4566

Quoted 2 lines
content.tex +1 -0
+    le64 tx_reset;            // The number of queue resets.

queue reset counter sounds like a useful generic capability. is this with the new features jason added recently? can we add it to all devices?

Quoted 8 lines

It's the one I submitted earlier.

	https://github.com/oasis-tcs/virtio-spec/issues/124

The purpose of adding this is that the network device will disable/enable the queue. The statistics of the corresponding device is queue_reset.

Thank you very much.

need to document the dependency on the feature.

Quoted 50 lines
Quoted 50 lines
Quoted 49 lines
content.tex +38 -1
+}
+\end{lstlisting}
+
+All device stats are divided into three categories:
+\begin{itemize}
+    \item the stats of the device.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_DEV
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_dev
+        \end{itemize}
+
+    \item the stats of the controlq.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_cvq
+        \end{itemize}
+
+    \item the stats of the queue pair. This contains the stats of rx and tx.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR
+            \item command-specific-data structure: virtio_net_ctrl_stats_queue_pair
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_queue_pair
+        \end{itemize}
+\end{itemize}
+
+\devicenormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+If \field{queue_pair_index} is out of range for a
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR command, the device MUST set \field{ack} in
+virtio_net_ctrl to VIRTIO_NET_ERR.
+
+If VIRTIO_NET_F_CTRL_STATS has been negotiated, the device MUST support the
+VIRTIO_NET_CTRL_STATS_GET_DEV, VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ and
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR commands.
+
+\drivernormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+\field{command-specific-data} MUST be empty for VIRTIO_NET_CTRL_STATS_GET_DEV
+and VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ.
 
 \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
 Types / Network Device / Legacy Interface: Framing Requirements}
--

2.31.0

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

author
#12 lore raw
Quoted 114 lines · Michael S. Tsirkin
Quoted 109 lines · Xuan Zhuo
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Xuan Zhuo

This patch allows the driver to obtain some statistics from the device.

In the back-end implementation, we can count a lot of such information, which can be used for debugging and judging the running status of the back-end. We hope to directly display it to the user through ethtool.

---

v8:
1. Modified based on comments by Cornelia Huck

v7:
1. add rx_reset, tx_reset
2. add device normative and dirver normative
3. add comments for *_packets, *_bytres

v6:
1. correct the names and descriptions of some stats items

v5:
1. add VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
2. more item for virtio_net_ctrl_reply_stats_queue_pair

v4:
1. remove dev_stats_num, {rx|tx}_stats_num
2. Use two commands to get the stats of queue pair and dev respectively

v3 changes:
1. add dev_version
2. use queue_pair_index replace rx_num, tx_num
3. Explain the processing when the device and driver support different numbers of stats

 conformance.tex |   2 +
 content.tex     | 133 +++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 134 insertions(+), 1 deletion(-)
conformance.tex +2 -0
index 42f8537..950924e 100644
@@ -142,6 +142,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
 \end{itemize}
 
 \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / Driver Conformance / Block Driver Conformance}
@@ -401,6 +402,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Gratuitous Packet Sending}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
 \end{itemize}
 
 \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / Device Conformance / Block Device Conformance}
content.tex +5 -0
index cf20570..fa385e2 100644
@@ -3092,6 +3092,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
     channel.
 
+\item[VIRTIO_NET_F_CTRL_STATS(55)] Device can provide device-level statistics
+    to the driver through the control channel.
+
 \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO
  (fragmenting the packet) the USO splits large UDP packet
  to several segments when each of these smaller packets has UDP header.
@@ -3137,6 +3140,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
 \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_CTRL_STATS] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
 \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
 \end{description}
@@ -4015,6 +4019,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
         u8 command;
         u8 command-specific-data[];
         u8 ack;
+        u8 command-specific-data-reply[];
 };
 
Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4025

Quoted 1 line
content.tex +0 -0
 /* ack values */

we now need to document which commands include the reply, otherwise reader will be confused.

OK.

Quoted 11 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4033

Quoted 9 lines
content.tex +2 -1
@@ -4023,7 +4028,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 \end{lstlisting}
 
 The \field{class}, \field{command} and command-specific-data are set by the
-driver, and the device sets the \field{ack} byte. There is little it can
+driver, and the device sets the \field{ack} byte and optionally
+\field{command-specific-data-reply}. There is little it can
 do

Now "it" refers to device where it used to refer to the driver.

Quoted 3 lines

There is little it can except issue a diagnostic if \field{ack} is not VIRTIO_NET_OK.

I don't feel like this has changed or affected anything.

Quoted 10 lines

Well we used to mention just driver. next sentence refers to driver. now you mention driver and device. what does "it" refer to"? it is now ambigous.

Quoted 1 line

Can you help me change it?

Replace "it" with "driver"?

OK. I see.

Quoted 52 lines
Quoted 52 lines
Quoted 30 lines
Quoted 30 lines

except issue a diagnostic if \field{ack} is not
VIRTIO_NET_OK.

content.tex +21 -0
@@ -4471,6 +4477,131 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 according to the native endian of the guest rather than
 (necessarily when not using the legacy interface) little-endian.
 
+\paragraph{Device stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+If the VIRTIO_NET_F_CTRL_STATS feature is negotiated, the driver can
+get device stats from the device in \field{command-specific-data-reply}.
+
+To get the stats, the following definitions are used:
+\begin{lstlisting}
+#define VIRTIO_NET_CTRL_STATS                  6
+#define VIRTIO_NET_CTRL_STATS_GET_DEV          0
+#define VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ      1
+#define VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR   2
+\end{lstlisting}
+
+The following layout structures are used:
+
+\field{command-specific-data}
+\begin{lstlisting}
+struct virtio_net_ctrl_stats_queue_pair {
+    le64 queue_pair_index;
+}
+\end{lstlisting}
Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4505

Quoted 5 lines
content.tex +5 -0
+
+\field{command-specific-data-reply}
+\begin{lstlisting}
+struct virtio_net_ctrl_reply_stats_dev {
+    le64 dev_reset;         // The number of device resets.

while we already have some places using // comments, they are not used consistently. Better to use /* text */ for now.

OK.

Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4506

Quoted 2 lines
content.tex +1 -0
+}

I am still worrying about this one. I think it's a bit vague. Some types of reset will zero this counter, others will not. Also, this looks more like a generic capability than anything specific to net. And this raises questions about .

Quoted 6 lines
Quoted 2 lines

I think, I should define that the device will not set this variable to 0 during reset.

again there's reset and there's reset. if you power cycle the system is device expected to retain this? the bus? what about pcie link level reset? etc

So it's wrong to record statistics in reset, right?

Quoted 6 lines
Quoted 4 lines

The purpose of adding this is to count the number of times the user deletes/adds the network card. The virtio-net device may be deleted/added by the user multiple times,

I don't get it really ... deletes/adds card how?

For virtio-net, "ip link del eth1" is not supported. I don't know why. I usually use this command.

echo virtio1 > /sys/class/net/eth1/device/driver/unbind

Quoted 22 lines
Quoted 3 lines

and this is reflected in the device layer as the number of resets.

Well your text does not explain it exactly.

Quoted 14 lines
Quoted 12 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4509

Quoted 6 lines
content.tex +3 -0
+
+struct virtio_net_ctrl_reply_stats_cvq {
+    le64 request_num; // The number of requests.

We call them commands elsewhere. It is however unclear whether the request for the stats itself is included in the counter. Also, do these counters reset on device reset? I would assume so but given dev_reset does not reset it's hard to be sure.

Quoted 7 lines
Quoted 4 lines

I will change request -> command.

We should really define that the device should not reset these statistics during reset

I don't think I agree. new device - new stats.. physical cards seem to reset these things.

ok, let's confirm this first. After reset, all statistics will start from 0 again.

Quoted 45 lines
Quoted 45 lines
Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4511

Quoted 5 lines
content.tex +2 -0
+    le64 ok_num;      // The number of ok acks.
+    le64 err_num;     // The number of err acks.

A bit more precise pls: e.g. "the number of commands where ack was VIRTIO_NET_OK"?

OK

Quoted 16 lines
Quoted 16 lines
content.tex +14 -0
+
+    le64 req_rx_promisc;         // The number of requests with command VIRTIO_NET_CTRL_RX_PROMISC.
+    le64 req_rx_allmulti;        // The number of requests with command VIRTIO_NET_CTRL_RX_ALLMULTI.
+    le64 req_rx_alluni;          // The number of requests with command VIRTIO_NET_CTRL_RX_ALLUNI.
+    le64 req_rx_nomulti;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOMULTI.
+    le64 req_rx_nouni;           // The number of requests with command VIRTIO_NET_CTRL_RX_NOUNI.
+    le64 req_rx_nobcast;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOBCAST.
+    le64 req_mac_table_set;      // The number of requests with command VIRTIO_NET_CTRL_MAC_TABLE_SET.
+    le64 req_mac_addr_set;       // The number of requests with command VIRTIO_NET_CTRL_MAC_ADDR_SET.
+    le64 req_vlan_add;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_ADD.
+    le64 req_vlan_del;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_DEL.
+    le64 req_announce_ack;       // The number of requests with command VIRTIO_NET_CTRL_ANNOUNCE_ACK.
+    le64 req_mq_vq_pairs_set;    // The number of requests with command VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.
+    le64 req_mq_rss_config;      // The number of requests with command VIRTIO_NET_CTRL_MQ_RSS_CONFIG.
Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4528

Quoted 3 lines
content.tex +3 -0
+    le64 req_mq_hash_config;     // The number of requests with command VIRTIO_NET_CTRL_MQ_HASH_CONFIG.
+    le64 req_guest_offloads_set; // The number of requests with command VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET.
+}

I just had an idea:
how about the query passes in the command we are debugging instead? this way we do not need to extend this each time.

Do you mean command-specific-data uses a structure like this:

{
	le64 class;
	le64 command;
}
Quoted 5 lines
Quoted 1 line

This is indeed a really great idea.

can't say what you mean exactly here. just pass the command you want stats about to the card, is my idea.

I mean pass the desired command (class, command) to the device in command-specific-data.

Include only a num in the command-specific-data-reply as a response from the device.

struct {
	le64 num;
}
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4533

Quoted 10 lines
content.tex +5 -0
+
+struct virtio_net_ctrl_reply_stats_queue_pair {
+    /* rx stats */
+    le64 rx_packets;          // The number of packets recived by device, including the dropped packets by device.
+    le64 rx_bytes;            // The number of bytes recived by device, including the dropped packets by device.

typos: received

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4540

Quoted 7 lines
content.tex +7 -0
+
+    le64 rx_notification;     // The number of notifications from driver.
+    le64 rx_interrupt;        // The number of interrupts generated by device.
+
+    le64 rx_drop;             // The number of packets dropped by the rx queue. Contains all kinds of packet drop.
+    le64 rx_drop_overruns;    // The number of packets dropped by the rx queue when no more descriptors were available.
+    le64 rx_drop_oversize;    // The number of oversized packets received by the rx queue.

What does oversized mean in this context? does it apply to mergeable buffers?

Quoted 3 lines
Quoted 1 line

For mergeable buffers this statistic is meaningless.

something to mention then. or should we bother at all?

sorry i didn't understand.

Quoted 65 lines
Quoted 30 lines
Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4545

Quoted 7 lines
content.tex +5 -0
+
+    le64 rx_csum_valid;       // The number of packets with VIRTIO_NET_HDR_F_DATA_VALID.
+    le64 rx_csum_partial;     // The number of packets with VIRTIO_NET_HDR_F_NEEDS_CSUM.
+    le64 rx_csum_bad;         // The number of packets with abnormal csum.
+    le64 rx_csum_none;        // The number of packets without hardware csum.

I assume they are only incremented with VIRTIO_NET_F_GUEST_CSUM?

Yes

Quoted 6 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4547

Quoted 3 lines
content.tex +2 -0
+
+    le64 rx_gso_packets;      // The number of gso packets received by rx.

let's spell it out: I think this means any packet put into VQ and with gso_type != VIRTIO_NET_HDR_GSO_NONE

Yes

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4548

Quoted 2 lines
content.tex +1 -0
+    le64 rx_gso_bytes;        // The number of gso bytes received by rx.

this one is a bit vague: I assume these are bytes written into the RX VQ buffers? the difference is that multiple packets are combined into a
single gso packet.
and does this include the virtio net header?

Quoted 6 lines

I think all *_bytes statistics do not include virtio net header, the packet bytes we care about does not include virtio net header.

Multiple packets are combined into one, and some network headers, such as ip/tcp, etc., are also removed. The final count is the bytes that are finalized to the RV VQ buffers.

sounds quite tricky to document. but go ahead and try if you like.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
content.tex +5 -0
+    le64 rx_reset;            // The number of queue resets.
+
+    /* tx stats */
+    le64 tx_packets;          // The number of packets sent by device, excluding the dropped packets by device.
+    le64 tx_bytes;            // The number of bytes sent by device, excluding the dropped packets by device.
Quoted 4 lines
Quoted 2 lines

same question: are we talking about bytes sent by device? or received from guest for transmission?

The number of bytes sent by device.

not symmetrical with rx then in case of gso.

Quoted 10 lines
Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4559

Quoted 7 lines
content.tex +6 -0
+
+    le64 tx_notification;     // The number of notifications from driver.
+    le64 tx_interrupt;        // The number of interrupts generated by device.
+
+    le64 tx_drop;             // The number of packets dropped by the tx queue. Contains all kinds of packet drop.
+    le64 tx_drop_desc_err;    // The number of packets dropped when the descriptor is in an error state.

what is a descriptor in error state?

Quoted 6 lines
Quoted 1 line

such as len == 0.

We never really documented what does it do.
maybe malformed packets would be better?
buffer too short?
something more specific.

OK.

Quoted 15 lines
Quoted 15 lines
Quoted 12 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4565

Quoted 9 lines
content.tex +6 -0
+
+    le64 tx_csum_none;        // The number of packets that doesn't require hardware csum.
+    le64 tx_csum_partial;     // The number of packets that requires hardware csum.
+
+    le64 tx_gso_packets;      // The number of gso packets transmitted.
+    le64 tx_gso_bytes;        // The number of gso bytes transmitted.

point is, gso packets are not transmitted. they are received from guest. is this what this means?

Yes.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

gso bytes probably somehow
means bytes in gso packets, but again this is a bit vague.

It should be defined, I think it should be defined as the number of bytes from the guest.

above you said bytes sent.

I mean, all about GSO are statistics based on big packets. So here is the package from Guest, and rx is the package passed to Guest.

Quoted 12 lines
Quoted 12 lines
Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4566

Quoted 3 lines
content.tex +1 -0
+    le64 tx_reset;            // The number of queue resets.

queue reset counter sounds like a useful generic capability. is this with the new features jason added recently? can we add it to all devices?

It's the one I submitted earlier.

	https://github.com/oasis-tcs/virtio-spec/issues/124
Quoted 7 lines
Quoted 4 lines

The purpose of adding this is that the network device will disable/enable the queue. The statistics of the corresponding device is queue_reset.

Thank you very much.

need to document the dependency on the feature.

OK.

Thanks.

Quoted 52 lines
Quoted 51 lines
Quoted 51 lines
Quoted 50 lines
content.tex +38 -1
+}
+\end{lstlisting}
+
+All device stats are divided into three categories:
+\begin{itemize}
+    \item the stats of the device.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_DEV
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_dev
+        \end{itemize}
+
+    \item the stats of the controlq.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_cvq
+        \end{itemize}
+
+    \item the stats of the queue pair. This contains the stats of rx and tx.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR
+            \item command-specific-data structure: virtio_net_ctrl_stats_queue_pair
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_queue_pair
+        \end{itemize}
+\end{itemize}
+
+\devicenormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+If \field{queue_pair_index} is out of range for a
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR command, the device MUST set \field{ack} in
+virtio_net_ctrl to VIRTIO_NET_ERR.
+
+If VIRTIO_NET_F_CTRL_STATS has been negotiated, the device MUST support the
+VIRTIO_NET_CTRL_STATS_GET_DEV, VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ and
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR commands.
+
+\drivernormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+\field{command-specific-data} MUST be empty for VIRTIO_NET_CTRL_STATS_GET_DEV
+and VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ.
 
 \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
 Types / Network Device / Legacy Interface: Framing Requirements}
--

2.31.0

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#13 lore raw
Quoted 180 lines · Xuan Zhuo
Quoted 114 lines · Michael S. Tsirkin
Quoted 109 lines · Xuan Zhuo
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Xuan Zhuo

This patch allows the driver to obtain some statistics from the device.

In the back-end implementation, we can count a lot of such information, which can be used for debugging and judging the running status of the back-end. We hope to directly display it to the user through ethtool.

---

v8:
1. Modified based on comments by Cornelia Huck

v7:
1. add rx_reset, tx_reset
2. add device normative and dirver normative
3. add comments for *_packets, *_bytres

v6:
1. correct the names and descriptions of some stats items

v5:
1. add VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
2. more item for virtio_net_ctrl_reply_stats_queue_pair

v4:
1. remove dev_stats_num, {rx|tx}_stats_num
2. Use two commands to get the stats of queue pair and dev respectively

v3 changes:
1. add dev_version
2. use queue_pair_index replace rx_num, tx_num
3. Explain the processing when the device and driver support different numbers of stats

 conformance.tex |   2 +
 content.tex     | 133 +++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 134 insertions(+), 1 deletion(-)
conformance.tex +2 -0
index 42f8537..950924e 100644
@@ -142,6 +142,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
 \end{itemize}
 
 \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / Driver Conformance / Block Driver Conformance}
@@ -401,6 +402,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Gratuitous Packet Sending}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
 \end{itemize}
 
 \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / Device Conformance / Block Device Conformance}
content.tex +5 -0
index cf20570..fa385e2 100644
@@ -3092,6 +3092,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
     channel.
 
+\item[VIRTIO_NET_F_CTRL_STATS(55)] Device can provide device-level statistics
+    to the driver through the control channel.
+
 \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO
  (fragmenting the packet) the USO splits large UDP packet
  to several segments when each of these smaller packets has UDP header.
@@ -3137,6 +3140,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
 \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_CTRL_STATS] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
 \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
 \end{description}
@@ -4015,6 +4019,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
         u8 command;
         u8 command-specific-data[];
         u8 ack;
+        u8 command-specific-data-reply[];
 };
 
Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4025

Quoted 1 line
content.tex +0 -0
 /* ack values */

we now need to document which commands include the reply, otherwise reader will be confused.

OK.

Quoted 11 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4033

Quoted 9 lines
content.tex +2 -1
@@ -4023,7 +4028,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 \end{lstlisting}
 
 The \field{class}, \field{command} and command-specific-data are set by the
-driver, and the device sets the \field{ack} byte. There is little it can
+driver, and the device sets the \field{ack} byte and optionally
+\field{command-specific-data-reply}. There is little it can
 do

Now "it" refers to device where it used to refer to the driver.

Quoted 3 lines

There is little it can except issue a diagnostic if \field{ack} is not VIRTIO_NET_OK.

I don't feel like this has changed or affected anything.

Quoted 10 lines

Well we used to mention just driver. next sentence refers to driver. now you mention driver and device. what does "it" refer to"? it is now ambigous.

Quoted 1 line

Can you help me change it?

Replace "it" with "driver"?

OK. I see.

Quoted 52 lines
Quoted 52 lines
Quoted 30 lines
Quoted 30 lines

except issue a diagnostic if \field{ack} is not
VIRTIO_NET_OK.

content.tex +21 -0
@@ -4471,6 +4477,131 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 according to the native endian of the guest rather than
 (necessarily when not using the legacy interface) little-endian.
 
+\paragraph{Device stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+If the VIRTIO_NET_F_CTRL_STATS feature is negotiated, the driver can
+get device stats from the device in \field{command-specific-data-reply}.
+
+To get the stats, the following definitions are used:
+\begin{lstlisting}
+#define VIRTIO_NET_CTRL_STATS                  6
+#define VIRTIO_NET_CTRL_STATS_GET_DEV          0
+#define VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ      1
+#define VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR   2
+\end{lstlisting}
+
+The following layout structures are used:
+
+\field{command-specific-data}
+\begin{lstlisting}
+struct virtio_net_ctrl_stats_queue_pair {
+    le64 queue_pair_index;
+}
+\end{lstlisting}
Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4505

Quoted 5 lines
content.tex +5 -0
+
+\field{command-specific-data-reply}
+\begin{lstlisting}
+struct virtio_net_ctrl_reply_stats_dev {
+    le64 dev_reset;         // The number of device resets.

while we already have some places using // comments, they are not used consistently. Better to use /* text */ for now.

OK.

Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4506

Quoted 2 lines
content.tex +1 -0
+}

I am still worrying about this one. I think it's a bit vague. Some types of reset will zero this counter, others will not. Also, this looks more like a generic capability than anything specific to net. And this raises questions about .

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

I think, I should define that the device will not set this variable to 0 during reset.

again there's reset and there's reset. if you power cycle the system is device expected to retain this? the bus? what about pcie link level reset? etc

So it's wrong to record statistics in reset, right?

depends on the kind of reset. for some kinds its outright impossible. it is certainly easier not to worry about this and just reset everything.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

The purpose of adding this is to count the number of times the user deletes/adds the network card. The virtio-net device may be deleted/added by the user multiple times,

I don't get it really ... deletes/adds card how?

For virtio-net, "ip link del eth1" is not supported. I don't know why.

because what it does is add a virtual link, and virtio does not support vlan offloading atm?

so I would maybe add vlan offloading instead ;)

Quoted 4 lines

I usually
use this command.

echo virtio1 > /sys/class/net/eth1/device/driver/unbind

and that unbinds the driver from the device

Quoted 122 lines
Quoted 22 lines
Quoted 3 lines

and this is reflected in the device layer as the number of resets.

Well your text does not explain it exactly.

Quoted 14 lines
Quoted 12 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4509

Quoted 6 lines
content.tex +3 -0
+
+struct virtio_net_ctrl_reply_stats_cvq {
+    le64 request_num; // The number of requests.

We call them commands elsewhere. It is however unclear whether the request for the stats itself is included in the counter. Also, do these counters reset on device reset? I would assume so but given dev_reset does not reset it's hard to be sure.

Quoted 7 lines
Quoted 4 lines

I will change request -> command.

We should really define that the device should not reset these statistics during reset

I don't think I agree. new device - new stats.. physical cards seem to reset these things.

ok, let's confirm this first. After reset, all statistics will start from 0 again.

Quoted 45 lines
Quoted 45 lines
Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4511

Quoted 5 lines
content.tex +2 -0
+    le64 ok_num;      // The number of ok acks.
+    le64 err_num;     // The number of err acks.

A bit more precise pls: e.g. "the number of commands where ack was VIRTIO_NET_OK"?

OK

Quoted 16 lines
Quoted 16 lines
content.tex +14 -0
+
+    le64 req_rx_promisc;         // The number of requests with command VIRTIO_NET_CTRL_RX_PROMISC.
+    le64 req_rx_allmulti;        // The number of requests with command VIRTIO_NET_CTRL_RX_ALLMULTI.
+    le64 req_rx_alluni;          // The number of requests with command VIRTIO_NET_CTRL_RX_ALLUNI.
+    le64 req_rx_nomulti;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOMULTI.
+    le64 req_rx_nouni;           // The number of requests with command VIRTIO_NET_CTRL_RX_NOUNI.
+    le64 req_rx_nobcast;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOBCAST.
+    le64 req_mac_table_set;      // The number of requests with command VIRTIO_NET_CTRL_MAC_TABLE_SET.
+    le64 req_mac_addr_set;       // The number of requests with command VIRTIO_NET_CTRL_MAC_ADDR_SET.
+    le64 req_vlan_add;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_ADD.
+    le64 req_vlan_del;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_DEL.
+    le64 req_announce_ack;       // The number of requests with command VIRTIO_NET_CTRL_ANNOUNCE_ACK.
+    le64 req_mq_vq_pairs_set;    // The number of requests with command VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.
+    le64 req_mq_rss_config;      // The number of requests with command VIRTIO_NET_CTRL_MQ_RSS_CONFIG.
Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4528

Quoted 3 lines
content.tex +3 -0
+    le64 req_mq_hash_config;     // The number of requests with command VIRTIO_NET_CTRL_MQ_HASH_CONFIG.
+    le64 req_guest_offloads_set; // The number of requests with command VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET.
+}

I just had an idea:
how about the query passes in the command we are debugging instead? this way we do not need to extend this each time.

Do you mean command-specific-data uses a structure like this:

{
	le64 class;
	le64 command;
}
Quoted 5 lines
Quoted 1 line

This is indeed a really great idea.

can't say what you mean exactly here. just pass the command you want stats about to the card, is my idea.

I mean pass the desired command (class, command) to the device in command-specific-data.

Include only a num in the command-specific-data-reply as a response from the device.

struct {
	le64 num;
}
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4533

Quoted 10 lines
content.tex +5 -0
+
+struct virtio_net_ctrl_reply_stats_queue_pair {
+    /* rx stats */
+    le64 rx_packets;          // The number of packets recived by device, including the dropped packets by device.
+    le64 rx_bytes;            // The number of bytes recived by device, including the dropped packets by device.

typos: received

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4540

Quoted 7 lines
content.tex +7 -0
+
+    le64 rx_notification;     // The number of notifications from driver.
+    le64 rx_interrupt;        // The number of interrupts generated by device.
+
+    le64 rx_drop;             // The number of packets dropped by the rx queue. Contains all kinds of packet drop.
+    le64 rx_drop_overruns;    // The number of packets dropped by the rx queue when no more descriptors were available.
+    le64 rx_drop_oversize;    // The number of oversized packets received by the rx queue.

What does oversized mean in this context? does it apply to mergeable buffers?

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

For mergeable buffers this statistic is meaningless.

something to mention then. or should we bother at all?

sorry i didn't understand.

is this a common thing? why do we need this counter?

Quoted 92 lines
Quoted 65 lines
Quoted 30 lines
Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4545

Quoted 7 lines
content.tex +5 -0
+
+    le64 rx_csum_valid;       // The number of packets with VIRTIO_NET_HDR_F_DATA_VALID.
+    le64 rx_csum_partial;     // The number of packets with VIRTIO_NET_HDR_F_NEEDS_CSUM.
+    le64 rx_csum_bad;         // The number of packets with abnormal csum.
+    le64 rx_csum_none;        // The number of packets without hardware csum.

I assume they are only incremented with VIRTIO_NET_F_GUEST_CSUM?

Yes

Quoted 6 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4547

Quoted 3 lines
content.tex +2 -0
+
+    le64 rx_gso_packets;      // The number of gso packets received by rx.

let's spell it out: I think this means any packet put into VQ and with gso_type != VIRTIO_NET_HDR_GSO_NONE

Yes

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4548

Quoted 2 lines
content.tex +1 -0
+    le64 rx_gso_bytes;        // The number of gso bytes received by rx.

this one is a bit vague: I assume these are bytes written into the RX VQ buffers? the difference is that multiple packets are combined into a
single gso packet.
and does this include the virtio net header?

Quoted 6 lines

I think all *_bytes statistics do not include virtio net header, the packet bytes we care about does not include virtio net header.

Multiple packets are combined into one, and some network headers, such as ip/tcp, etc., are also removed. The final count is the bytes that are finalized to the RV VQ buffers.

sounds quite tricky to document. but go ahead and try if you like.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
content.tex +5 -0
+    le64 rx_reset;            // The number of queue resets.
+
+    /* tx stats */
+    le64 tx_packets;          // The number of packets sent by device, excluding the dropped packets by device.
+    le64 tx_bytes;            // The number of bytes sent by device, excluding the dropped packets by device.
Quoted 4 lines
Quoted 2 lines

same question: are we talking about bytes sent by device? or received from guest for transmission?

The number of bytes sent by device.

not symmetrical with rx then in case of gso.

Quoted 10 lines
Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4559

Quoted 7 lines
content.tex +6 -0
+
+    le64 tx_notification;     // The number of notifications from driver.
+    le64 tx_interrupt;        // The number of interrupts generated by device.
+
+    le64 tx_drop;             // The number of packets dropped by the tx queue. Contains all kinds of packet drop.
+    le64 tx_drop_desc_err;    // The number of packets dropped when the descriptor is in an error state.

what is a descriptor in error state?

Quoted 6 lines
Quoted 1 line

such as len == 0.

We never really documented what does it do.
maybe malformed packets would be better?
buffer too short?
something more specific.

OK.

Quoted 18 lines
Quoted 18 lines
Quoted 12 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4565

Quoted 9 lines
content.tex +6 -0
+
+    le64 tx_csum_none;        // The number of packets that doesn't require hardware csum.
+    le64 tx_csum_partial;     // The number of packets that requires hardware csum.
+
+    le64 tx_gso_packets;      // The number of gso packets transmitted.
+    le64 tx_gso_bytes;        // The number of gso bytes transmitted.

point is, gso packets are not transmitted. they are received from guest. is this what this means?

Yes.

Quoted 2 lines

gso bytes probably somehow
means bytes in gso packets, but again this is a bit vague.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

It should be defined, I think it should be defined as the number of bytes from the guest.

above you said bytes sent.

I mean, all about GSO are statistics based on big packets. So here is the package from Guest, and rx is the package passed to Guest.

it's a subtle distinction that we need to make.

Quoted 77 lines
Quoted 13 lines
Quoted 13 lines
Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4566

Quoted 4 lines
content.tex +1 -0
+    le64 tx_reset;            // The number of queue resets.

queue reset counter sounds like a useful generic capability. is this with the new features jason added recently? can we add it to all devices?

It's the one I submitted earlier.

	https://github.com/oasis-tcs/virtio-spec/issues/124
Quoted 7 lines
Quoted 4 lines

The purpose of adding this is that the network device will disable/enable the queue. The statistics of the corresponding device is queue_reset.

Thank you very much.

need to document the dependency on the feature.

OK.

Thanks.

Quoted 52 lines
Quoted 51 lines
Quoted 51 lines
Quoted 50 lines
content.tex +38 -1
+}
+\end{lstlisting}
+
+All device stats are divided into three categories:
+\begin{itemize}
+    \item the stats of the device.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_DEV
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_dev
+        \end{itemize}
+
+    \item the stats of the controlq.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_cvq
+        \end{itemize}
+
+    \item the stats of the queue pair. This contains the stats of rx and tx.
+        \begin{itemize}
+            \item command: VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR
+            \item command-specific-data structure: virtio_net_ctrl_stats_queue_pair
+            \item command-specific-data-reply structure: virtio_net_ctrl_reply_stats_queue_pair
+        \end{itemize}
+\end{itemize}
+
+\devicenormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+If \field{queue_pair_index} is out of range for a
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR command, the device MUST set \field{ack} in
+virtio_net_ctrl to VIRTIO_NET_ERR.
+
+If VIRTIO_NET_F_CTRL_STATS has been negotiated, the device MUST support the
+VIRTIO_NET_CTRL_STATS_GET_DEV, VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ and
+VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR commands.
+
+\drivernormative{\subparagraph}{Device stats}{Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+\field{command-specific-data} MUST be empty for VIRTIO_NET_CTRL_STATS_GET_DEV
+and VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ.
 
 \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
 Types / Network Device / Legacy Interface: Framing Requirements}
--

2.31.0

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

author
#14 lore raw
Quoted 188 lines · Michael S. Tsirkin
Quoted 187 lines · Xuan Zhuo
Quoted 114 lines · Michael S. Tsirkin
Quoted 109 lines · Xuan Zhuo
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Xuan Zhuo

This patch allows the driver to obtain some statistics from the device.

In the back-end implementation, we can count a lot of such information, which can be used for debugging and judging the running status of the back-end. We hope to directly display it to the user through ethtool.

---

v8:
1. Modified based on comments by Cornelia Huck

v7:
1. add rx_reset, tx_reset
2. add device normative and dirver normative
3. add comments for *_packets, *_bytres

v6:
1. correct the names and descriptions of some stats items

v5:
1. add VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ
2. more item for virtio_net_ctrl_reply_stats_queue_pair

v4:
1. remove dev_stats_num, {rx|tx}_stats_num
2. Use two commands to get the stats of queue pair and dev respectively

v3 changes:
1. add dev_version
2. use queue_pair_index replace rx_num, tx_num
3. Explain the processing when the device and driver support different numbers of stats

 conformance.tex |   2 +
 content.tex     | 133 +++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 134 insertions(+), 1 deletion(-)
conformance.tex +2 -0
index 42f8537..950924e 100644
@@ -142,6 +142,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
 \end{itemize}
 
 \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / Driver Conformance / Block Driver Conformance}
@@ -401,6 +402,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Gratuitous Packet Sending}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
 \end{itemize}
 
 \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / Device Conformance / Block Device Conformance}
content.tex +5 -0
index cf20570..fa385e2 100644
@@ -3092,6 +3092,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
     channel.
 
+\item[VIRTIO_NET_F_CTRL_STATS(55)] Device can provide device-level statistics
+    to the driver through the control channel.
+
 \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO
  (fragmenting the packet) the USO splits large UDP packet
  to several segments when each of these smaller packets has UDP header.
@@ -3137,6 +3140,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
 \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_CTRL_STATS] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
 \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
 \end{description}
@@ -4015,6 +4019,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
         u8 command;
         u8 command-specific-data[];
         u8 ack;
+        u8 command-specific-data-reply[];
 };
 
Quoted 4 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4025

Quoted 1 line
content.tex +0 -0
 /* ack values */

we now need to document which commands include the reply, otherwise reader will be confused.

OK.

Quoted 11 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4033

Quoted 9 lines
content.tex +2 -1
@@ -4023,7 +4028,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 \end{lstlisting}
 
 The \field{class}, \field{command} and command-specific-data are set by the
-driver, and the device sets the \field{ack} byte. There is little it can
+driver, and the device sets the \field{ack} byte and optionally
+\field{command-specific-data-reply}. There is little it can
 do

Now "it" refers to device where it used to refer to the driver.

Quoted 3 lines

There is little it can except issue a diagnostic if \field{ack} is not VIRTIO_NET_OK.

I don't feel like this has changed or affected anything.

Quoted 10 lines

Well we used to mention just driver. next sentence refers to driver. now you mention driver and device. what does "it" refer to"? it is now ambigous.

Quoted 1 line

Can you help me change it?

Replace "it" with "driver"?

OK. I see.

Quoted 58 lines
Quoted 46 lines
Quoted 30 lines
Quoted 30 lines

except issue a diagnostic if \field{ack} is not
VIRTIO_NET_OK.

content.tex +21 -0
@@ -4471,6 +4477,131 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 according to the native endian of the guest rather than
 (necessarily when not using the legacy interface) little-endian.
 
+\paragraph{Device stats}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device stats}
+
+If the VIRTIO_NET_F_CTRL_STATS feature is negotiated, the driver can
+get device stats from the device in \field{command-specific-data-reply}.
+
+To get the stats, the following definitions are used:
+\begin{lstlisting}
+#define VIRTIO_NET_CTRL_STATS                  6
+#define VIRTIO_NET_CTRL_STATS_GET_DEV          0
+#define VIRTIO_NET_CTRL_STATS_GET_CTRL_VQ      1
+#define VIRTIO_NET_CTRL_STATS_GET_QUEUE_PAIR   2
+\end{lstlisting}
+
+The following layout structures are used:
+
+\field{command-specific-data}
+\begin{lstlisting}
+struct virtio_net_ctrl_stats_queue_pair {
+    le64 queue_pair_index;
+}
+\end{lstlisting}
Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4505

Quoted 5 lines
content.tex +5 -0
+
+\field{command-specific-data-reply}
+\begin{lstlisting}
+struct virtio_net_ctrl_reply_stats_dev {
+    le64 dev_reset;         // The number of device resets.

while we already have some places using // comments, they are not used consistently. Better to use /* text */ for now.

OK.

Quoted 5 lines
Quoted 2 lines
content.tex +1 -0
+}
Quoted 8 lines
Quoted 4 lines

I am still worrying about this one. I think it's a bit vague. Some types of reset will zero this counter, others will not. Also, this looks more like a generic capability than anything specific to net. And this raises questions about .

I think, I should define that the device will not set this variable to 0 during reset.

again there's reset and there's reset. if you power cycle the system is device expected to retain this? the bus? what about pcie link level reset? etc

Quoted 5 lines
Quoted 1 line

So it's wrong to record statistics in reset, right?

depends on the kind of reset. for some kinds its outright impossible. it is certainly easier not to worry about this and just reset everything.

OK, I'll delete the dev_reset stat.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

The purpose of adding this is to count the number of times the user deletes/adds the network card. The virtio-net device may be deleted/added by the user multiple times,

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

I don't get it really ... deletes/adds card how?

For virtio-net, "ip link del eth1" is not supported. I don't know why.

because what it does is add a virtual link, and virtio does not support vlan offloading atm?

so I would maybe add vlan offloading instead ;)

You mean to add vlan-related statistics to the dev category, right?

Quoted 8 lines
Quoted 5 lines

I usually
use this command.

echo virtio1 > /sys/class/net/eth1/device/driver/unbind

and that unbinds the driver from the device

Yes, I do this often to simulate a drive reset.

Quoted 123 lines
Quoted 123 lines
Quoted 23 lines
Quoted 4 lines

and this is reflected in the device layer as the number of resets.

Well your text does not explain it exactly.

Quoted 14 lines
Quoted 12 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4509

Quoted 6 lines
content.tex +3 -0
+
+struct virtio_net_ctrl_reply_stats_cvq {
+    le64 request_num; // The number of requests.

We call them commands elsewhere. It is however unclear whether the request for the stats itself is included in the counter. Also, do these counters reset on device reset? I would assume so but given dev_reset does not reset it's hard to be sure.

Quoted 7 lines
Quoted 4 lines

I will change request -> command.

We should really define that the device should not reset these statistics during reset

I don't think I agree. new device - new stats.. physical cards seem to reset these things.

ok, let's confirm this first. After reset, all statistics will start from 0 again.

Quoted 45 lines
Quoted 45 lines
Quoted 8 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4511

Quoted 5 lines
content.tex +2 -0
+    le64 ok_num;      // The number of ok acks.
+    le64 err_num;     // The number of err acks.

A bit more precise pls: e.g. "the number of commands where ack was VIRTIO_NET_OK"?

OK

Quoted 16 lines
Quoted 16 lines
content.tex +14 -0
+
+    le64 req_rx_promisc;         // The number of requests with command VIRTIO_NET_CTRL_RX_PROMISC.
+    le64 req_rx_allmulti;        // The number of requests with command VIRTIO_NET_CTRL_RX_ALLMULTI.
+    le64 req_rx_alluni;          // The number of requests with command VIRTIO_NET_CTRL_RX_ALLUNI.
+    le64 req_rx_nomulti;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOMULTI.
+    le64 req_rx_nouni;           // The number of requests with command VIRTIO_NET_CTRL_RX_NOUNI.
+    le64 req_rx_nobcast;         // The number of requests with command VIRTIO_NET_CTRL_RX_NOBCAST.
+    le64 req_mac_table_set;      // The number of requests with command VIRTIO_NET_CTRL_MAC_TABLE_SET.
+    le64 req_mac_addr_set;       // The number of requests with command VIRTIO_NET_CTRL_MAC_ADDR_SET.
+    le64 req_vlan_add;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_ADD.
+    le64 req_vlan_del;           // The number of requests with command VIRTIO_NET_CTRL_VLAN_DEL.
+    le64 req_announce_ack;       // The number of requests with command VIRTIO_NET_CTRL_ANNOUNCE_ACK.
+    le64 req_mq_vq_pairs_set;    // The number of requests with command VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.
+    le64 req_mq_rss_config;      // The number of requests with command VIRTIO_NET_CTRL_MQ_RSS_CONFIG.
Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4528

Quoted 3 lines
content.tex +3 -0
+    le64 req_mq_hash_config;     // The number of requests with command VIRTIO_NET_CTRL_MQ_HASH_CONFIG.
+    le64 req_guest_offloads_set; // The number of requests with command VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET.
+}

I just had an idea:
how about the query passes in the command we are debugging instead? this way we do not need to extend this each time.

Do you mean command-specific-data uses a structure like this:

{
	le64 class;
	le64 command;
}
Quoted 5 lines
Quoted 1 line

This is indeed a really great idea.

can't say what you mean exactly here. just pass the command you want stats about to the card, is my idea.

I mean pass the desired command (class, command) to the device in command-specific-data.

Include only a num in the command-specific-data-reply as a response from the device.

struct {
	le64 num;
}
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4533

Quoted 10 lines
content.tex +5 -0
+
+struct virtio_net_ctrl_reply_stats_queue_pair {
+    /* rx stats */
+    le64 rx_packets;          // The number of packets recived by device, including the dropped packets by device.
+    le64 rx_bytes;            // The number of bytes recived by device, including the dropped packets by device.

typos: received

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4540

Quoted 7 lines
content.tex +7 -0
+
+    le64 rx_notification;     // The number of notifications from driver.
+    le64 rx_interrupt;        // The number of interrupts generated by device.
+
+    le64 rx_drop;             // The number of packets dropped by the rx queue. Contains all kinds of packet drop.
+    le64 rx_drop_overruns;    // The number of packets dropped by the rx queue when no more descriptors were available.
+    le64 rx_drop_oversize;    // The number of oversized packets received by the rx queue.

What does oversized mean in this context? does it apply to mergeable buffers?

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

For mergeable buffers this statistic is meaningless.

something to mention then. or should we bother at all?

sorry i didn't understand.

is this a common thing? why do we need this counter?

For small mode, this statistic is necessary.

Quoted 97 lines
Quoted 97 lines
Quoted 66 lines
Quoted 31 lines
Quoted 10 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4545

Quoted 8 lines
content.tex +5 -0
+
+    le64 rx_csum_valid;       // The number of packets with VIRTIO_NET_HDR_F_DATA_VALID.
+    le64 rx_csum_partial;     // The number of packets with VIRTIO_NET_HDR_F_NEEDS_CSUM.
+    le64 rx_csum_bad;         // The number of packets with abnormal csum.
+    le64 rx_csum_none;        // The number of packets without hardware csum.

I assume they are only incremented with VIRTIO_NET_F_GUEST_CSUM?

Yes

Quoted 6 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4547

Quoted 3 lines
content.tex +2 -0
+
+    le64 rx_gso_packets;      // The number of gso packets received by rx.

let's spell it out: I think this means any packet put into VQ and with gso_type != VIRTIO_NET_HDR_GSO_NONE

Yes

Quoted 7 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4548

Quoted 2 lines
content.tex +1 -0
+    le64 rx_gso_bytes;        // The number of gso bytes received by rx.

this one is a bit vague: I assume these are bytes written into the RX VQ buffers? the difference is that multiple packets are combined into a
single gso packet.
and does this include the virtio net header?

Quoted 6 lines

I think all *_bytes statistics do not include virtio net header, the packet bytes we care about does not include virtio net header.

Multiple packets are combined into one, and some network headers, such as ip/tcp, etc., are also removed. The final count is the bytes that are finalized to the RV VQ buffers.

sounds quite tricky to document. but go ahead and try if you like.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
content.tex +5 -0
+    le64 rx_reset;            // The number of queue resets.
+
+    /* tx stats */
+    le64 tx_packets;          // The number of packets sent by device, excluding the dropped packets by device.
+    le64 tx_bytes;            // The number of bytes sent by device, excluding the dropped packets by device.
Quoted 4 lines
Quoted 2 lines

same question: are we talking about bytes sent by device? or received from guest for transmission?

The number of bytes sent by device.

not symmetrical with rx then in case of gso.

Quoted 10 lines
Quoted 9 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4559

Quoted 7 lines
content.tex +6 -0
+
+    le64 tx_notification;     // The number of notifications from driver.
+    le64 tx_interrupt;        // The number of interrupts generated by device.
+
+    le64 tx_drop;             // The number of packets dropped by the tx queue. Contains all kinds of packet drop.
+    le64 tx_drop_desc_err;    // The number of packets dropped when the descriptor is in an error state.

what is a descriptor in error state?

Quoted 6 lines
Quoted 1 line

such as len == 0.

We never really documented what does it do.
maybe malformed packets would be better?
buffer too short?
something more specific.

OK.

Quoted 22 lines
Quoted 21 lines
Quoted 12 lines

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4565

Quoted 9 lines
content.tex +6 -0
+
+    le64 tx_csum_none;        // The number of packets that doesn't require hardware csum.
+    le64 tx_csum_partial;     // The number of packets that requires hardware csum.
+
+    le64 tx_gso_packets;      // The number of gso packets transmitted.
+    le64 tx_gso_bytes;        // The number of gso bytes transmitted.

point is, gso packets are not transmitted. they are received from guest. is this what this means?

Yes.

Quoted 2 lines

gso bytes probably somehow
means bytes in gso packets, but again this is a bit vague.

It should be defined, I think it should be defined as the number of bytes from the guest.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

above you said bytes sent.

I mean, all about GSO are statistics based on big packets. So here is the package from Guest, and rx is the package passed to Guest.

it's a subtle distinction that we need to make.

I will make this clear in the next version.

Thanks

#15 lore raw
Quoted 8 lines · Xuan Zhuo
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

above you said bytes sent.

I mean, all about GSO are statistics based on big packets. So here is the package from Guest, and rx is the package passed to Guest.

it's a subtle distinction that we need to make.

I will make this clear in the next version.

Given that there will be a next version, would you like us to withdraw the vote for the current version?

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

author
#16 lore raw
Quoted 7 lines · Cornelia Huck
Quoted 5 lines · Xuan Zhuo
Quoted 5 lines
Quoted 4 lines
Quoted 1 line

above you said bytes sent.

I mean, all about GSO are statistics based on big packets. So here is the package from Guest, and rx is the package passed to Guest.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

it's a subtle distinction that we need to make.

I will make this clear in the next version.

Given that there will be a next version, would you like us to withdraw the vote for the current version?

At present, it seems that the current preparation is still insufficient.

Please help to withdraw the previous vote. Let me re-vote when I am ready.

Thank you very much.

Quoted 1 line
#17 lore raw
Quoted 12 lines · Xuan Zhuo
Quoted 11 lines · Cornelia Huck
Quoted 8 lines · Xuan Zhuo
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

above you said bytes sent.

I mean, all about GSO are statistics based on big packets. So here is the package from Guest, and rx is the package passed to Guest.

it's a subtle distinction that we need to make.

I will make this clear in the next version.

Quoted 8 lines
Quoted 2 lines

Given that there will be a next version, would you like us to withdraw the vote for the current version?

At present, it seems that the current preparation is still insufficient.

Please help to withdraw the previous vote. Let me re-vote when I am ready.

Thank you very much.

Done.

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]