About the plan of Admin Queue
Hi,
hi, I would want to know some plans and progress about admin queue.
At the current spec, it seems that there is only one framework and no specific commands. I'd like to know if anyone is currently working on this and what the plans are.
We also faced some similar issues, and we think admin queue is a good way to manage sr-iov.
Thanks.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 8 lines · Xuan Zhuo
Hi,
hi, I would want to know some plans and progress about admin queue.
At the current spec, it seems that there is only one framework and no specific commands. I'd like to know if anyone is currently working on this and what the plans are.
Quoted 5 lines
We also faced some similar issues, and we think admin queue is a good way to manage sr-iov.
Thanks.
I plan to rebase original transport vq on admin vq.
https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
Thanks,
Zhu Lingshan
Quoted 11 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 11 lines · Lingshan
Quoted 8 lines · Xuan Zhuo
Hi,
hi, I would want to know some plans and progress about admin queue.
At the current spec, it seems that there is only one framework and no specific commands. I'd like to know if anyone is currently working on this and what the plans are.
Quoted 8 lines
Quoted 5 lines
We also faced some similar issues, and we think admin queue is a good way to manage sr-iov.
Thanks.
I plan to rebase original transport vq on admin vq.
https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
I review the patch, that is for S-IOV, right?
I think it is good.
I would if all is configured by the transport vq/admin vq from the OS?
Can we create a managed dev from the backend?
Such as, the DPU sends a command to the driver, then the driver creates a new managed dev.
Thanks.
Quoted 32 lines
Thanks,
Zhu LingshanQuoted 11 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 15 lines · Xuan Zhuo
Quoted 14 lines · Lingshan
Quoted 12 lines · Xuan Zhuo
Hi,
hi, I would want to know some plans and progress about admin queue.
At the current spec, it seems that there is only one framework and no specific commands. I'd like to know if anyone is currently working on this and what the plans are.
We also faced some similar issues, and we think admin queue is a good way to manage sr-iov.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Thanks.
I plan to rebase original transport vq on admin vq.
https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
I review the patch, that is for S-IOV, right?
Yes, for SIOV and other similar devices
Quoted 4 lines
I think it is good.
I would if all is configured by the transport vq/admin vq from the OS?
For SIOV ADIs, this transport vq is the transport layer, so they are configured by the OS through transport vq.
Quoted 5 lines
Can we create a managed dev from the backend?
Such as, the DPU sends a command to the driver, then the driver creates a new managed dev.
I think the group owner, usually the PCI PF is the management device.
Thanks
Quoted 49 lines
Thanks.
Quoted 29 lines
Thanks,
Zhu LingshanQuoted 10 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 29 lines · Lingshan
Quoted 15 lines · Xuan Zhuo
Quoted 14 lines · Lingshan
Quoted 12 lines · Xuan Zhuo
Hi,
hi, I would want to know some plans and progress about admin queue.
At the current spec, it seems that there is only one framework and no specific commands. I'd like to know if anyone is currently working on this and what the plans are.
We also faced some similar issues, and we think admin queue is a good way to manage sr-iov.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Thanks.
I plan to rebase original transport vq on admin vq.
https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
I review the patch, that is for S-IOV, right?
Yes, for SIOV and other similar devices
Quoted 4 lines
I think it is good.
I would if all is configured by the transport vq/admin vq from the OS?
Quoted 8 lines
For SIOV ADIs, this transport vq is the transport layer, so they are configured by the OS through transport vq.
Quoted 5 lines
Can we create a managed dev from the backend?
Such as, the DPU sends a command to the driver, then the driver creates a new managed dev.
I think the group owner, usually the PCI PF is the management device.
I mean the DPU hot-plug a new device. Not the managament device create a new device.
The managament device is in the OS, we want the device is plugged by the DPU.
Thanks.
Quoted 61 lines
Thanks
Quoted 58 lines
Thanks.
Quoted 29 lines
Thanks,
Zhu LingshanQuoted 10 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 30 lines · Xuan Zhuo
Quoted 29 lines · Lingshan
Quoted 14 lines · Xuan Zhuo
Quoted 13 lines · Lingshan
Quoted 12 lines · Xuan Zhuo
Hi,
hi, I would want to know some plans and progress about admin queue.
At the current spec, it seems that there is only one framework and no specific commands. I'd like to know if anyone is currently working on this and what the plans are.
We also faced some similar issues, and we think admin queue is a good way to manage sr-iov.
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
Thanks.
I plan to rebase original transport vq on admin vq.
https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
I review the patch, that is for S-IOV, right?
Yes, for SIOV and other similar devices
Quoted 3 lines
I think it is good.
I would if all is configured by the transport vq/admin vq from the OS?
For SIOV ADIs, this transport vq is the transport layer, so they are configured by the OS through transport vq.
Quoted 2 lines
Can we create a managed dev from the backend?
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
Such as, the DPU sends a command to the driver, then the driver creates a new managed dev.
I think the group owner, usually the PCI PF is the management device.
I mean the DPU hot-plug a new device. Not the managament device create a new device.
The managament device is in the OS, we want the device is plugged by the DPU.
The PCI management(SIOV group owner) device is on the DPU, when create an ADI, OS send a command to the DPU/PF through transport vq,
then the PF hot plugged in a new ADI through the transport vq specific channel. Or did I misunderstand your question?
Thanks
Quoted 60 lines
Thanks.
Quoted 56 lines
Thanks
Quoted 55 lines
Thanks.
Quoted 27 lines
Thanks,
Zhu LingshanQuoted 9 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 40 lines · Lingshan
Quoted 37 lines · Xuan Zhuo
Quoted 24 lines · Lingshan
Quoted 14 lines · Xuan Zhuo
Quoted 13 lines · Lingshan
Quoted 12 lines · Xuan Zhuo
Hi,
hi, I would want to know some plans and progress about admin queue.
At the current spec, it seems that there is only one framework and no specific commands. I'd like to know if anyone is currently working on this and what the plans are.
We also faced some similar issues, and we think admin queue is a good way to manage sr-iov.
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
Thanks.
I plan to rebase original transport vq on admin vq.
https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
I review the patch, that is for S-IOV, right?
Yes, for SIOV and other similar devices
Quoted 2 lines
I think it is good.
Quoted 8 lines
Quoted 1 line
I would if all is configured by the transport vq/admin vq from the OS?
For SIOV ADIs, this transport vq is the transport layer, so they are configured by the OS through transport vq.
Quoted 4 lines
Can we create a managed dev from the backend?
Such as, the DPU sends a command to the driver, then the driver creates a new managed dev.
I think the group owner, usually the PCI PF is the management device.
I mean the DPU hot-plug a new device. Not the managament device create a new device.
Quoted 6 lines
Quoted 1 line
The managament device is in the OS, we want the device is plugged by the DPU.
The PCI management(SIOV group owner) device is on the DPU, when create an ADI,
OS send a command to the DPU/PF through transport vq,
then the PF hot plugged in a new ADI through the transport vq specific channel. Or did I misunderstand your question?
Your first step is the os send a command. Right?
Can we let the DPU notify the driver to create a new devicer from the backend?
The key point is who want to create a new device.
Thanks.
Quoted 63 lines
Thanks
Quoted 60 lines
Thanks.
Quoted 56 lines
Thanks
Quoted 55 lines
Thanks.
Quoted 27 lines
Thanks,
Zhu LingshanQuoted 9 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 44 lines · Xuan Zhuo
Quoted 37 lines · Lingshan
Quoted 35 lines · Xuan Zhuo
Quoted 23 lines · Lingshan
Quoted 14 lines · Xuan Zhuo
Quoted 13 lines · Lingshan
Quoted 12 lines · Xuan Zhuo
Hi,
hi, I would want to know some plans and progress about admin queue.
At the current spec, it seems that there is only one framework and no specific commands. I'd like to know if anyone is currently working on this and what the plans are.
We also faced some similar issues, and we think admin queue is a good way to manage sr-iov.
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
Thanks.
I plan to rebase original transport vq on admin vq.
https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
I review the patch, that is for S-IOV, right?
Yes, for SIOV and other similar devices
Quoted 2 lines
I think it is good.
Quoted 8 lines
Quoted 1 line
I would if all is configured by the transport vq/admin vq from the OS?
For SIOV ADIs, this transport vq is the transport layer, so they are configured by the OS through transport vq.
Quoted 4 lines
Can we create a managed dev from the backend?
Such as, the DPU sends a command to the driver, then the driver creates a new managed dev.
I think the group owner, usually the PCI PF is the management device.
I mean the DPU hot-plug a new device. Not the managament device create a new device.
Quoted 6 lines
Quoted 1 line
The managament device is in the OS, we want the device is plugged by the DPU.
The PCI management(SIOV group owner) device is on the DPU, when create an ADI,
OS send a command to the DPU/PF through transport vq,
then the PF hot plugged in a new ADI through the transport vq specific channel. Or did I misunderstand your question?
Quoted 5 lines
Your first step is the os send a command. Right?
Can we let the DPU notify the driver to create a new devicer from the backend?
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?
Thanks
Quoted 65 lines
Thanks.
Quoted 59 lines
Thanks
Quoted 58 lines
Thanks.
Quoted 55 lines
Thanks
Quoted 54 lines
Thanks.
Quoted 26 lines
Thanks,
Zhu LingshanQuoted 8 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 56 lines · Lingshan
Quoted 45 lines · Xuan Zhuo
Quoted 38 lines · Lingshan
Quoted 36 lines · Xuan Zhuo
Quoted 24 lines · Lingshan
Quoted 14 lines · Xuan Zhuo
Quoted 13 lines · Lingshan
Quoted 12 lines · Xuan Zhuo
Hi,
hi, I would want to know some plans and progress about admin queue.
At the current spec, it seems that there is only one framework and no specific commands. I'd like to know if anyone is currently working on this and what the plans are.
We also faced some similar issues, and we think admin queue is a good way to manage sr-iov.
Quoted 6 lines
Quoted 5 lines
Quoted 1 line
Thanks.
I plan to rebase original transport vq on admin vq.
https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
I review the patch, that is for S-IOV, right?
Yes, for SIOV and other similar devices
Quoted 2 lines
I think it is good.
Quoted 8 lines
Quoted 1 line
I would if all is configured by the transport vq/admin vq from the OS?
For SIOV ADIs, this transport vq is the transport layer, so they are configured by the OS through transport vq.
Quoted 4 lines
Can we create a managed dev from the backend?
Such as, the DPU sends a command to the driver, then the driver creates a new managed dev.
I think the group owner, usually the PCI PF is the management device.
I mean the DPU hot-plug a new device. Not the managament device create a new device.
Quoted 6 lines
Quoted 1 line
The managament device is in the OS, we want the device is plugged by the DPU.
The PCI management(SIOV group owner) device is on the DPU, when create an ADI,
OS send a command to the DPU/PF through transport vq,
then the PF hot plugged in a new ADI through the transport vq specific channel. Or did I misunderstand your question?Quoted 5 lines
Your first step is the os send a command. Right?
Can we let the DPU notify the driver to create a new devicer from the backend?
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
Quoted 6 lines
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?Thanks
And how about this, I can try provide a new command can help query the onboard ADIs, report the IDs, then the driver can probe them by the IDs and read their config space to detect their config like num_vqs
Thanks
Quoted 56 lines
Quoted 56 lines
Thanks.
Quoted 50 lines
Thanks
Quoted 49 lines
Thanks.
Quoted 46 lines
Thanks
Quoted 45 lines
Thanks.
Quoted 26 lines
Thanks,
Zhu LingshanQuoted 8 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 52 lines · Lingshan
Quoted 44 lines · Xuan Zhuo
Quoted 37 lines · Lingshan
Quoted 35 lines · Xuan Zhuo
Quoted 23 lines · Lingshan
Quoted 14 lines · Xuan Zhuo
Quoted 13 lines · Lingshan
Quoted 12 lines · Xuan Zhuo
Hi,
hi, I would want to know some plans and progress about admin queue.
At the current spec, it seems that there is only one framework and no specific commands. I'd like to know if anyone is currently working on this and what the plans are.
We also faced some similar issues, and we think admin queue is a good way to manage sr-iov.
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
Thanks.
I plan to rebase original transport vq on admin vq.
https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
I review the patch, that is for S-IOV, right?
Yes, for SIOV and other similar devices
Quoted 2 lines
I think it is good.
Quoted 8 lines
Quoted 1 line
I would if all is configured by the transport vq/admin vq from the OS?
For SIOV ADIs, this transport vq is the transport layer, so they are configured by the OS through transport vq.
Quoted 4 lines
Can we create a managed dev from the backend?
Such as, the DPU sends a command to the driver, then the driver creates a new managed dev.
I think the group owner, usually the PCI PF is the management device.
I mean the DPU hot-plug a new device. Not the managament device create a new device.
Quoted 6 lines
Quoted 1 line
The managament device is in the OS, we want the device is plugged by the DPU.
The PCI management(SIOV group owner) device is on the DPU, when create an ADI,
OS send a command to the DPU/PF through transport vq,
then the PF hot plugged in a new ADI through the transport vq specific channel. Or did I misunderstand your question?Quoted 5 lines
Your first step is the os send a command. Right?
Can we let the DPU notify the driver to create a new devicer from the backend?
The key point is who want to create a new device.
Quoted 8 lines
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?
In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
Thanks.
Quoted 68 lines
Thanks
Quoted 65 lines
Thanks.
Quoted 59 lines
Thanks
Quoted 58 lines
Thanks.
Quoted 55 lines
Thanks
Quoted 54 lines
Thanks.
Quoted 26 lines
Thanks,
Zhu LingshanQuoted 8 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 63 lines · Xuan Zhuo
Quoted 50 lines · Lingshan
Quoted 43 lines · Xuan Zhuo
Quoted 36 lines · Lingshan
Quoted 35 lines · Xuan Zhuo
Quoted 23 lines · Lingshan
Quoted 14 lines · Xuan Zhuo
Quoted 13 lines · Lingshan
Quoted 12 lines · Xuan Zhuo
Hi,
hi, I would want to know some plans and progress about admin queue.
At the current spec, it seems that there is only one framework and no specific commands. I'd like to know if anyone is currently working on this and what the plans are.
We also faced some similar issues, and we think admin queue is a good way to manage sr-iov.
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
Thanks.
I plan to rebase original transport vq on admin vq.
https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
I review the patch, that is for S-IOV, right?
Yes, for SIOV and other similar devices
Quoted 2 lines
I think it is good.
Quoted 8 lines
Quoted 1 line
I would if all is configured by the transport vq/admin vq from the OS?
For SIOV ADIs, this transport vq is the transport layer, so they are configured by the OS through transport vq.
Quoted 4 lines
Can we create a managed dev from the backend?
Such as, the DPU sends a command to the driver, then the driver creates a new managed dev.
I think the group owner, usually the PCI PF is the management device.
I mean the DPU hot-plug a new device. Not the managament device create a new device.
Quoted 6 lines
Quoted 1 line
The managament device is in the OS, we want the device is plugged by the DPU.
The PCI management(SIOV group owner) device is on the DPU, when create an ADI,
OS send a command to the DPU/PF through transport vq,
then the PF hot plugged in a new ADI through the transport vq specific channel. Or did I misunderstand your question?Quoted 5 lines
Your first step is the os send a command. Right?
Can we let the DPU notify the driver to create a new devicer from the backend?
The key point is who want to create a new device.
Quoted 8 lines
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
Quoted 7 lines
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC to the host. However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
To address an ADI, there is only a device_id.
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
Quoted 68 lines
Thanks.
Quoted 64 lines
Thanks
Quoted 63 lines
Thanks.
Quoted 58 lines
Thanks
Quoted 57 lines
Thanks.
Quoted 54 lines
Thanks
Quoted 53 lines
Thanks.
Quoted 25 lines
Thanks,
Zhu LingshanQuoted 7 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 9 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Quoted 1 line
Quoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 14 lines
Quoted 14 lines
Quoted 5 lines
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.
That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.
SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
Quoted 59 lines · Parav Pandit
Quoted 7 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Quoted 1 line
Quoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
Quoted 8 lines
For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device, and the device return ADIs id and status. Looks good? and work for you @Xuan?
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 67 lines · Lingshan
Quoted 64 lines · Parav Pandit
Quoted 7 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Quoted 1 line
Quoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
Quoted 7 lines
Quoted 3 lines
For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?
Can the device notifies the driver by a interrupt?
Thanks.
Quoted 3 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 66 lines · Xuan Zhuo
Quoted 65 lines · Lingshan
Quoted 63 lines · Parav Pandit
Quoted 6 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Quoted 1 line
Quoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Can the device notifies the driver by a interrupt?
Of course this can be an option as previously discussed.
That would be a config interrupt(with proper suppression) from the management device, which requires a new cap in a bar, then you still need to fetch the information.
A proposal:
a new cap, at least contains:
total number of the ADIs.
Then once any ADIs are created or removed, the PF driver get a notification.
Works for you?
Quoted 13 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 83 lines · Lingshan
Quoted 66 lines · Xuan Zhuo
Quoted 65 lines · Lingshan
Quoted 63 lines · Parav Pandit
Quoted 6 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Quoted 1 line
Quoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Can the device notifies the driver by a interrupt?
Of course this can be an option as previously discussed. That would be a config interrupt(with proper suppression) from the management device,
which requires a new cap in a bar, then you still need to fetch the information.
Quoted 8 lines
A proposal:
a new cap, at least contains:
total number of the ADIs.Then once any ADIs are created or removed, the PF driver get a notification.
Works for you?
Look good to me.
Thanks.
Quoted 13 lines
Quoted 13 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 10 lines
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 84 lines · Xuan Zhuo
Quoted 83 lines · Lingshan
Quoted 65 lines · Xuan Zhuo
Quoted 64 lines · Lingshan
Quoted 63 lines · Parav Pandit
Quoted 6 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Quoted 1 line
Quoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Can the device notifies the driver by a interrupt?
Of course this can be an option as previously discussed. That would be a config interrupt(with proper suppression) from the management device,
which requires a new cap in a bar, then you still need to fetch the information.A proposal:
Quoted 8 lines
Quoted 6 lines
a new cap, at least contains:
total number of the ADIs.Then once any ADIs are created or removed, the PF driver get a notification.
Works for you?
Look good to me.
OK, you can review the details when I post the series
Thanks
Quoted 25 lines
Thanks.
Quoted 21 lines
Quoted 21 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 67 lines · Lingshan
Quoted 64 lines · Parav Pandit
Quoted 7 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Quoted 1 line
Quoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
Quoted 7 lines
Quoted 3 lines
For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?
Could I have your plan for this?
If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc.
Thanks.
Quoted 3 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 69 lines · Xuan Zhuo
Quoted 68 lines · Lingshan
Quoted 66 lines · Parav Pandit
Quoted 6 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Quoted 1 line
Quoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
Quoted 8 lines
Quoted 4 lines
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Could I have your plan for this?
If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc.
I think the query commands for SIOV is a little more complex, e.g., need to report device type and its scale(e.g., features, mq). There can be thousands of SIOV ADIs and we don't want output flood.
We have discussed implementation a config interrupt to report new created / deleted
ADIs on the DPU side, therefore there must be a cap contains related information,
my rough approach of the process is:
1) a cap contains the total number of existing ADIs and the max dev id
2) driver queries detailed information of a certain ADI or a bunch of ADIs in a [dev_id....dev_id2] range.
I am not sure whether a NIC stores its IP
Thanks
Quoted 14 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 84 lines · Lingshan
Quoted 69 lines · Xuan Zhuo
Quoted 68 lines · Lingshan
Quoted 66 lines · Parav Pandit
Quoted 6 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Quoted 1 line
Quoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
Quoted 8 lines
Quoted 4 lines
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Could I have your plan for this?
If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc.
I think the query commands for SIOV is a little more complex, e.g., need to report device type and its
scale(e.g., features, mq). There can be thousands of SIOV ADIs and we don't want output flood.
Quoted 8 lines
We have discussed implementation a config interrupt to report new created / deleted
ADIs on the DPU side, therefore there must be a cap contains related information,
my rough approach of the process is:
1) a cap contains the total number of existing ADIs and the max dev id
2) driver queries detailed information of a certain ADI or a bunch of ADIs in a [dev_id....dev_id2] range.
Yes, Admin Queue can obtain the info of the specific one or more devices.
Quoted 2 lines
I am not sure whether a NIC stores its IP
IP is the other topic. I want the Admin Queue manage the switch. So the switch know about the IP of every device, and the
Admin Queue will has the ability to config the IP of the device inside the switch.
Thanks.
Quoted 26 lines
Thanks
Quoted 23 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 91 lines · Xuan Zhuo
Quoted 82 lines · Lingshan
Quoted 68 lines · Xuan Zhuo
Quoted 67 lines · Lingshan
Quoted 66 lines · Parav Pandit
Quoted 6 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Quoted 1 line
Quoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
Quoted 8 lines
Quoted 4 lines
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Could I have your plan for this?
If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc.
I think the query commands for SIOV is a little more complex, e.g., need to report device type and its
scale(e.g., features, mq). There can be thousands of SIOV ADIs and we don't want output flood.Quoted 8 lines
We have discussed implementation a config interrupt to report new created / deleted
ADIs on the DPU side, therefore there must be a cap contains related information,
my rough approach of the process is:
1) a cap contains the total number of existing ADIs and the max dev id
2) driver queries detailed information of a certain ADI or a bunch of ADIs in a [dev_id....dev_id2] range.
Quoted 8 lines
Yes, Admin Queue can obtain the info of the specific one or more devices.
Quoted 1 line
I am not sure whether a NIC stores its IP
IP is the other topic. I want the Admin Queue manage the switch. So the switch know about the IP of every device, and the
Admin Queue will has the ability to config the IP of the device inside the switch.
DPU onboard switch? OVS? Does it beyond virtio spec?
Quoted 28 lines
Thanks.
Quoted 23 lines
Thanks
Quoted 22 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 96 lines · Lingshan
Quoted 93 lines · Xuan Zhuo
Quoted 82 lines · Lingshan
Quoted 68 lines · Xuan Zhuo
Quoted 67 lines · Lingshan
Quoted 66 lines · Parav Pandit
Quoted 6 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Quoted 1 line
Quoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
Quoted 8 lines
Quoted 4 lines
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Could I have your plan for this?
If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc.
I think the query commands for SIOV is a little more complex, e.g., need to report device type and its
scale(e.g., features, mq). There can be thousands of SIOV ADIs and we don't want output flood.Quoted 8 lines
We have discussed implementation a config interrupt to report new created / deleted
ADIs on the DPU side, therefore there must be a cap contains related information,
my rough approach of the process is:
1) a cap contains the total number of existing ADIs and the max dev id
2) driver queries detailed information of a certain ADI or a bunch of ADIs in a [dev_id....dev_id2] range.Yes, Admin Queue can obtain the info of the specific one or more devices.
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
I am not sure whether a NIC stores its IP
IP is the other topic. I want the Admin Queue manage the switch. So the switch know about the IP of every device, and the
Admin Queue will has the ability to config the IP of the device inside the switch.DPU onboard switch? OVS? Does it beyond virtio spec?
YES.
For SIOV, I think this is MUST. Maybe you have one simple implementation. But you have to solve the IP steering. So admin queue should has the ability to config the IP steering.
Thanks.
Quoted 37 lines
Quoted 28 lines
Thanks.
Quoted 23 lines
Thanks
Quoted 22 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 100 lines · Xuan Zhuo
Quoted 98 lines · Lingshan
Quoted 95 lines · Xuan Zhuo
Quoted 82 lines · Lingshan
Quoted 68 lines · Xuan Zhuo
Quoted 67 lines · Lingshan
Quoted 66 lines · Parav Pandit
Quoted 6 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Quoted 1 line
Quoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
Quoted 8 lines
Quoted 4 lines
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Could I have your plan for this?
If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc.
I think the query commands for SIOV is a little more complex, e.g., need to report device type and its
scale(e.g., features, mq). There can be thousands of SIOV ADIs and we don't want output flood.Quoted 8 lines
We have discussed implementation a config interrupt to report new created / deleted
ADIs on the DPU side, therefore there must be a cap contains related information,
my rough approach of the process is:
1) a cap contains the total number of existing ADIs and the max dev id
2) driver queries detailed information of a certain ADI or a bunch of ADIs in a [dev_id....dev_id2] range.Yes, Admin Queue can obtain the info of the specific one or more devices.
Quoted 1 line
I am not sure whether a NIC stores its IP
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
IP is the other topic. I want the Admin Queue manage the switch. So the switch know about the IP of every device, and the
Admin Queue will has the ability to config the IP of the device inside the switch.DPU onboard switch? OVS? Does it beyond virtio spec?
YES.
Adding Washizu.
We can have a switch/dpa defined in the networking device for sure.
Quoted 2 lines
For SIOV, I think this is MUST.
A learning bridge would be fine as a starter. It's better not to couple new scalable capability with any device specific features.
Quoted 3 lines
Maybe you have one simple implementation.
But you have to solve the IP steering. So admin queue should has the ability to config the IP steering.
I think not. Those L2/LN tables/filters are networking specific. Control virtqueue is better than admin virtqueue here.
Thanks
Quoted 57 lines
Thanks.
Quoted 37 lines
Quoted 28 lines
Thanks.
Quoted 23 lines
Thanks
Quoted 22 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 105 lines · Jason Wang
Quoted 104 lines · Xuan Zhuo
Quoted 102 lines · Lingshan
Quoted 99 lines · Xuan Zhuo
Quoted 82 lines · Lingshan
Quoted 68 lines · Xuan Zhuo
Quoted 67 lines · Lingshan
Quoted 66 lines · Parav Pandit
Quoted 6 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Quoted 1 line
Quoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
Quoted 8 lines
Quoted 4 lines
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Could I have your plan for this?
If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc.
I think the query commands for SIOV is a little more complex, e.g., need to report device type and its
scale(e.g., features, mq). There can be thousands of SIOV ADIs and we don't want output flood.Quoted 8 lines
We have discussed implementation a config interrupt to report new created / deleted
ADIs on the DPU side, therefore there must be a cap contains related information,
my rough approach of the process is:
1) a cap contains the total number of existing ADIs and the max dev id
2) driver queries detailed information of a certain ADI or a bunch of ADIs in a [dev_id....dev_id2] range.Yes, Admin Queue can obtain the info of the specific one or more devices.
Quoted 1 line
I am not sure whether a NIC stores its IP
IP is the other topic. I want the Admin Queue manage the switch. So the switch know about the IP of every device, and the
Admin Queue will has the ability to config the IP of the device inside the switch.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
DPU onboard switch? OVS? Does it beyond virtio spec?
YES.
Adding Washizu.
We can have a switch/dpa defined in the networking device for sure.
Yes, I think we should introduce that for the sr-iov. Or for other.
I would like to know who is doing this?
Another question, @Jason are you referring to a new device type or a new virtio-net feature.
Quoted 4 lines
Quoted 3 lines
For SIOV, I think this is MUST.
Quoted 8 lines
A learning bridge would be fine as a starter. It's better not to couple new scalable capability with any device specific features.
Quoted 3 lines
Maybe you have one simple implementation.
But you have to solve the IP steering. So admin queue should has the ability to config the IP steering.I think not. Those L2/LN tables/filters are networking specific.
Let us assume that there is a switch/bridge firstly. The VFs may be passed to different VMs.
I also think this is the networking specific. But I want to config the ip for every vf from the pf. Because the user of the vf may be unreliable. We need a manager to config the ip for every vf.
Quoted 1 line
Control virtqueue is better than admin virtqueue here.
by cq?
What case?
Thanks.
Quoted 70 lines
Thanks
Quoted 66 lines
Thanks.
Quoted 37 lines
Quoted 28 lines
Thanks.
Quoted 23 lines
Thanks
Quoted 22 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 109 lines · Xuan Zhuo
Quoted 107 lines · Jason Wang
Quoted 106 lines · Xuan Zhuo
Quoted 103 lines · Lingshan
Quoted 91 lines · Xuan Zhuo
Quoted 82 lines · Lingshan
Quoted 68 lines · Xuan Zhuo
Quoted 67 lines · Lingshan
Quoted 66 lines · Parav Pandit
Quoted 6 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Quoted 1 line
Quoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
Quoted 8 lines
Quoted 4 lines
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Could I have your plan for this?
If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc.
I think the query commands for SIOV is a little more complex, e.g., need to report device type and its
scale(e.g., features, mq). There can be thousands of SIOV ADIs and we don't want output flood.Quoted 8 lines
We have discussed implementation a config interrupt to report new created / deleted
ADIs on the DPU side, therefore there must be a cap contains related information,
my rough approach of the process is:
1) a cap contains the total number of existing ADIs and the max dev id
2) driver queries detailed information of a certain ADI or a bunch of ADIs in a [dev_id....dev_id2] range.Quoted 8 lines
Yes, Admin Queue can obtain the info of the specific one or more devices.
Quoted 1 line
I am not sure whether a NIC stores its IP
IP is the other topic. I want the Admin Queue manage the switch. So the switch know about the IP of every device, and the
Admin Queue will has the ability to config the IP of the device inside the switch.DPU onboard switch? OVS? Does it beyond virtio spec?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
YES.
Adding Washizu.
We can have a switch/dpa defined in the networking device for sure.
Yes, I think we should introduce that for the sr-iov. Or for other.
This should be a general one as a switch should be transport independent.
Quoted 2 lines
I would like to know who is doing this?
Washizu, could you confirm if you want to do this or not?
Quoted 3 lines
Another question, @Jason are you referring to a new device type or a new virtio-net feature.
Extending virtio-net should be fine, did you see any issues for this?
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
For SIOV, I think this is MUST.
A learning bridge would be fine as a starter. It's better not to couple new scalable capability with any device specific features.
Quoted 3 lines
Maybe you have one simple implementation.
But you have to solve the IP steering. So admin queue should has the ability to config the IP steering.
Quoted 7 lines
Quoted 1 line
I think not. Those L2/LN tables/filters are networking specific.
Let us assume that there is a switch/bridge firstly. The VFs may be passed to different VMs.
I also think this is the networking specific. But I want to config the ip for every vf from the pf.
What do you mean by ip here (e.g who is the user for this ip?)
Quoted 2 lines
Because the user of the vf may be unreliable.
We need a manager to config the ip for every vf.
Did you mean you're using a tunnel or not?
Quoted 7 lines
Quoted 3 lines
Control virtqueue is better than admin virtqueue here.
by cq?
What case?
We've already used control virtqueue for steering.
Thanks
Quoted 74 lines
Thanks.
Quoted 70 lines
Thanks
Quoted 66 lines
Thanks.
Quoted 37 lines
Quoted 28 lines
Thanks.
Quoted 23 lines
Thanks
Quoted 22 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 112 lines · Jason Wang
Quoted 111 lines · Xuan Zhuo
Quoted 109 lines · Jason Wang
Quoted 107 lines · Xuan Zhuo
Quoted 96 lines · Lingshan
Quoted 93 lines · Xuan Zhuo
Quoted 82 lines · Lingshan
Quoted 68 lines · Xuan Zhuo
Quoted 67 lines · Lingshan
Quoted 66 lines · Parav Pandit
Quoted 6 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Quoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make
sure the orchestration SW is aware of their device IDs.Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
If you want the DPU randomly create ADIs and notify the driver, I
think we need interrupt, e.g., re-use config interrupt. But why DPU
wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal
machines?In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
Quoted 8 lines
Quoted 4 lines
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Could I have your plan for this?
If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc.
I think the query commands for SIOV is a little more complex, e.g., need to report device type and its
scale(e.g., features, mq). There can be thousands of SIOV ADIs and we don't want output flood.Quoted 8 lines
We have discussed implementation a config interrupt to report new created / deleted
ADIs on the DPU side, therefore there must be a cap contains related information,
my rough approach of the process is:
1) a cap contains the total number of existing ADIs and the max dev id
2) driver queries detailed information of a certain ADI or a bunch of ADIs in a [dev_id....dev_id2] range.Yes, Admin Queue can obtain the info of the specific one or more devices.
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
I am not sure whether a NIC stores its IP
IP is the other topic. I want the Admin Queue manage the switch. So the switch know about the IP of every device, and the
Admin Queue will has the ability to config the IP of the device inside the switch.DPU onboard switch? OVS? Does it beyond virtio spec?
YES.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Adding Washizu.
We can have a switch/dpa defined in the networking device for sure.
Yes, I think we should introduce that for the sr-iov. Or for other.
This should be a general one as a switch should be transport independent.
I agree.
Quoted 11 lines
Quoted 3 lines
I would like to know who is doing this?
Washizu, could you confirm if you want to do this or not?
Quoted 3 lines
Another question, @Jason are you referring to a new device type or a new virtio-net feature.
Extending virtio-net should be fine, did you see any issues for this?
You kwnow the packet will be steered to different virtio-net devices. So I think this is odd if the feature is the extending of virtio-net.
How do you think about this?
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 5 lines
For SIOV, I think this is MUST.
A learning bridge would be fine as a starter. It's better not to couple new scalable capability with any device specific features.
Quoted 3 lines
Maybe you have one simple implementation.
But you have to solve the IP steering. So admin queue should has the ability to config the IP steering.I think not. Those L2/LN tables/filters are networking specific.
Quoted 7 lines
Quoted 5 lines
Let us assume that there is a switch/bridge firstly. The VFs may be passed to different VMs.
I also think this is the networking specific. But I want to config the ip for every vf from the pf.
What do you mean by ip here (e.g who is the user for this ip?)
1. the admin queue config the ip for the vf to the switch
2. the vf get the ip by dhcp from the switch. (or other way)
Here we config an ip for the vf, and we also limit that
just this vf can use the ip for security. Other vf can not use the ip configured by self.
Quoted 5 lines
Quoted 3 lines
Because the user of the vf may be unreliable.
We need a manager to config the ip for every vf.Did you mean you're using a tunnel or not?
No.
Multiple VMs on a host may be used by different users. Some users are not trustworthy. We want to prevent a user from maliciously using ip
Quoted 11 lines
Quoted 8 lines
Quoted 4 lines
Control virtqueue is better than admin virtqueue here.
by cq?
What case?
We've already used control virtqueue for steering.
More details?
You config the steering rule to other virtio-net device by the cq of one virtio-net?
Thanks.
Quoted 77 lines
Thanks
Quoted 74 lines
Thanks.
Quoted 70 lines
Thanks
Quoted 66 lines
Thanks.
Quoted 37 lines
Quoted 28 lines
Thanks.
Quoted 23 lines
Thanks
Quoted 22 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 135 lines · Xuan Zhuo
Quoted 112 lines · Jason Wang
Quoted 111 lines · Xuan Zhuo
Quoted 109 lines · Jason Wang
Quoted 107 lines · Xuan Zhuo
Quoted 96 lines · Lingshan
Quoted 93 lines · Xuan Zhuo
Quoted 82 lines · Lingshan
Quoted 68 lines · Xuan Zhuo
Quoted 67 lines · Lingshan
Quoted 66 lines · Parav Pandit
Quoted 6 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make
sure the orchestration SW is aware of their device IDs.Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
If you want the DPU randomly create ADIs and notify the driver, I
think we need interrupt, e.g., re-use config interrupt. But why DPU
wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal
machines?
In your scenario, the supervisor is on the os, which controls the DPU
to create new devices.In the cloud scenario, the vendor manager is in the DPU, and the
entire host is for users. Of course, there are situations where the
vendor manager are in the HOST. But for bare metal machines, the host
belongs to the customer, the vendor manager is only in the DPU.Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager
will plug a device to the host from the DPU.I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
Quoted 8 lines
Quoted 4 lines
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Could I have your plan for this?
If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc.
I think the query commands for SIOV is a little more complex, e.g., need to report device type and its
scale(e.g., features, mq). There can be thousands of SIOV ADIs and we don't want output flood.Quoted 8 lines
We have discussed implementation a config interrupt to report new created / deleted
ADIs on the DPU side, therefore there must be a cap contains related information,
my rough approach of the process is:
1) a cap contains the total number of existing ADIs and the max dev id
2) driver queries detailed information of a certain ADI or a bunch of ADIs in a [dev_id....dev_id2] range.Yes, Admin Queue can obtain the info of the specific one or more devices.
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
I am not sure whether a NIC stores its IP
IP is the other topic. I want the Admin Queue manage the switch. So the switch know about the IP of every device, and the
Admin Queue will has the ability to config the IP of the device inside the switch.DPU onboard switch? OVS? Does it beyond virtio spec?
YES.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Adding Washizu.
We can have a switch/dpa defined in the networking device for sure.
Yes, I think we should introduce that for the sr-iov. Or for other.
This should be a general one as a switch should be transport independent.
I agree.
Quoted 10 lines
Quoted 3 lines
I would like to know who is doing this?
Washizu, could you confirm if you want to do this or not?
Quoted 3 lines
Another question, @Jason are you referring to a new device type or a new virtio-net feature.
Quoted 6 lines
Quoted 1 line
Extending virtio-net should be fine, did you see any issues for this?
You kwnow the packet will be steered to different virtio-net devices. So I think this is odd if the feature is the extending of virtio-net.
How do you think about this?
I think this is the common model of modern NICs? For example the PF with switch is usually an ethernet device itself.
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 8 lines
Quoted 7 lines
For SIOV, I think this is MUST.
Quoted 8 lines
A learning bridge would be fine as a starter. It's better not to couple new scalable capability with any device specific features.
Quoted 3 lines
Maybe you have one simple implementation.
But you have to solve the IP steering. So admin queue should has the ability to config the IP steering.I think not. Those L2/LN tables/filters are networking specific.
Let us assume that there is a switch/bridge firstly. The VFs may be passed to different VMs.
I also think this is the networking specific. But I want to config the ip for every vf from the pf.
Quoted 8 lines
Quoted 1 line
What do you mean by ip here (e.g who is the user for this ip?)
1. the admin queue config the ip for the vf to the switch
2. the vf get the ip by dhcp from the switch. (or other way)Here we config an ip for the vf, and we also limit that
just this vf can use the ip for security. Other vf can not use the ip configured by self.
It looks like a feature which is orthogonal to switch.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Because the user of the vf may be unreliable.
We need a manager to config the ip for every vf.
Quoted 6 lines
Quoted 1 line
Did you mean you're using a tunnel or not?
No.
Multiple VMs on a host may be used by different users. Some users are not trustworthy. We want to prevent a user from maliciously using ip
Ok, so it's an independent feature which could be done separately.
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
Control virtqueue is better than admin virtqueue here.
by cq?
What case?
Quoted 8 lines
Quoted 2 lines
We've already used control virtqueue for steering.
More details?
You config the steering rule to other virtio-net device by the cq of one virtio-net?
Exactly, we had already had the owner and group definition. So it's not that hard? Actually, there should be layers in the middle for example, the switch should see only ports and there should be some compositions of ports to a device I guess.
Thanks
Quoted 83 lines
Thanks.
Quoted 77 lines
Thanks
Quoted 74 lines
Thanks.
Quoted 70 lines
Thanks
Quoted 66 lines
Thanks.
Quoted 37 lines
Quoted 28 lines
Thanks.
Quoted 23 lines
Thanks
Quoted 22 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 138 lines · Jason Wang
Quoted 137 lines · Xuan Zhuo
Quoted 112 lines · Jason Wang
Quoted 111 lines · Xuan Zhuo
Quoted 109 lines · Jason Wang
Quoted 107 lines · Xuan Zhuo
Quoted 96 lines · Lingshan
Quoted 93 lines · Xuan Zhuo
Quoted 82 lines · Lingshan
Quoted 68 lines · Xuan Zhuo
Quoted 67 lines · Lingshan
Quoted 66 lines · Parav Pandit
Quoted 6 lines
From: [email protected] <[email protected]
open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMCan we let the DPU notify the driver to create a new devicer from the
backend?Yes, why not.
Quoted 4 lines
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make
sure the orchestration SW is aware of their device IDs.Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 21 lines
If you want the DPU randomly create ADIs and notify the driver, I
think we need interrupt, e.g., re-use config interrupt. But why DPU
wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal
machines?
In your scenario, the supervisor is on the os, which controls the DPU
to create new devices.In the cloud scenario, the vendor manager is in the DPU, and the
entire host is for users. Of course, there are situations where the
vendor manager are in the HOST. But for bare metal machines, the host
belongs to the customer, the vendor manager is only in the DPU.So when the customers buy a new nic for the host, the vendor manager
will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC
to the host.
However you only owns the DPU and the customer owns the host, that means
this creation and hot plug must be transparent to the host and there may not be
a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to
the host by spanning the device across the PCI hierarchy.Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a device_id.
SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of
device your DPU provide to the host? Lets see whether DPU can mediate this by
its own?It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
Quoted 8 lines
Quoted 4 lines
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Could I have your plan for this?
If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc.
I think the query commands for SIOV is a little more complex, e.g., need to report device type and its
scale(e.g., features, mq). There can be thousands of SIOV ADIs and we don't want output flood.Quoted 8 lines
We have discussed implementation a config interrupt to report new created / deleted
ADIs on the DPU side, therefore there must be a cap contains related information,
my rough approach of the process is:
1) a cap contains the total number of existing ADIs and the max dev id
2) driver queries detailed information of a certain ADI or a bunch of ADIs in a [dev_id....dev_id2] range.Yes, Admin Queue can obtain the info of the specific one or more devices.
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
I am not sure whether a NIC stores its IP
IP is the other topic. I want the Admin Queue manage the switch. So the switch know about the IP of every device, and the
Admin Queue will has the ability to config the IP of the device inside the switch.DPU onboard switch? OVS? Does it beyond virtio spec?
YES.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Adding Washizu.
We can have a switch/dpa defined in the networking device for sure.
Yes, I think we should introduce that for the sr-iov. Or for other.
This should be a general one as a switch should be transport independent.
I agree.
Quoted 11 lines
Quoted 3 lines
I would like to know who is doing this?
Washizu, could you confirm if you want to do this or not?
Quoted 3 lines
Another question, @Jason are you referring to a new device type or a new virtio-net feature.
Extending virtio-net should be fine, did you see any issues for this?
Quoted 7 lines
Quoted 4 lines
You kwnow the packet will be steered to different virtio-net devices. So I think this is odd if the feature is the extending of virtio-net.
How do you think about this?
I think this is the common model of modern NICs? For example the PF with switch is usually an ethernet device itself.
Great!
Quoted 41 lines
Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
Quoted 9 lines
Quoted 8 lines
For SIOV, I think this is MUST.
Quoted 8 lines
A learning bridge would be fine as a starter. It's better not to couple new scalable capability with any device specific features.
Quoted 3 lines
Maybe you have one simple implementation.
But you have to solve the IP steering. So admin queue should has the ability to config the IP steering.I think not. Those L2/LN tables/filters are networking specific.
Let us assume that there is a switch/bridge firstly. The VFs may be passed to different VMs.
I also think this is the networking specific. But I want to config the ip for every vf from the pf.
Quoted 8 lines
Quoted 1 line
What do you mean by ip here (e.g who is the user for this ip?)
1. the admin queue config the ip for the vf to the switch
2. the vf get the ip by dhcp from the switch. (or other way)Here we config an ip for the vf, and we also limit that
just this vf can use the ip for security. Other vf can not use the ip configured by self.It looks like a feature which is orthogonal to switch.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Because the user of the vf may be unreliable.
We need a manager to config the ip for every vf.
Quoted 20 lines
Quoted 6 lines
Quoted 1 line
Did you mean you're using a tunnel or not?
No.
Multiple VMs on a host may be used by different users. Some users are not trustworthy. We want to prevent a user from maliciously using ip
Ok, so it's an independent feature which could be done separately.
Quoted 11 lines
Quoted 11 lines
Quoted 9 lines
Quoted 5 lines
Control virtqueue is better than admin virtqueue here.
by cq?
What case?
We've already used control virtqueue for steering.
Now, the spec includes this?
Do you mean that you config the steering by the control virtqueue of pf?
If so, I think is ok.
Thanks.
Quoted 8 lines
Quoted 7 lines
Quoted 1 line
More details?
You config the steering rule to other virtio-net device by the cq of one virtio-net?
Quoted 91 lines
Exactly, we had already had the owner and group definition. So it's not that hard? Actually, there should be layers in the middle for example, the switch should see only ports and there should be some compositions of ports to a device I guess.
Thanks
Quoted 83 lines
Thanks.
Quoted 77 lines
Thanks
Quoted 74 lines
Thanks.
Quoted 70 lines
Thanks
Quoted 66 lines
Thanks.
Quoted 37 lines
Quoted 28 lines
Thanks.
Quoted 23 lines
Thanks
Quoted 22 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 124 lines · Yui Washizu
Quoted 105 lines · Jason Wang
Quoted 104 lines · Xuan Zhuo
Quoted 103 lines · Jason Wang
Quoted 102 lines · Xuan Zhuo
Quoted 92 lines · Lingshan
Quoted 90 lines · Xuan Zhuo
Quoted 89 lines · Lingshan
Quoted 68 lines · Xuan Zhuo
Quoted 67 lines · Lingshan
Quoted 66 lines · Parav Pandit
Quoted 6 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make
sure the orchestration SW is aware of their device IDs.Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
If you want the DPU randomly create ADIs and notify the driver, I
think we need interrupt, e.g., re-use config interrupt. But why DPU
wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal
machines?
In your scenario, the supervisor is on the os, which controls the DPU
to create new devices.In the cloud scenario, the vendor manager is in the DPU, and the
entire host is for users. Of course, there are situations where the
vendor manager are in the HOST. But for bare metal machines, the host
belongs to the customer, the vendor manager is only in the DPU.Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager
will plug a device to the host from the DPU.I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
Quoted 8 lines
Quoted 4 lines
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Could I have your plan for this?
If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc.
I think the query commands for SIOV is a little more complex, e.g., need to report device type and its
scale(e.g., features, mq). There can be thousands of SIOV ADIs and we don't want output flood.We have discussed implementation a config interrupt to report new created / deleted
ADIs on the DPU side, therefore there must be a cap contains related information,
my rough approach of the process is:
1) a cap contains the total number of existing ADIs and the max dev id
2) driver queries detailed information of a certain ADI or a bunch of ADIs in a [dev_id....dev_id2] range.Quoted 8 lines
Quoted 7 lines
Yes, Admin Queue can obtain the info of the specific one or more devices.
Quoted 1 line
I am not sure whether a NIC stores its IP
IP is the other topic. I want the Admin Queue manage the switch. So the switch know about the IP of every device, and the
Admin Queue will has the ability to config the IP of the device inside the switch.DPU onboard switch? OVS? Does it beyond virtio spec?
YES.
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
Adding Washizu.
We can have a switch/dpa defined in the networking device for sure.
Yes, I think we should introduce that for the sr-iov. Or for other.
This should be a general one as a switch should be transport independent.
Quoted 1 line
I would like to know who is doing this?
Washizu, could you confirm if you want to do this or not?
Does this mean adding a switch definition to the virtio spec?
If so, it will be necessary for the implementation of my plan,
but it may take time (probably several months?) to get started,
Quoted 8 lines
as I'm currently working on another task (virtio-net SR-IOV feature in qemu).
Anyone is welcome to work on adding the switch definition in the meantime,
it's completely fine with me.
I think I'll work on that if no one has finished the work.
OK, I got.
Because we have a need in this area, I will push the work in this area.
Thanks.
Quoted 93 lines
Quoted 93 lines
Quoted 6 lines
Another question, @Jason are you referring to a new device type or a new virtio-net feature.
Extending virtio-net should be fine, did you see any issues for this?
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
For SIOV, I think this is MUST.
A learning bridge would be fine as a starter. It's better not to couple new scalable capability with any device specific features.
Quoted 3 lines
Maybe you have one simple implementation.
But you have to solve the IP steering. So admin queue should has the ability to config the IP steering.Quoted 6 lines
Quoted 1 line
I think not. Those L2/LN tables/filters are networking specific.
Let us assume that there is a switch/bridge firstly. The VFs may be passed to different VMs.
I also think this is the networking specific. But I want to config the ip for every vf from the pf.
What do you mean by ip here (e.g who is the user for this ip?)
Quoted 2 lines
Because the user of the vf may be unreliable.
We need a manager to config the ip for every vf.Did you mean you're using a tunnel or not?
Quoted 5 lines
Quoted 2 lines
Control virtqueue is better than admin virtqueue here.
by cq?
What case?
We've already used control virtqueue for steering.
Thanks
Quoted 57 lines
Thanks.
Quoted 55 lines
Thanks
Quoted 53 lines
Thanks.
Quoted 35 lines
Quoted 27 lines
Thanks.
Quoted 23 lines
Thanks
Quoted 22 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 125 lines · Yui Washizu
Quoted 105 lines · Jason Wang
Quoted 104 lines · Xuan Zhuo
Quoted 103 lines · Jason Wang
Quoted 102 lines · Xuan Zhuo
Quoted 92 lines · Lingshan
Quoted 90 lines · Xuan Zhuo
Quoted 89 lines · Lingshan
Quoted 68 lines · Xuan Zhuo
Quoted 67 lines · Lingshan
Quoted 66 lines · Parav Pandit
Quoted 6 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make
sure the orchestration SW is aware of their device IDs.Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
If you want the DPU randomly create ADIs and notify the driver, I
think we need interrupt, e.g., re-use config interrupt. But why DPU
wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal
machines?
In your scenario, the supervisor is on the os, which controls the DPU
to create new devices.In the cloud scenario, the vendor manager is in the DPU, and the
entire host is for users. Of course, there are situations where the
vendor manager are in the HOST. But for bare metal machines, the host
belongs to the customer, the vendor manager is only in the DPU.Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager
will plug a device to the host from the DPU.I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
Quoted 8 lines
Quoted 4 lines
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Could I have your plan for this?
If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc.
I think the query commands for SIOV is a little more complex, e.g., need to report device type and its
scale(e.g., features, mq). There can be thousands of SIOV ADIs and we don't want output flood.We have discussed implementation a config interrupt to report new created / deleted
ADIs on the DPU side, therefore there must be a cap contains related information,
my rough approach of the process is:
1) a cap contains the total number of existing ADIs and the max dev id
2) driver queries detailed information of a certain ADI or a bunch of ADIs in a [dev_id....dev_id2] range.Quoted 8 lines
Quoted 7 lines
Yes, Admin Queue can obtain the info of the specific one or more devices.
Quoted 1 line
I am not sure whether a NIC stores its IP
IP is the other topic. I want the Admin Queue manage the switch. So the switch know about the IP of every device, and the
Admin Queue will has the ability to config the IP of the device inside the switch.DPU onboard switch? OVS? Does it beyond virtio spec?
YES.
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
Adding Washizu.
We can have a switch/dpa defined in the networking device for sure.
Yes, I think we should introduce that for the sr-iov. Or for other.
This should be a general one as a switch should be transport independent.
Quoted 1 line
I would like to know who is doing this?
Washizu, could you confirm if you want to do this or not?
Does this mean adding a switch definition to the virtio spec?
If so, it will be necessary for the implementation of my plan,
but it may take time (probably several months?) to get started,
Quoted 8 lines
as I'm currently working on another task (virtio-net SR-IOV feature in qemu).
Anyone is welcome to work on adding the switch definition in the meantime,
it's completely fine with me.
I think I'll work on that if no one has finished the work.
Ok, great, I think we need to start this by considering reusing one of the existing DPA spec. (For example, ofdpa? or any other?)
Thanks
Quoted 93 lines
Quoted 93 lines
Quoted 6 lines
Another question, @Jason are you referring to a new device type or a new virtio-net feature.
Extending virtio-net should be fine, did you see any issues for this?
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
For SIOV, I think this is MUST.
A learning bridge would be fine as a starter. It's better not to couple new scalable capability with any device specific features.
Quoted 3 lines
Maybe you have one simple implementation.
But you have to solve the IP steering. So admin queue should has the ability to config the IP steering.Quoted 6 lines
Quoted 1 line
I think not. Those L2/LN tables/filters are networking specific.
Let us assume that there is a switch/bridge firstly. The VFs may be passed to different VMs.
I also think this is the networking specific. But I want to config the ip for every vf from the pf.
What do you mean by ip here (e.g who is the user for this ip?)
Quoted 2 lines
Because the user of the vf may be unreliable.
We need a manager to config the ip for every vf.Did you mean you're using a tunnel or not?
Quoted 5 lines
Quoted 2 lines
Control virtqueue is better than admin virtqueue here.
by cq?
What case?
We've already used control virtqueue for steering.
Thanks
Quoted 57 lines
Thanks.
Quoted 55 lines
Thanks
Quoted 53 lines
Thanks.
Quoted 35 lines
Quoted 27 lines
Thanks.
Quoted 23 lines
Thanks
Quoted 22 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 129 lines · Jason Wang
Quoted 128 lines · Yui Washizu
Quoted 105 lines · Jason Wang
Quoted 104 lines · Xuan Zhuo
Quoted 103 lines · Jason Wang
Quoted 102 lines · Xuan Zhuo
Quoted 92 lines · Lingshan
Quoted 90 lines · Xuan Zhuo
Quoted 89 lines · Lingshan
Quoted 68 lines · Xuan Zhuo
Quoted 67 lines · Lingshan
Quoted 66 lines · Parav Pandit
Quoted 6 lines
From: [email protected] <[email protected]
open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMCan we let the DPU notify the driver to create a new devicer from the
backend?Yes, why not.
Quoted 4 lines
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make
sure the orchestration SW is aware of their device IDs.Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 21 lines
If you want the DPU randomly create ADIs and notify the driver, I
think we need interrupt, e.g., re-use config interrupt. But why DPU
wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal
machines?
In your scenario, the supervisor is on the os, which controls the DPU
to create new devices.In the cloud scenario, the vendor manager is in the DPU, and the
entire host is for users. Of course, there are situations where the
vendor manager are in the HOST. But for bare metal machines, the host
belongs to the customer, the vendor manager is only in the DPU.So when the customers buy a new nic for the host, the vendor manager
will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC
to the host.
However you only owns the DPU and the customer owns the host, that means
this creation and hot plug must be transparent to the host and there may not be
a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to
the host by spanning the device across the PCI hierarchy.Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a device_id.
SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of
device your DPU provide to the host? Lets see whether DPU can mediate this by
its own?It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
Quoted 8 lines
Quoted 4 lines
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Could I have your plan for this?
If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc.
I think the query commands for SIOV is a little more complex, e.g., need to report device type and its
scale(e.g., features, mq). There can be thousands of SIOV ADIs and we don't want output flood.We have discussed implementation a config interrupt to report new created / deleted
ADIs on the DPU side, therefore there must be a cap contains related information,
my rough approach of the process is:
1) a cap contains the total number of existing ADIs and the max dev id
2) driver queries detailed information of a certain ADI or a bunch of ADIs in a [dev_id....dev_id2] range.Quoted 8 lines
Quoted 7 lines
Yes, Admin Queue can obtain the info of the specific one or more devices.
Quoted 1 line
I am not sure whether a NIC stores its IP
IP is the other topic. I want the Admin Queue manage the switch. So the switch know about the IP of every device, and the
Admin Queue will has the ability to config the IP of the device inside the switch.DPU onboard switch? OVS? Does it beyond virtio spec?
YES.
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
Adding Washizu.
We can have a switch/dpa defined in the networking device for sure.
Yes, I think we should introduce that for the sr-iov. Or for other.
This should be a general one as a switch should be transport independent.
Quoted 1 line
I would like to know who is doing this?
Washizu, could you confirm if you want to do this or not?
Does this mean adding a switch definition to the virtio spec?
If so, it will be necessary for the implementation of my plan,
but it may take time (probably several months?) to get started,
as I'm currently working on another task (virtio-net SR-IOV feature in qemu).
Quoted 8 lines
Quoted 5 lines
Anyone is welcome to work on adding the switch definition in the meantime,
it's completely fine with me.
I think I'll work on that if no one has finished the work.
Ok, great, I think we need to start this by considering reusing one of the existing DPA spec. (For example, ofdpa? or any other?)
We have a need on this area. I want to start this work now.
Could you give me the link of the spec?
Thanks
Quoted 115 lines
Thanks
Quoted 93 lines
Quoted 93 lines
Quoted 6 lines
Another question, @Jason are you referring to a new device type or a new virtio-net feature.
Extending virtio-net should be fine, did you see any issues for this?
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
For SIOV, I think this is MUST.
A learning bridge would be fine as a starter. It's better not to couple new scalable capability with any device specific features.
Quoted 3 lines
Maybe you have one simple implementation.
But you have to solve the IP steering. So admin queue should has the ability to config the IP steering.Quoted 6 lines
Quoted 1 line
I think not. Those L2/LN tables/filters are networking specific.
Let us assume that there is a switch/bridge firstly. The VFs may be passed to different VMs.
I also think this is the networking specific. But I want to config the ip for every vf from the pf.
What do you mean by ip here (e.g who is the user for this ip?)
Quoted 2 lines
Because the user of the vf may be unreliable.
We need a manager to config the ip for every vf.Did you mean you're using a tunnel or not?
Quoted 5 lines
Quoted 2 lines
Control virtqueue is better than admin virtqueue here.
by cq?
What case?
We've already used control virtqueue for steering.
Thanks
Quoted 57 lines
Thanks.
Quoted 55 lines
Thanks
Quoted 53 lines
Thanks.
Quoted 35 lines
Quoted 27 lines
Thanks.
Quoted 23 lines
Thanks
Quoted 22 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 135 lines · Xuan Zhuo
Quoted 133 lines · Jason Wang
Quoted 132 lines · Yui Washizu
Quoted 105 lines · Jason Wang
Quoted 104 lines · Xuan Zhuo
Quoted 103 lines · Jason Wang
Quoted 102 lines · Xuan Zhuo
Quoted 92 lines · Lingshan
Quoted 90 lines · Xuan Zhuo
Quoted 89 lines · Lingshan
Quoted 68 lines · Xuan Zhuo
Quoted 67 lines · Lingshan
Quoted 66 lines · Parav Pandit
From: [email protected] <[email protected]
open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMCan we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make
sure the orchestration SW is aware of their device IDs.Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
Xuan partly explained it.If you want the DPU randomly create ADIs and notify the driver, I
think we need interrupt, e.g., re-use config interrupt. But why DPU
wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal
machines?
In your scenario, the supervisor is on the os, which controls the DPU
to create new devices.In the cloud scenario, the vendor manager is in the DPU, and the
entire host is for users. Of course, there are situations where the
vendor manager are in the HOST. But for bare metal machines, the host
belongs to the customer, the vendor manager is only in the DPU.So when the customers buy a new nic for the host, the vendor manager
will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC
to the host.
However you only owns the DPU and the customer owns the host, that means
this creation and hot plug must be transparent to the host and there may not be
a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
However this is not PCI which has a tree/switch and can enumerate devices to
the host by spanning the device across the PCI hierarchy.Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
To address an ADI, there is only a device_id.
SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.So, do you mind share how your DPU offload the device model? What kind of
device your DPU provide to the host? Lets see whether DPU can mediate this by
its own?It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
[1] https://github.com/Mellanox/scalablefunctions/wiki
Quoted 8 lines
Quoted 4 lines
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Could I have your plan for this?
If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc.
I think the query commands for SIOV is a little more complex, e.g., need to report device type and its
scale(e.g., features, mq). There can be thousands of SIOV ADIs and we don't want output flood.We have discussed implementation a config interrupt to report new created / deleted
ADIs on the DPU side, therefore there must be a cap contains related information,
my rough approach of the process is:
1) a cap contains the total number of existing ADIs and the max dev id
2) driver queries detailed information of a certain ADI or a bunch of ADIs in a [dev_id....dev_id2] range.Quoted 8 lines
Quoted 7 lines
Yes, Admin Queue can obtain the info of the specific one or more devices.
Quoted 1 line
I am not sure whether a NIC stores its IP
IP is the other topic. I want the Admin Queue manage the switch. So the switch know about the IP of every device, and the
Admin Queue will has the ability to config the IP of the device inside the switch.DPU onboard switch? OVS? Does it beyond virtio spec?
YES.
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
Adding Washizu.
We can have a switch/dpa defined in the networking device for sure.
Yes, I think we should introduce that for the sr-iov. Or for other.
This should be a general one as a switch should be transport independent.
Quoted 1 line
I would like to know who is doing this?
Washizu, could you confirm if you want to do this or not?
Does this mean adding a switch definition to the virtio spec?
If so, it will be necessary for the implementation of my plan,
but it may take time (probably several months?) to get started,
as I'm currently working on another task (virtio-net SR-IOV feature in qemu).
Anyone is welcome to work on adding the switch definition in the meantime,
it's completely fine with me.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
I think I'll work on that if no one has finished the work.
Ok, great, I think we need to start this by considering reusing one of the existing DPA spec. (For example, ofdpa? or any other?)
We have a need on this area. I want to start this work now.
Could you give me the link of the spec?
I'm not sure this is the best one, we can hear from others. I mention it since it has an emulation code that is done in Qemu (rocker switch).
The spec is:
https://docs.broadcom.com/doc/12378911
(not sure this is the recent one though).
Thanks
Quoted 137 lines
Thanks
Quoted 115 lines
Thanks
Quoted 93 lines
Quoted 93 lines
Quoted 6 lines
Another question, @Jason are you referring to a new device type or a new virtio-net feature.
Extending virtio-net should be fine, did you see any issues for this?
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
For SIOV, I think this is MUST.
A learning bridge would be fine as a starter. It's better not to couple new scalable capability with any device specific features.
Quoted 3 lines
Maybe you have one simple implementation.
But you have to solve the IP steering. So admin queue should has the ability to config the IP steering.Quoted 6 lines
Quoted 1 line
I think not. Those L2/LN tables/filters are networking specific.
Let us assume that there is a switch/bridge firstly. The VFs may be passed to different VMs.
I also think this is the networking specific. But I want to config the ip for every vf from the pf.
What do you mean by ip here (e.g who is the user for this ip?)
Quoted 2 lines
Because the user of the vf may be unreliable.
We need a manager to config the ip for every vf.Did you mean you're using a tunnel or not?
Quoted 5 lines
Quoted 2 lines
Control virtqueue is better than admin virtqueue here.
by cq?
What case?
We've already used control virtqueue for steering.
Thanks
Quoted 57 lines
Thanks.
Quoted 55 lines
Thanks
Quoted 53 lines
Thanks.
Quoted 35 lines
Quoted 27 lines
Thanks.
Quoted 23 lines
Thanks
Quoted 22 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 145 lines · Jason Wang
Quoted 135 lines · Xuan Zhuo
Quoted 133 lines · Jason Wang
Quoted 132 lines · Yui Washizu
Quoted 105 lines · Jason Wang
Quoted 104 lines · Xuan Zhuo
Quoted 103 lines · Jason Wang
Quoted 102 lines · Xuan Zhuo
Quoted 92 lines · Lingshan
Quoted 90 lines · Xuan Zhuo
Quoted 89 lines · Lingshan
Quoted 68 lines · Xuan Zhuo
Quoted 67 lines · Lingshan
On 6/30/2023 7:35 PM, Parav Pandit wrote:
From: [email protected] <[email protected]
open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMCan we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make
sure the orchestration SW is aware of their device IDs.Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
Xuan partly explained it.If you want the DPU randomly create ADIs and notify the driver, I
think we need interrupt, e.g., re-use config interrupt. But why DPU
wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal
machines?
In your scenario, the supervisor is on the os, which controls the DPU
to create new devices.In the cloud scenario, the vendor manager is in the DPU, and the
entire host is for users. Of course, there are situations where the
vendor manager are in the HOST. But for bare metal machines, the host
belongs to the customer, the vendor manager is only in the DPU.So when the customers buy a new nic for the host, the vendor manager
will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC
to the host.
However you only owns the DPU and the customer owns the host, that means
this creation and hot plug must be transparent to the host and there may not be
a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
However this is not PCI which has a tree/switch and can enumerate devices to
the host by spanning the device across the PCI hierarchy.Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
To address an ADI, there is only a device_id.
SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.So, do you mind share how your DPU offload the device model? What kind of
device your DPU provide to the host? Lets see whether DPU can mediate this by
its own?It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
[1] https://github.com/Mellanox/scalablefunctions/wiki
Quoted 8 lines
Quoted 4 lines
well I see SF facing the similar challenge, I can add a command for the
driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you
@Xuan?Could I have your plan for this?
If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc.
I think the query commands for SIOV is a little more complex, e.g., need to report device type and its
scale(e.g., features, mq). There can be thousands of SIOV ADIs and we don't want output flood.We have discussed implementation a config interrupt to report new created / deleted
ADIs on the DPU side, therefore there must be a cap contains related information,
my rough approach of the process is:
1) a cap contains the total number of existing ADIs and the max dev id
2) driver queries detailed information of a certain ADI or a bunch of ADIs in a [dev_id....dev_id2] range.Quoted 8 lines
Quoted 7 lines
Yes, Admin Queue can obtain the info of the specific one or more devices.
Quoted 1 line
I am not sure whether a NIC stores its IP
IP is the other topic. I want the Admin Queue manage the switch. So the switch know about the IP of every device, and the
Admin Queue will has the ability to config the IP of the device inside the switch.DPU onboard switch? OVS? Does it beyond virtio spec?
YES.
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
Adding Washizu.
We can have a switch/dpa defined in the networking device for sure.
Yes, I think we should introduce that for the sr-iov. Or for other.
This should be a general one as a switch should be transport independent.
Quoted 1 line
I would like to know who is doing this?
Washizu, could you confirm if you want to do this or not?
Does this mean adding a switch definition to the virtio spec?
If so, it will be necessary for the implementation of my plan,
but it may take time (probably several months?) to get started,
as I'm currently working on another task (virtio-net SR-IOV feature in qemu).
Anyone is welcome to work on adding the switch definition in the meantime,
it's completely fine with me.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
I think I'll work on that if no one has finished the work.
Ok, great, I think we need to start this by considering reusing one of the existing DPA spec. (For example, ofdpa? or any other?)
We have a need on this area. I want to start this work now.
Could you give me the link of the spec?
Quoted 7 lines
I'm not sure this is the best one, we can hear from others. I mention it since it has an emulation code that is done in Qemu (rocker switch).
The spec is:
OK.
I will study it. But I think we can start with a simple model. For virtio-net, we don't need a full-featured switch. I would mainly consider the SR-IOV case.
Thanks.
Quoted 152 lines
(not sure this is the recent one though).
Thanks
Quoted 146 lines
Thanks
Quoted 115 lines
Thanks
Quoted 93 lines
Quoted 93 lines
Quoted 6 lines
Another question, @Jason are you referring to a new device type or a new virtio-net feature.
Extending virtio-net should be fine, did you see any issues for this?
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
For SIOV, I think this is MUST.
A learning bridge would be fine as a starter. It's better not to couple new scalable capability with any device specific features.
Quoted 3 lines
Maybe you have one simple implementation.
But you have to solve the IP steering. So admin queue should has the ability to config the IP steering.Quoted 6 lines
Quoted 1 line
I think not. Those L2/LN tables/filters are networking specific.
Let us assume that there is a switch/bridge firstly. The VFs may be passed to different VMs.
I also think this is the networking specific. But I want to config the ip for every vf from the pf.
What do you mean by ip here (e.g who is the user for this ip?)
Quoted 2 lines
Because the user of the vf may be unreliable.
We need a manager to config the ip for every vf.Did you mean you're using a tunnel or not?
Quoted 5 lines
Quoted 2 lines
Control virtqueue is better than admin virtqueue here.
by cq?
What case?
We've already used control virtqueue for steering.
Thanks
Quoted 57 lines
Thanks.
Quoted 55 lines
Thanks
Quoted 53 lines
Thanks.
Quoted 35 lines
Quoted 27 lines
Thanks.
Quoted 23 lines
Thanks
Quoted 22 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 119 lines · Jason Wang
Quoted 100 lines · Xuan Zhuo
Quoted 98 lines · Lingshan
Quoted 95 lines · Xuan Zhuo
Quoted 82 lines · Lingshan
Quoted 68 lines · Xuan Zhuo
Quoted 67 lines · Lingshan
Quoted 66 lines · Parav Pandit
Quoted 6 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Quoted 1 line
Quoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
Quoted 8 lines
Quoted 4 lines
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Could I have your plan for this?
If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc.
I think the query commands for SIOV is a little more complex, e.g., need to report device type and its
scale(e.g., features, mq). There can be thousands of SIOV ADIs and we don't want output flood.Quoted 8 lines
We have discussed implementation a config interrupt to report new created / deleted
ADIs on the DPU side, therefore there must be a cap contains related information,
my rough approach of the process is:
1) a cap contains the total number of existing ADIs and the max dev id
2) driver queries detailed information of a certain ADI or a bunch of ADIs in a [dev_id....dev_id2] range.Yes, Admin Queue can obtain the info of the specific one or more devices.
Quoted 1 line
I am not sure whether a NIC stores its IP
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
IP is the other topic. I want the Admin Queue manage the switch. So the switch know about the IP of every device, and the
Admin Queue will has the ability to config the IP of the device inside the switch.DPU onboard switch? OVS? Does it beyond virtio spec?
YES.
Adding Washizu.
We can have a switch/dpa defined in the networking device for sure.
Quoted 2 lines
For SIOV, I think this is MUST.
A learning bridge would be fine as a starter. It's better not to couple new scalable capability with any device specific features.
Quoted 6 lines
Quoted 3 lines
Maybe you have one simple implementation.
But you have to solve the IP steering. So admin queue should has the ability to config the IP steering.I think not. Those L2/LN tables/filters are networking specific. Control virtqueue is better than admin virtqueue here.
So I think some things like ring size, queue num, interrupts etc. are configured by the admin queue. VF's ip, mac should be configured by PF's cvq.
Is that right?
@Michael @Jason
Thanks.
Quoted 70 lines
Thanks
Quoted 66 lines
Thanks.
Quoted 37 lines
Quoted 28 lines
Thanks.
Quoted 23 lines
Thanks
Quoted 22 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 128 lines · Xuan Zhuo
Quoted 119 lines · Jason Wang
Quoted 100 lines · Xuan Zhuo
Quoted 98 lines · Lingshan
Quoted 95 lines · Xuan Zhuo
Quoted 82 lines · Lingshan
Quoted 68 lines · Xuan Zhuo
Quoted 67 lines · Lingshan
Quoted 66 lines · Parav Pandit
Quoted 6 lines
From: [email protected] <[email protected] open.org> On Behalf Of Zhu, Lingshan
Sent: Friday, June 30, 2023 6:33 AMQuoted 1 line
Quoted 1 line
Quoted 1 line
Can we let the DPU notify the driver to create a new devicer from the
backend?
Yes, why not.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs.
Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal machines?In your scenario, the supervisor is on the os, which controls the DPU to create new devices.
In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU.
Quoted 8 lines
Quoted 2 lines
So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the NIC to the host.
However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe.That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
Quoted 3 lines
However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy.
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
Quoted 2 lines
To address an ADI, there is only a
device_id.SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.Quoted 4 lines
So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own?
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
Quoted 8 lines
Quoted 4 lines
well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you @Xuan?Could I have your plan for this?
If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc.
I think the query commands for SIOV is a little more complex, e.g., need to report device type and its
scale(e.g., features, mq). There can be thousands of SIOV ADIs and we don't want output flood.Quoted 8 lines
We have discussed implementation a config interrupt to report new created / deleted
ADIs on the DPU side, therefore there must be a cap contains related information,
my rough approach of the process is:
1) a cap contains the total number of existing ADIs and the max dev id
2) driver queries detailed information of a certain ADI or a bunch of ADIs in a [dev_id....dev_id2] range.Yes, Admin Queue can obtain the info of the specific one or more devices.
Quoted 1 line
I am not sure whether a NIC stores its IP
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
IP is the other topic. I want the Admin Queue manage the switch. So the switch know about the IP of every device, and the
Admin Queue will has the ability to config the IP of the device inside the switch.DPU onboard switch? OVS? Does it beyond virtio spec?
YES.
Adding Washizu.
We can have a switch/dpa defined in the networking device for sure.
Quoted 2 lines
For SIOV, I think this is MUST.
A learning bridge would be fine as a starter. It's better not to couple new scalable capability with any device specific features.
Quoted 6 lines
Quoted 3 lines
Maybe you have one simple implementation.
But you have to solve the IP steering. So admin queue should has the ability to config the IP steering.I think not. Those L2/LN tables/filters are networking specific. Control virtqueue is better than admin virtqueue here.
Quoted 6 lines
So I think some things like ring size, queue num, interrupts etc. are configured by the admin queue. VF's ip, mac should be configured by PF's cvq.
Is that right?
@Michael @Jason
That's my understanding.
Thanks
Quoted 75 lines
Thanks.
Quoted 70 lines
Thanks
Quoted 66 lines
Thanks.
Quoted 37 lines
Quoted 28 lines
Thanks.
Quoted 23 lines
Thanks
Quoted 22 lines
Thanks.
Quoted 2 lines
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 5 lines
From: Jason Wang <[email protected]>
Sent: Monday, July 31, 2023 6:50 AMQuoted 2 lines
So I think some things like ring size, queue num, interrupts etc. are configured by the admin queue. VF's ip, mac should be configured by PF's cvq.
VF's mac should be configured by itself using its own cvq. There is no mediation required.
Quoted 17 lines · Xuan Zhuo
Quoted 15 lines · Lingshan
Quoted 12 lines · Xuan Zhuo
Hi,
hi, I would want to know some plans and progress about admin queue.
At the current spec, it seems that there is only one framework and no specific commands. I'd like to know if anyone is currently working on this and what the plans are.
We also faced some similar issues, and we think admin queue is a good way to manage sr-iov.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Thanks.
I plan to rebase original transport vq on admin vq.
https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
I review the patch, that is for S-IOV, right?
It is not limited to SIOV. So you won't see any SIOV terminology in the proposal.
Quoted 4 lines
I think it is good.
I would if all is configured by the transport vq/admin vq from the OS?
One thing to clarify is that, we will define a set of admin commands, and admin virtqueue is one interface for that. The admin commands is designed to be more general to be carried via other type of interfaces (if not, let's fix):
1) MMIO
2) Other transport that already use DMA
3) software IPC
Thanks
Quoted 58 lines
Can we create a managed dev from the backend?
Such as, the DPU sends a command to the driver, then the driver creates a new managed dev.
Thanks.
Quoted 32 lines
Thanks,
Zhu LingshanQuoted 11 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 21 lines · Jason Wang
Quoted 20 lines · Xuan Zhuo
Quoted 18 lines · Lingshan
Quoted 8 lines · Xuan Zhuo
Hi,
hi, I would want to know some plans and progress about admin queue.
At the current spec, it seems that there is only one framework and no specific commands. I'd like to know if anyone is currently working on this and what the plans are.
Quoted 5 lines
We also faced some similar issues, and we think admin queue is a good way to manage sr-iov.
Thanks.
I plan to rebase original transport vq on admin vq.
Quoted 7 lines
Quoted 4 lines
I review the patch, that is for S-IOV, right?
It is not limited to SIOV. So you won't see any SIOV terminology in the proposal.
I got.
Quoted 6 lines
Quoted 5 lines
I think it is good.
I would if all is configured by the transport vq/admin vq from the OS?
Quoted 8 lines
One thing to clarify is that, we will define a set of admin commands, and admin virtqueue is one interface for that. The admin commands is designed to be more general to be carried via other type of interfaces (if not, let's fix):
1) MMIO
2) Other transport that already use DMA
3) software IPC
Great!!
Thanks.
Quoted 71 lines
Thanks
Quoted 67 lines
Can we create a managed dev from the backend?
Such as, the DPU sends a command to the driver, then the driver creates a new managed dev.
Thanks.
Quoted 32 lines
Thanks,
Zhu LingshanQuoted 11 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
OK it looks like at least Parav and Jason understand what's going on, maybe one of you can explain:
1- there's a management stack that can bind an ID
to a group of properties
2- the proposed command then binds an ID to a VF
3- after this, VF can be assigned to a VM
Is this a fair summary?
In that case what is gained compared to configuring properties on the VF directly?
Why is not the ID an internal property of the management?
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
Quoted 8 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Monday, July 3, 2023 9:05 AMOK it looks like at least Parav and Jason understand what's going on, maybe one of you can explain:
Yet to catch up on the thread today, will be able to respond in few hours.
Quoted 5 lines
1- there's a management stack that can bind an ID
to a group of properties
2- the proposed command then binds an ID to a VF 3- after this, VF can be assigned to a VM
Quoted 8 lines
Is this a fair summary?
In that case what is gained compared to configuring properties on the VF directly?
Why is not the ID an internal property of the management?--
MST
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 13 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Monday, July 3, 2023 9:05 AMOK it looks like at least Parav and Jason understand what's going on, maybe one of you can explain:
1- there's a management stack that can bind an ID
to a group of properties
2- the proposed command then binds an ID to a VF 3- after this, VF can be assigned to a VM
Quoted 5 lines
Is this a fair summary?
In that case what is gained compared to configuring properties on the VF directly?
Why is not the ID an internal property of the management?
For VFs I also believe we can live with an internal property.
For migratable SIOV devices which are spawned by the management system has similar ID assigned by the mgmt. system. Having such node local ID is needed anyway.
There is a struggle (scale wise) to make such ID network wide unique if each virtio device to expose such id at transport level.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 17 lines · Parav Pandit
Quoted 12 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Monday, July 3, 2023 9:05 AMOK it looks like at least Parav and Jason understand what's going on, maybe one of you can explain:
1- there's a management stack that can bind an ID
to a group of properties
2- the proposed command then binds an ID to a VF 3- after this, VF can be assigned to a VMQuoted 5 lines
Is this a fair summary?
In that case what is gained compared to configuring properties on the VF directly?
Why is not the ID an internal property of the management?
Quoted 6 lines
For VFs I also believe we can live with an internal property.
For migratable SIOV devices which are spawned by the management system has similar ID assigned by the mgmt. system. Having such node local ID is needed anyway.
There is a struggle (scale wise) to make such ID network wide unique if each virtio device to expose such id at transport level.
Maybe you don't want a guest talking to the PF through admin queue directly, so there can be a wrapper for SIOV ADIs, therefore the ID is not a problem for migration.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 9 lines · Michael S. Tsirkin
OK it looks like at least Parav and Jason understand what's going on, maybe one of you can explain:
1- there's a management stack that can bind an ID
to a group of properties
2- the proposed command then binds an ID to a VF 3- after this, VF can be assigned to a VMIs this a fair summary?
This sounds like what I need.
This thread discusses another point, the device notifies the pf driver that there is a new device.
Thanks.
Quoted 8 lines
In that case what is gained compared to configuring properties on the VF directly?
Why is not the ID an internal property of the management?--
MST
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 8 lines · Michael S. Tsirkin
OK it looks like at least Parav and Jason understand what's going on, maybe one of you can explain:
1- there's a management stack that can bind an ID
to a group of properties
2- the proposed command then binds an ID to a VF
3- after this, VF can be assigned to a VM
Quoted 4 lines
Is this a fair summary?
In that case what is gained compared to
configuring properties on the VF directly?
I have been thinking about this question for a long time, and there maybe is a gap of understanding here.
In our scenario, we should have two roles, one is the user and the other is the vendor's management software. Do you think that the management software is running on the host, and it manages the DPU?
But in the scenario I mentioned, the vendor's management software runs on the DPU, and all hosts belong to the user.
The problem we want to solve is that this user may want 1000 network cards. So it must use VFs as a network cards. It cannot and will not configure detailed NIC properties based on the admin queue.
I don't know if I found the key to the problem.
Thanks.
Quoted 5 lines
Why is not the ID an internal property of the management?
--
MST
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 7 lines
From: Xuan Zhuo <[email protected]>
Sent: Tuesday, July 4, 2023 8:14 AMBut in the scenario I mentioned, the vendor's management software runs on the DPU, and all hosts belong to the user.
How did vendor management software configure the parameters on the src side? Same channel can be used on destination side, right?
Quoted 4 lines
The problem we want to solve is that this user may want 1000 network cards. So it must use VFs as a network cards. It cannot and will not configure detailed NIC properties based on the admin queue.
Sure.
It will not.
The idea is to use same out of band channel as what is used on the src side, also on the destination side.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 8 lines · Parav Pandit
Quoted 7 lines
From: Xuan Zhuo <[email protected]>
Sent: Tuesday, July 4, 2023 8:14 AMBut in the scenario I mentioned, the vendor's management software runs on the DPU, and all hosts belong to the user.
How did vendor management software configure the parameters on the src side?
Sorry, the src side you mean the host, right?
Quoted 1 line
Same channel can be used on destination side, right?
Do you mean that the bandwidth of the net device can be configured by the user? I think the user can only do this on the platform.
Thanks.
Quoted 8 lines
Quoted 5 lines
The problem we want to solve is that this user may want 1000 network cards. So it must use VFs as a network cards. It cannot and will not configure detailed NIC properties based on the admin queue.
Sure.
It will not.
The idea is to use same out of band channel as what is used on the src side, also on the destination side.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 10 lines
From: Xuan Zhuo <[email protected]>
Sent: Wednesday, July 5, 2023 12:30 AMQuoted 4 lines · Parav Pandit
Quoted 3 lines
From: Xuan Zhuo <[email protected]>
Sent: Tuesday, July 4, 2023 8:14 AM
Quoted 9 lines
Quoted 4 lines
Quoted 3 lines
But in the scenario I mentioned, the vendor's management software runs on the DPU, and all hosts belong to the user.
How did vendor management software configure the parameters on the src
side?
Sorry, the src side you mean the host, right?
No. in your live migration (LM) example src side mean src side of hypervisor where a virtio device is used for the first time.
Quoted 5 lines
Quoted 2 lines
Same channel can be used on destination side, right?
Do you mean that the bandwidth of the net device can be configured by the user?
No no.
Quoted 2 lines
I think the user can only do this on the platform.
Right, only by the platform.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 13 lines · Parav Pandit
Quoted 13 lines
From: Xuan Zhuo <[email protected]>
Sent: Wednesday, July 5, 2023 12:30 AMQuoted 7 lines · Parav Pandit
Quoted 7 lines
From: Xuan Zhuo <[email protected]>
Sent: Tuesday, July 4, 2023 8:14 AMBut in the scenario I mentioned, the vendor's management software runs on the DPU, and all hosts belong to the user.
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
How did vendor management software configure the parameters on the src
side?
Sorry, the src side you mean the host, right?
No. in your live migration (LM) example src side mean src side of hypervisor where a virtio device is used for the first time.
Oh, I mistake.
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
Same channel can be used on destination side, right?
Do you mean that the bandwidth of the net device can be configured by the user?
No no.
Quoted 2 lines
I think the user can only do this on the platform.
Right, only by the platform.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 9 lines · Parav Pandit
Quoted 7 lines
From: Xuan Zhuo <[email protected]>
Sent: Tuesday, July 4, 2023 8:14 AMBut in the scenario I mentioned, the vendor's management software runs on the DPU, and all hosts belong to the user.
How did vendor management software configure the parameters on the src side? Same channel can be used on destination side, right?
Yes.
Quoted 8 lines
Quoted 5 lines
The problem we want to solve is that this user may want 1000 network cards. So it must use VFs as a network cards. It cannot and will not configure detailed NIC properties based on the admin queue.
Sure.
It will not.
The idea is to use same out of band channel as what is used on the src side, also on the destination side.
Sure.
Thanks.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/