drm/xen-zcopy: Add Xen zero-copy helper DRM driver

From: Oleksandr Andrushchenko <[email protected]>

drivers/gpu/drm/xen/xen_drm_zcopy.c new file +9 -0

+static int to_refs_grant_foreign_access(struct xen_gem_object *xen_obj)
+{
+	grant_ref_t priv_gref_head;
+	int ret, j, cur_ref, num_pages;
+	struct sg_page_iter sg_iter;
+
+	ret = gnttab_alloc_grant_references(xen_obj->num_pages,
+			&priv_gref_head);
+	if (ret < 0) {

Quoted 13 lines · Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

drivers/gpu/drm/xen/xen_drm_zcopy.c new file +12 -0

+static int to_refs_grant_foreign_access(struct xen_gem_object *xen_obj)
+{
+	grant_ref_t priv_gref_head;
+	int ret, j, cur_ref, num_pages;
+	struct sg_page_iter sg_iter;
+
+	ret = gnttab_alloc_grant_references(xen_obj->num_pages,
+			&priv_gref_head);
+	if (ret < 0) {
+		DRM_ERROR("Cannot allocate grant references\n");
+		return ret;
+	}

drivers/gpu/drm/xen/xen_drm_zcopy.c line 374

Quoted 7 lines

drivers/gpu/drm/xen/xen_drm_zcopy.c new file +7 -0

+
+	j = 0;
+	num_pages = xen_obj->num_pages;
+	for_each_sg_page(xen_obj->sgt->sgl, &sg_iter, xen_obj->sgt->nents, 0) {
+		struct page *page;
+
+		page = sg_page_iter_page(&sg_iter);

Quick drive-by: You can't assume that an sgt is struct page backed.

And you probably want to check this at import/attach time.

-Daniel

Quoted 14 lines · Daniel Vetter

Quoted 13 lines · Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

drivers/gpu/drm/xen/xen_drm_zcopy.c new file +12 -0

+static int to_refs_grant_foreign_access(struct xen_gem_object *xen_obj)
+{
+	grant_ref_t priv_gref_head;
+	int ret, j, cur_ref, num_pages;
+	struct sg_page_iter sg_iter;
+
+	ret = gnttab_alloc_grant_references(xen_obj->num_pages,
+			&priv_gref_head);
+	if (ret < 0) {
+		DRM_ERROR("Cannot allocate grant references\n");
+		return ret;
+	}

Quoted 8 lines

drivers/gpu/drm/xen/xen_drm_zcopy.c line 374

Quoted 7 lines

drivers/gpu/drm/xen/xen_drm_zcopy.c new file +7 -0

+
+	j = 0;
+	num_pages = xen_obj->num_pages;
+	for_each_sg_page(xen_obj->sgt->sgl, &sg_iter, xen_obj->sgt->nents, 0) {
+		struct page *page;
+
+		page = sg_page_iter_page(&sg_iter);

Quick drive-by: You can't assume that an sgt is struct page backed.

Do you mean that someone could give me sgt which never seen sg_assign_page for its entries?

What are the other use-cases for that to happen?

Quoted 1 line

And you probably want to check this at import/attach time.

The check you mean is to make sure that when I call page = sg_page_iter_page(&sg_iter);
I have to make sure that I get a valid page?

Quoted 1 line

-Daniel

Thank you,
Oleksandr

Quoted 14 lines · Daniel Vetter

Quoted 13 lines · Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

drivers/gpu/drm/xen/xen_drm_zcopy.c new file +12 -0

+static int to_refs_grant_foreign_access(struct xen_gem_object *xen_obj)
+{
+	grant_ref_t priv_gref_head;
+	int ret, j, cur_ref, num_pages;
+	struct sg_page_iter sg_iter;
+
+	ret = gnttab_alloc_grant_references(xen_obj->num_pages,
+			&priv_gref_head);
+	if (ret < 0) {
+		DRM_ERROR("Cannot allocate grant references\n");
+		return ret;
+	}

Quoted 8 lines

drivers/gpu/drm/xen/xen_drm_zcopy.c line 374

Quoted 7 lines

drivers/gpu/drm/xen/xen_drm_zcopy.c new file +7 -0

+
+	j = 0;
+	num_pages = xen_obj->num_pages;
+	for_each_sg_page(xen_obj->sgt->sgl, &sg_iter, xen_obj->sgt->nents, 0) {
+		struct page *page;
+
+		page = sg_page_iter_page(&sg_iter);

Quick drive-by: You can't assume that an sgt is struct page backed.

Do you mean that someone could give me sgt which never seen sg_assign_page for its entries?

What are the other use-cases for that to happen?

Quoted 1 line

And you probably want to check this at import/attach time.

The check you mean is to make sure that when I call page = sg_page_iter_page(&sg_iter);
I have to make sure that I get a valid page?

Quoted 1 line

-Daniel

Thank you,
Oleksandr

Quoted 13 lines · Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

drivers/gpu/drm/xen/xen_drm_zcopy.c new file +12 -0

+static int to_refs_grant_foreign_access(struct xen_gem_object *xen_obj)
+{
+	grant_ref_t priv_gref_head;
+	int ret, j, cur_ref, num_pages;
+	struct sg_page_iter sg_iter;
+
+	ret = gnttab_alloc_grant_references(xen_obj->num_pages,
+			&priv_gref_head);
+	if (ret < 0) {
+		DRM_ERROR("Cannot allocate grant references\n");
+		return ret;
+	}

drivers/gpu/drm/xen/xen_drm_zcopy.c line 374

Quoted 7 lines

drivers/gpu/drm/xen/xen_drm_zcopy.c new file +7 -0

+
+	j = 0;
+	num_pages = xen_obj->num_pages;
+	for_each_sg_page(xen_obj->sgt->sgl, &sg_iter, xen_obj->sgt->nents, 0) {
+		struct page *page;
+
+		page = sg_page_iter_page(&sg_iter);

Quick drive-by: You can't assume that an sgt is struct page backed.

And you probably want to check this at import/attach time.

-Daniel

From: Oleksandr Andrushchenko <[email protected]>

drivers/gpu/drm/xen/xen_drm_zcopy.c new file +9 -0

+static int to_refs_grant_foreign_access(struct xen_gem_object *xen_obj)
+{
+	grant_ref_t priv_gref_head;
+	int ret, j, cur_ref, num_pages;
+	struct sg_page_iter sg_iter;
+
+	ret = gnttab_alloc_grant_references(xen_obj->num_pages,
+			&priv_gref_head);
+	if (ret < 0) {

From: Oleksandr Andrushchenko <[email protected]>

Hello!

When using Xen PV DRM frontend driver then on backend side one will need to do copying of display buffers' contents (filled by the frontend's user-space) into buffers allocated at the backend side. Taking into account the size of display buffers and frames per seconds it may result in unneeded huge data bus occupation and performance loss.

This helper driver allows implementing zero-copying use-cases when using Xen para-virtualized frontend display driver by implementing a DRM/KMS helper driver running on backend's side. It utilizes PRIME buffers API to share frontend's buffers with physical device drivers on backend's side:

- a dumb buffer created on backend's side can be shared with the Xen PV frontend driver, so it directly writes into backend's domain memory (into the buffer exported from DRM/KMS driver of a physical display device)
- a dumb buffer allocated by the frontend can be imported into physical device DRM/KMS driver, thus allowing to achieve no copying as well

For that reason number of IOCTLs are introduced:
- DRM_XEN_ZCOPY_DUMB_FROM_REFS
This will create a DRM dumb buffer from grant references provided by the frontend
- DRM_XEN_ZCOPY_DUMB_TO_REFS
This will grant references to a dumb/display buffer's memory provided by the backend
- DRM_XEN_ZCOPY_DUMB_WAIT_FREE
This will block until the dumb buffer with the wait handle provided be freed

With this helper driver I was able to drop CPU usage from 17% to 3% on Renesas R-Car M3 board.

This was tested with Renesas' Wayland-KMS and backend running as DRM master.

Thank you,
Oleksandr

Oleksandr Andrushchenko (1):
drm/xen-zcopy: Add Xen zero-copy helper DRM driver

  Documentation/gpu/drivers.rst               |   1 +
  Documentation/gpu/xen-zcopy.rst             |  32 +
  drivers/gpu/drm/xen/Kconfig                 |  25 +
  drivers/gpu/drm/xen/Makefile                |   5 +
  drivers/gpu/drm/xen/xen_drm_zcopy.c         | 880 ++++++++++++++++++++++++++++
  drivers/gpu/drm/xen/xen_drm_zcopy_balloon.c | 154 +++++
  drivers/gpu/drm/xen/xen_drm_zcopy_balloon.h |  38 ++
  include/uapi/drm/xen_zcopy_drm.h            | 129 ++++
  8 files changed, 1264 insertions(+)

create mode 100644 Documentation/gpu/xen-zcopy.rst create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy.c create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.c create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.h create mode 100644 include/uapi/drm/xen_zcopy_drm.h

Hello, all!

After discussing xen-zcopy and hyper-dmabuf [1] approaches

it seems that xen-zcopy can be made not depend on DRM core any more

and be dma-buf centric (which it in fact is).

The DRM code was mostly there for dma-buf's FD import/export

with DRM PRIME UAPI and with DRM use-cases in mind, but it comes out that if

the proposed 2 IOCTLs (DRM_XEN_ZCOPY_DUMB_FROM_REFS and DRM_XEN_ZCOPY_DUMB_TO_REFS)

are extended to also provide a file descriptor of the corresponding dma-buf, then

PRIME stuff in the driver is not needed anymore.

That being said, xen-zcopy can safely be detached from DRM and moved from

drivers/gpu/drm/xen into drivers/xen/dma-buf-backend(?).

This driver then becomes a universal way to turn any shared buffer between Dom0/DomD

and DomU(s) into a dma-buf, e.g. one can create a dma-buf from any grant references

or represent a dma-buf as grant-references for export.

This way the driver can be used not only for DRM use-cases, but also for other

use-cases which may require zero copying between domains.

For example, the use-cases we are about to work in the nearest future will use

V4L, e.g. we plan to support cameras, codecs etc. and all these will benefit

from zero copying much. Potentially, even block/net devices may benefit,

but this needs some evaluation.

I would love to hear comments for authors of the hyper-dmabuf

and Xen community, as well as DRI-Devel and other interested parties.

Thank you,

Oleksandr

Quoted 46 lines · Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

Hello!

When using Xen PV DRM frontend driver then on backend side one will need to do copying of display buffers' contents (filled by the frontend's user-space) into buffers allocated at the backend side. Taking into account the size of display buffers and frames per seconds it may result in unneeded huge data bus occupation and performance loss.

This helper driver allows implementing zero-copying use-cases when using Xen para-virtualized frontend display driver by implementing a DRM/KMS helper driver running on backend's side. It utilizes PRIME buffers API to share frontend's buffers with physical device drivers on backend's side:

- a dumb buffer created on backend's side can be shared with the Xen PV frontend driver, so it directly writes into backend's domain memory (into the buffer exported from DRM/KMS driver of a physical display device)
- a dumb buffer allocated by the frontend can be imported into physical device DRM/KMS driver, thus allowing to achieve no copying as well

For that reason number of IOCTLs are introduced:
- DRM_XEN_ZCOPY_DUMB_FROM_REFS
This will create a DRM dumb buffer from grant references provided by the frontend
- DRM_XEN_ZCOPY_DUMB_TO_REFS
This will grant references to a dumb/display buffer's memory provided by the backend
- DRM_XEN_ZCOPY_DUMB_WAIT_FREE
This will block until the dumb buffer with the wait handle provided be freed

With this helper driver I was able to drop CPU usage from 17% to 3% on Renesas R-Car M3 board.

This was tested with Renesas' Wayland-KMS and backend running as DRM master.

Thank you,
Oleksandr

Oleksandr Andrushchenko (1):
drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 15 lines

 Documentation/gpu/drivers.rst               |   1 +
 Documentation/gpu/xen-zcopy.rst             |  32 +
 drivers/gpu/drm/xen/Kconfig                 |  25 +
 drivers/gpu/drm/xen/Makefile                |   5 +
 drivers/gpu/drm/xen/xen_drm_zcopy.c         | 880 ++++++++++++++++++++++++++++
 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.c | 154 +++++
 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.h |  38 ++
 include/uapi/drm/xen_zcopy_drm.h            | 129 ++++
 8 files changed, 1264 insertions(+)
 create mode 100644 Documentation/gpu/xen-zcopy.rst
 create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy.c
 create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.c
 create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.h
 create mode 100644 include/uapi/drm/xen_zcopy_drm.h

[1]
https://lists.xenproject.org/archives/html/xen-devel/2018-02/msg01202.html

Yeah, I definitely agree on the idea of expanding the use case to the general domain where dmabuf sharing is used. However, what you are targetting with proposed changes is identical to the core design of hyper_dmabuf.

On top of this basic functionalities, hyper_dmabuf has driver level inter-domain communication, that is needed for dma-buf remote tracking (no fence forwarding though), event triggering and event handling, extra meta data exchange and hyper_dmabuf_id that represents grefs (grefs are shared implicitly on driver level)

Also it is designed with frontend (common core framework) + backend (hyper visor specific comm and memory sharing) structure for portability. We just can't limit this feature to Xen because we want to use the same uapis not only for Xen but also other applicable hypervisor, like ACORN.

So I am wondering we can start with this hyper_dmabuf then modify it for your use-case if needed and polish and fix any glitches if we want to to use this for all general dma-buf usecases.

Also, I still have one unresolved question regarding the export/import flow in both of hyper_dmabuf and xen-zcopy.

@danvet: Would this flow (guest1->import existing dmabuf->share underlying pages->guest2->map shared pages->create/export dmabuf) be acceptable now?

Regards,
DW

Quoted 119 lines · Oleksandr Andrushchenko

Hello, all!

After discussing xen-zcopy and hyper-dmabuf [1] approaches

it seems that xen-zcopy can be made not depend on DRM core any more

and be dma-buf centric (which it in fact is).

The DRM code was mostly there for dma-buf's FD import/export

with DRM PRIME UAPI and with DRM use-cases in mind, but it comes out that if

the proposed 2 IOCTLs (DRM_XEN_ZCOPY_DUMB_FROM_REFS and DRM_XEN_ZCOPY_DUMB_TO_REFS)

are extended to also provide a file descriptor of the corresponding dma-buf, then

PRIME stuff in the driver is not needed anymore.

That being said, xen-zcopy can safely be detached from DRM and moved from

drivers/gpu/drm/xen into drivers/xen/dma-buf-backend(?).

This driver then becomes a universal way to turn any shared buffer between Dom0/DomD

and DomU(s) into a dma-buf, e.g. one can create a dma-buf from any grant references

or represent a dma-buf as grant-references for export.

This way the driver can be used not only for DRM use-cases, but also for other

use-cases which may require zero copying between domains.

For example, the use-cases we are about to work in the nearest future will use

V4L, e.g. we plan to support cameras, codecs etc. and all these will benefit

from zero copying much. Potentially, even block/net devices may benefit,

but this needs some evaluation.

I would love to hear comments for authors of the hyper-dmabuf

and Xen community, as well as DRI-Devel and other interested parties.

Thank you,

Oleksandr

Quoted 46 lines · Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

Hello!

When using Xen PV DRM frontend driver then on backend side one will need to do copying of display buffers' contents (filled by the frontend's user-space) into buffers allocated at the backend side. Taking into account the size of display buffers and frames per seconds it may result in unneeded huge data bus occupation and performance loss.

This helper driver allows implementing zero-copying use-cases when using Xen para-virtualized frontend display driver by implementing a DRM/KMS helper driver running on backend's side. It utilizes PRIME buffers API to share frontend's buffers with physical device drivers on backend's side:

- a dumb buffer created on backend's side can be shared with the Xen PV frontend driver, so it directly writes into backend's domain memory (into the buffer exported from DRM/KMS driver of a physical display device)
- a dumb buffer allocated by the frontend can be imported into physical device DRM/KMS driver, thus allowing to achieve no copying as well

For that reason number of IOCTLs are introduced:
- DRM_XEN_ZCOPY_DUMB_FROM_REFS
This will create a DRM dumb buffer from grant references provided by the frontend
- DRM_XEN_ZCOPY_DUMB_TO_REFS
This will grant references to a dumb/display buffer's memory provided by the backend
- DRM_XEN_ZCOPY_DUMB_WAIT_FREE
This will block until the dumb buffer with the wait handle provided be freed

With this helper driver I was able to drop CPU usage from 17% to 3% on Renesas R-Car M3 board.

This was tested with Renesas' Wayland-KMS and backend running as DRM master.

Thank you,
Oleksandr

Oleksandr Andrushchenko (1):
drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 15 lines

 Documentation/gpu/drivers.rst               |   1 +
 Documentation/gpu/xen-zcopy.rst             |  32 +
 drivers/gpu/drm/xen/Kconfig                 |  25 +
 drivers/gpu/drm/xen/Makefile                |   5 +
 drivers/gpu/drm/xen/xen_drm_zcopy.c         | 880 ++++++++++++++++++++++++++++
 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.c | 154 +++++
 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.h |  38 ++
 include/uapi/drm/xen_zcopy_drm.h            | 129 ++++
 8 files changed, 1264 insertions(+)
 create mode 100644 Documentation/gpu/xen-zcopy.rst
 create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy.c
 create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.c
 create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.h
 create mode 100644 include/uapi/drm/xen_zcopy_drm.h

Quoted 2 lines

[1]
https://lists.xenproject.org/archives/html/xen-devel/2018-02/msg01202.html

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Quoted 10 lines · Dongwon Kim

Yeah, I definitely agree on the idea of expanding the use case to the general domain where dmabuf sharing is used. However, what you are targetting with proposed changes is identical to the core design of hyper_dmabuf.

On top of this basic functionalities, hyper_dmabuf has driver level inter-domain communication, that is needed for dma-buf remote tracking (no fence forwarding though), event triggering and event handling, extra meta data exchange and hyper_dmabuf_id that represents grefs (grefs are shared implicitly on driver level)

This really isn't a positive design aspect of hyperdmabuf imo. The core code in xen-zcopy (ignoring the ioctl side, which will be cleaned up) is very simple & clean.

If there's a clear need later on we can extend that. But for now xen-zcopy seems to cover the basic use-case needs, so gets the job done.

Quoted 4 lines

Also it is designed with frontend (common core framework) + backend (hyper visor specific comm and memory sharing) structure for portability. We just can't limit this feature to Xen because we want to use the same uapis not only for Xen but also other applicable hypervisor, like ACORN.

See the discussion around udmabuf and the needs for kvm. I think trying to make an ioctl/uapi that works for multiple hypervisors is misguided - it likely won't work.

On top of that the 2nd hypervisor you're aiming to support is ACRN. That's not even upstream yet, nor have I seen any patches proposing to land linux support for ACRN. Since it's not upstream, it doesn't really matter for upstream consideration. I'm doubting that ACRN will use the same grant references as xen, so the same uapi won't work on ACRN as on Xen anyway.

Quoted 3 lines

So I am wondering we can start with this hyper_dmabuf then modify it for your use-case if needed and polish and fix any glitches if we want to to use this for all general dma-buf usecases.

Imo xen-zcopy is a much more reasonable starting point for upstream, which can then be extended (if really proven to be necessary).

Quoted 3 lines

Also, I still have one unresolved question regarding the export/import flow in both of hyper_dmabuf and xen-zcopy.

Quoted 2 lines

@danvet: Would this flow (guest1->import existing dmabuf->share underlying pages->guest2->map shared pages->create/export dmabuf) be acceptable now?

I think if you just look at the pages, and make sure you handle the sg_page == NULL case it's ok-ish. It's not great, but mostly it should work. The real trouble with hyperdmabuf was the forwarding of all these calls, instead of just passing around a list of grant references. -Daniel

Quoted 7 lines

Regards,
DW

Quoted 3 lines · Oleksandr Andrushchenko

Hello, all!

After discussing xen-zcopy and hyper-dmabuf [1] approaches

Quoted 115 lines

Quoted 115 lines

it seems that xen-zcopy can be made not depend on DRM core any more

and be dma-buf centric (which it in fact is).

The DRM code was mostly there for dma-buf's FD import/export

with DRM PRIME UAPI and with DRM use-cases in mind, but it comes out that if

the proposed 2 IOCTLs (DRM_XEN_ZCOPY_DUMB_FROM_REFS and DRM_XEN_ZCOPY_DUMB_TO_REFS)

are extended to also provide a file descriptor of the corresponding dma-buf, then

PRIME stuff in the driver is not needed anymore.

That being said, xen-zcopy can safely be detached from DRM and moved from

drivers/gpu/drm/xen into drivers/xen/dma-buf-backend(?).

This driver then becomes a universal way to turn any shared buffer between Dom0/DomD

and DomU(s) into a dma-buf, e.g. one can create a dma-buf from any grant references

or represent a dma-buf as grant-references for export.

This way the driver can be used not only for DRM use-cases, but also for other

use-cases which may require zero copying between domains.

For example, the use-cases we are about to work in the nearest future will use

V4L, e.g. we plan to support cameras, codecs etc. and all these will benefit

from zero copying much. Potentially, even block/net devices may benefit,

but this needs some evaluation.

I would love to hear comments for authors of the hyper-dmabuf

and Xen community, as well as DRI-Devel and other interested parties.

Thank you,

Oleksandr

Quoted 61 lines · Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

Hello!

When using Xen PV DRM frontend driver then on backend side one will need to do copying of display buffers' contents (filled by the frontend's user-space) into buffers allocated at the backend side. Taking into account the size of display buffers and frames per seconds it may result in unneeded huge data bus occupation and performance loss.

This helper driver allows implementing zero-copying use-cases when using Xen para-virtualized frontend display driver by implementing a DRM/KMS helper driver running on backend's side. It utilizes PRIME buffers API to share frontend's buffers with physical device drivers on backend's side:

- a dumb buffer created on backend's side can be shared with the Xen PV frontend driver, so it directly writes into backend's domain memory (into the buffer exported from DRM/KMS driver of a physical display device)
- a dumb buffer allocated by the frontend can be imported into physical device DRM/KMS driver, thus allowing to achieve no copying as well

For that reason number of IOCTLs are introduced:
- DRM_XEN_ZCOPY_DUMB_FROM_REFS
This will create a DRM dumb buffer from grant references provided by the frontend
- DRM_XEN_ZCOPY_DUMB_TO_REFS
This will grant references to a dumb/display buffer's memory provided by the backend
- DRM_XEN_ZCOPY_DUMB_WAIT_FREE
This will block until the dumb buffer with the wait handle provided be freed

With this helper driver I was able to drop CPU usage from 17% to 3% on Renesas R-Car M3 board.

This was tested with Renesas' Wayland-KMS and backend running as DRM master.

Thank you,
Oleksandr

Oleksandr Andrushchenko (1):
drm/xen-zcopy: Add Xen zero-copy helper DRM driver

  Documentation/gpu/drivers.rst               |   1 +
  Documentation/gpu/xen-zcopy.rst             |  32 +
  drivers/gpu/drm/xen/Kconfig                 |  25 +
  drivers/gpu/drm/xen/Makefile                |   5 +
  drivers/gpu/drm/xen/xen_drm_zcopy.c         | 880 ++++++++++++++++++++++++++++
  drivers/gpu/drm/xen/xen_drm_zcopy_balloon.c | 154 +++++
  drivers/gpu/drm/xen/xen_drm_zcopy_balloon.h |  38 ++
  include/uapi/drm/xen_zcopy_drm.h            | 129 ++++
  8 files changed, 1264 insertions(+)

create mode 100644 Documentation/gpu/xen-zcopy.rst create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy.c create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.c create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.h create mode 100644 include/uapi/drm/xen_zcopy_drm.h

Quoted 6 lines

Quoted 2 lines

[1]
https://lists.xenproject.org/archives/html/xen-devel/2018-02/msg01202.html

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Quoted 10 lines · Dongwon Kim

Yeah, I definitely agree on the idea of expanding the use case to the general domain where dmabuf sharing is used. However, what you are targetting with proposed changes is identical to the core design of hyper_dmabuf.

On top of this basic functionalities, hyper_dmabuf has driver level inter-domain communication, that is needed for dma-buf remote tracking (no fence forwarding though), event triggering and event handling, extra meta data exchange and hyper_dmabuf_id that represents grefs (grefs are shared implicitly on driver level)

This really isn't a positive design aspect of hyperdmabuf imo. The core code in xen-zcopy (ignoring the ioctl side, which will be cleaned up) is very simple & clean.

If there's a clear need later on we can extend that. But for now xen-zcopy seems to cover the basic use-case needs, so gets the job done.

Quoted 4 lines

Also it is designed with frontend (common core framework) + backend (hyper visor specific comm and memory sharing) structure for portability. We just can't limit this feature to Xen because we want to use the same uapis not only for Xen but also other applicable hypervisor, like ACORN.

See the discussion around udmabuf and the needs for kvm. I think trying to make an ioctl/uapi that works for multiple hypervisors is misguided - it likely won't work.

On top of that the 2nd hypervisor you're aiming to support is ACRN. That's not even upstream yet, nor have I seen any patches proposing to land linux support for ACRN. Since it's not upstream, it doesn't really matter for upstream consideration. I'm doubting that ACRN will use the same grant references as xen, so the same uapi won't work on ACRN as on Xen anyway.

Quoted 3 lines

So I am wondering we can start with this hyper_dmabuf then modify it for your use-case if needed and polish and fix any glitches if we want to to use this for all general dma-buf usecases.

Imo xen-zcopy is a much more reasonable starting point for upstream, which can then be extended (if really proven to be necessary).

Quoted 3 lines

Also, I still have one unresolved question regarding the export/import flow in both of hyper_dmabuf and xen-zcopy.

Quoted 2 lines

@danvet: Would this flow (guest1->import existing dmabuf->share underlying pages->guest2->map shared pages->create/export dmabuf) be acceptable now?

I think if you just look at the pages, and make sure you handle the sg_page == NULL case it's ok-ish. It's not great, but mostly it should work. The real trouble with hyperdmabuf was the forwarding of all these calls, instead of just passing around a list of grant references. -Daniel

Quoted 7 lines

Regards,
DW

Quoted 3 lines · Oleksandr Andrushchenko

Hello, all!

After discussing xen-zcopy and hyper-dmabuf [1] approaches

Quoted 115 lines

Quoted 115 lines

it seems that xen-zcopy can be made not depend on DRM core any more

and be dma-buf centric (which it in fact is).

The DRM code was mostly there for dma-buf's FD import/export

with DRM PRIME UAPI and with DRM use-cases in mind, but it comes out that if

the proposed 2 IOCTLs (DRM_XEN_ZCOPY_DUMB_FROM_REFS and DRM_XEN_ZCOPY_DUMB_TO_REFS)

are extended to also provide a file descriptor of the corresponding dma-buf, then

PRIME stuff in the driver is not needed anymore.

That being said, xen-zcopy can safely be detached from DRM and moved from

drivers/gpu/drm/xen into drivers/xen/dma-buf-backend(?).

This driver then becomes a universal way to turn any shared buffer between Dom0/DomD

and DomU(s) into a dma-buf, e.g. one can create a dma-buf from any grant references

or represent a dma-buf as grant-references for export.

This way the driver can be used not only for DRM use-cases, but also for other

use-cases which may require zero copying between domains.

For example, the use-cases we are about to work in the nearest future will use

V4L, e.g. we plan to support cameras, codecs etc. and all these will benefit

from zero copying much. Potentially, even block/net devices may benefit,

but this needs some evaluation.

I would love to hear comments for authors of the hyper-dmabuf

and Xen community, as well as DRI-Devel and other interested parties.

Thank you,

Oleksandr

Quoted 61 lines · Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

Hello!

When using Xen PV DRM frontend driver then on backend side one will need to do copying of display buffers' contents (filled by the frontend's user-space) into buffers allocated at the backend side. Taking into account the size of display buffers and frames per seconds it may result in unneeded huge data bus occupation and performance loss.

This helper driver allows implementing zero-copying use-cases when using Xen para-virtualized frontend display driver by implementing a DRM/KMS helper driver running on backend's side. It utilizes PRIME buffers API to share frontend's buffers with physical device drivers on backend's side:

- a dumb buffer created on backend's side can be shared with the Xen PV frontend driver, so it directly writes into backend's domain memory (into the buffer exported from DRM/KMS driver of a physical display device)
- a dumb buffer allocated by the frontend can be imported into physical device DRM/KMS driver, thus allowing to achieve no copying as well

For that reason number of IOCTLs are introduced:
- DRM_XEN_ZCOPY_DUMB_FROM_REFS
This will create a DRM dumb buffer from grant references provided by the frontend
- DRM_XEN_ZCOPY_DUMB_TO_REFS
This will grant references to a dumb/display buffer's memory provided by the backend
- DRM_XEN_ZCOPY_DUMB_WAIT_FREE
This will block until the dumb buffer with the wait handle provided be freed

With this helper driver I was able to drop CPU usage from 17% to 3% on Renesas R-Car M3 board.

This was tested with Renesas' Wayland-KMS and backend running as DRM master.

Thank you,
Oleksandr

Oleksandr Andrushchenko (1):
drm/xen-zcopy: Add Xen zero-copy helper DRM driver

  Documentation/gpu/drivers.rst               |   1 +
  Documentation/gpu/xen-zcopy.rst             |  32 +
  drivers/gpu/drm/xen/Kconfig                 |  25 +
  drivers/gpu/drm/xen/Makefile                |   5 +
  drivers/gpu/drm/xen/xen_drm_zcopy.c         | 880 ++++++++++++++++++++++++++++
  drivers/gpu/drm/xen/xen_drm_zcopy_balloon.c | 154 +++++
  drivers/gpu/drm/xen/xen_drm_zcopy_balloon.h |  38 ++
  include/uapi/drm/xen_zcopy_drm.h            | 129 ++++
  8 files changed, 1264 insertions(+)

create mode 100644 Documentation/gpu/xen-zcopy.rst create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy.c create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.c create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.h create mode 100644 include/uapi/drm/xen_zcopy_drm.h

Quoted 6 lines

Quoted 2 lines

[1]
https://lists.xenproject.org/archives/html/xen-devel/2018-02/msg01202.html

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Quoted 10 lines · Dongwon Kim

Yeah, I definitely agree on the idea of expanding the use case to the general domain where dmabuf sharing is used. However, what you are targetting with proposed changes is identical to the core design of hyper_dmabuf.

On top of this basic functionalities, hyper_dmabuf has driver level inter-domain communication, that is needed for dma-buf remote tracking (no fence forwarding though), event triggering and event handling, extra meta data exchange and hyper_dmabuf_id that represents grefs (grefs are shared implicitly on driver level)

This really isn't a positive design aspect of hyperdmabuf imo. The core code in xen-zcopy (ignoring the ioctl side, which will be cleaned up) is very simple & clean.

If there's a clear need later on we can extend that. But for now xen-zcopy seems to cover the basic use-case needs, so gets the job done.

Quoted 4 lines

Also it is designed with frontend (common core framework) + backend (hyper visor specific comm and memory sharing) structure for portability. We just can't limit this feature to Xen because we want to use the same uapis not only for Xen but also other applicable hypervisor, like ACORN.

See the discussion around udmabuf and the needs for kvm. I think trying to make an ioctl/uapi that works for multiple hypervisors is misguided - it likely won't work.

On top of that the 2nd hypervisor you're aiming to support is ACRN. That's not even upstream yet, nor have I seen any patches proposing to land linux support for ACRN. Since it's not upstream, it doesn't really matter for upstream consideration. I'm doubting that ACRN will use the same grant references as xen, so the same uapi won't work on ACRN as on Xen anyway.

Quoted 4 lines

So I am wondering we can start with this hyper_dmabuf then modify it for your use-case if needed and polish and fix any glitches if we want to to use this for all general dma-buf usecases.

Imo xen-zcopy is a much more reasonable starting point for upstream, which can then be extended (if really proven to be necessary).

Quoted 3 lines

Also, I still have one unresolved question regarding the export/import flow in both of hyper_dmabuf and xen-zcopy.

Quoted 2 lines

@danvet: Would this flow (guest1->import existing dmabuf->share underlying pages->guest2->map shared pages->create/export dmabuf) be acceptable now?

I think if you just look at the pages, and make sure you handle the sg_page == NULL case it's ok-ish. It's not great, but mostly it should work. The real trouble with hyperdmabuf was the forwarding of all these calls, instead of just passing around a list of grant references.

-Daniel

Quoted 130 lines

Regards,
DW

Quoted 119 lines · Oleksandr Andrushchenko

Hello, all!

After discussing xen-zcopy and hyper-dmabuf [1] approaches

it seems that xen-zcopy can be made not depend on DRM core any more

and be dma-buf centric (which it in fact is).

The DRM code was mostly there for dma-buf's FD import/export

with DRM PRIME UAPI and with DRM use-cases in mind, but it comes out that if

the proposed 2 IOCTLs (DRM_XEN_ZCOPY_DUMB_FROM_REFS and DRM_XEN_ZCOPY_DUMB_TO_REFS)

are extended to also provide a file descriptor of the corresponding dma-buf, then

PRIME stuff in the driver is not needed anymore.

That being said, xen-zcopy can safely be detached from DRM and moved from

drivers/gpu/drm/xen into drivers/xen/dma-buf-backend(?).

This driver then becomes a universal way to turn any shared buffer between Dom0/DomD

and DomU(s) into a dma-buf, e.g. one can create a dma-buf from any grant references

or represent a dma-buf as grant-references for export.

This way the driver can be used not only for DRM use-cases, but also for other

use-cases which may require zero copying between domains.

For example, the use-cases we are about to work in the nearest future will use

V4L, e.g. we plan to support cameras, codecs etc. and all these will benefit

from zero copying much. Potentially, even block/net devices may benefit,

but this needs some evaluation.

I would love to hear comments for authors of the hyper-dmabuf

and Xen community, as well as DRI-Devel and other interested parties.

Thank you,

Oleksandr

Quoted 46 lines · Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

Hello!

When using Xen PV DRM frontend driver then on backend side one will need to do copying of display buffers' contents (filled by the frontend's user-space) into buffers allocated at the backend side. Taking into account the size of display buffers and frames per seconds it may result in unneeded huge data bus occupation and performance loss.

This helper driver allows implementing zero-copying use-cases when using Xen para-virtualized frontend display driver by implementing a DRM/KMS helper driver running on backend's side. It utilizes PRIME buffers API to share frontend's buffers with physical device drivers on backend's side:

- a dumb buffer created on backend's side can be shared with the Xen PV frontend driver, so it directly writes into backend's domain memory (into the buffer exported from DRM/KMS driver of a physical display device)
- a dumb buffer allocated by the frontend can be imported into physical device DRM/KMS driver, thus allowing to achieve no copying as well

For that reason number of IOCTLs are introduced:
- DRM_XEN_ZCOPY_DUMB_FROM_REFS
This will create a DRM dumb buffer from grant references provided by the frontend
- DRM_XEN_ZCOPY_DUMB_TO_REFS
This will grant references to a dumb/display buffer's memory provided by the backend
- DRM_XEN_ZCOPY_DUMB_WAIT_FREE
This will block until the dumb buffer with the wait handle provided be freed

With this helper driver I was able to drop CPU usage from 17% to 3% on Renesas R-Car M3 board.

This was tested with Renesas' Wayland-KMS and backend running as DRM master.

Thank you,
Oleksandr

Oleksandr Andrushchenko (1):
drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 15 lines

 Documentation/gpu/drivers.rst               |   1 +
 Documentation/gpu/xen-zcopy.rst             |  32 +
 drivers/gpu/drm/xen/Kconfig                 |  25 +
 drivers/gpu/drm/xen/Makefile                |   5 +
 drivers/gpu/drm/xen/xen_drm_zcopy.c         | 880 ++++++++++++++++++++++++++++
 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.c | 154 +++++
 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.h |  38 ++
 include/uapi/drm/xen_zcopy_drm.h            | 129 ++++
 8 files changed, 1264 insertions(+)
 create mode 100644 Documentation/gpu/xen-zcopy.rst
 create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy.c
 create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.c
 create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.h
 create mode 100644 include/uapi/drm/xen_zcopy_drm.h

Quoted 2 lines

[1]
https://lists.xenproject.org/archives/html/xen-devel/2018-02/msg01202.html

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Signature · 3 lines

Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Quoted 26 lines · Daniel Vetter

Quoted 10 lines · Dongwon Kim

Yeah, I definitely agree on the idea of expanding the use case to the general domain where dmabuf sharing is used. However, what you are targetting with proposed changes is identical to the core design of hyper_dmabuf.

On top of this basic functionalities, hyper_dmabuf has driver level inter-domain communication, that is needed for dma-buf remote tracking (no fence forwarding though), event triggering and event handling, extra meta data exchange and hyper_dmabuf_id that represents grefs (grefs are shared implicitly on driver level)

This really isn't a positive design aspect of hyperdmabuf imo. The core code in xen-zcopy (ignoring the ioctl side, which will be cleaned up) is very simple & clean.

If there's a clear need later on we can extend that. But for now xen-zcopy seems to cover the basic use-case needs, so gets the job done.

Quoted 4 lines

Also it is designed with frontend (common core framework) + backend (hyper visor specific comm and memory sharing) structure for portability. We just can't limit this feature to Xen because we want to use the same uapis not only for Xen but also other applicable hypervisor, like ACORN.

See the discussion around udmabuf and the needs for kvm. I think trying to make an ioctl/uapi that works for multiple hypervisors is misguided - it likely won't work.

Quoted 5 lines

On top of that the 2nd hypervisor you're aiming to support is ACRN. That's not even upstream yet, nor have I seen any patches proposing to land linux support for ACRN. Since it's not upstream, it doesn't really matter for upstream consideration. I'm doubting that ACRN will use the same grant references as xen, so the same uapi won't work on ACRN as on Xen anyway.

Yeah, ACRN doesn't have grant-table. Only Xen supports it. But that is why hyper_dmabuf has been architectured with the concept of backend. If you look at the structure of backend, you will find that backend is just a set of standard function calls as shown here:

struct hyper_dmabuf_bknd_ops {
         /* backend initialization routine (optional) */
         int (*init)(void);

         /* backend cleanup routine (optional) */
         int (*cleanup)(void);

         /* retreiving id of current virtual machine */
         int (*get_vm_id)(void);

         /* get pages shared via hypervisor-specific method */
         int (*share_pages)(struct page **pages, int vm_id,
                            int nents, void **refs_info);

         /* make shared pages unshared via hypervisor specific method */
         int (*unshare_pages)(void **refs_info, int nents);

         /* map remotely shared pages on importer's side via
          * hypervisor-specific method
          */
         struct page ** (*map_shared_pages)(unsigned long ref, int vm_id,
                                            int nents, void **refs_info);

         /* unmap and free shared pages on importer's side via
          * hypervisor-specific method
          */
         int (*unmap_shared_pages)(void **refs_info, int nents);

         /* initialize communication environment */
         int (*init_comm_env)(void);

         void (*destroy_comm)(void);

         /* upstream ch setup (receiving and responding) */
         int (*init_rx_ch)(int vm_id);

         /* downstream ch setup (transmitting and parsing responses) */
         int (*init_tx_ch)(int vm_id);

         int (*send_req)(int vm_id, struct hyper_dmabuf_req *req, int wait);
};

All of these can be mapped with any hypervisor specific implementation. We designed backend implementation for Xen using grant-table, Xen event and ring buffer communication. For ACRN, we have another backend using Virt-IO for both memory sharing and communication.

We tried to define this structure of backend to make it general enough (or it can be even modified or extended to support more cases.) so that it can fit to other hypervisor cases. Only requirements/expectation on the hypervisor are page-level memory sharing and inter-domain communication, which I think are standard features of modern hypervisor.

And please review common UAPIs that hyper_dmabuf and xen-zcopy supports. They are very general. One is getting FD (dmabuf) and get those shared. The other is generating dmabuf from global handle (secure handle hiding gref behind it). On top of this, hyper_dmabuf has "unshare" and "query" which are also useful for any cases.

So I don't know why we wouldn't want to try to make these standard in most of hypervisor cases instead of limiting it to certain hypervisor like Xen. Frontend-backend structre is optimal for this I think.

Quoted 9 lines

Quoted 3 lines

So I am wondering we can start with this hyper_dmabuf then modify it for your use-case if needed and polish and fix any glitches if we want to to use this for all general dma-buf usecases.

Imo xen-zcopy is a much more reasonable starting point for upstream, which can then be extended (if really proven to be necessary).

Quoted 3 lines

Also, I still have one unresolved question regarding the export/import flow in both of hyper_dmabuf and xen-zcopy.

Quoted 6 lines

Quoted 2 lines

@danvet: Would this flow (guest1->import existing dmabuf->share underlying pages->guest2->map shared pages->create/export dmabuf) be acceptable now?

I think if you just look at the pages, and make sure you handle the sg_page == NULL case it's ok-ish. It's not great, but mostly it should work. The real trouble with hyperdmabuf was the forwarding of all these calls, instead of just passing around a list of grant references.

I talked to danvet about this litte bit.

I think there was some misunderstanding on this "forwarding". Exporting and importing flow in hyper_dmabuf are basically same as xen-zcopy's. I think what made confusion was that importing domain notifies exporting domain when there are dmabuf operations (like attach, mapping, detach and release) so that exporting domain can track the usage of dmabuf on the importing domain.

I designed this for some basic tracking. We may not need to notify for every different activity but if none of them is there, exporting domain can't determine if it is ok to unshare the buffer or the originator (like i915) can free the object even if it's being accessed in importing domain.

Anyway I really hope we can have enough discussion and resolve all concerns before nailing it down.

Quoted 127 lines

-Daniel

Quoted 125 lines

Regards,
DW

Quoted 121 lines · Oleksandr Andrushchenko

Hello, all!

After discussing xen-zcopy and hyper-dmabuf [1] approaches

it seems that xen-zcopy can be made not depend on DRM core any more

and be dma-buf centric (which it in fact is).

The DRM code was mostly there for dma-buf's FD import/export

with DRM PRIME UAPI and with DRM use-cases in mind, but it comes out that if

the proposed 2 IOCTLs (DRM_XEN_ZCOPY_DUMB_FROM_REFS and DRM_XEN_ZCOPY_DUMB_TO_REFS)

are extended to also provide a file descriptor of the corresponding dma-buf, then

PRIME stuff in the driver is not needed anymore.

That being said, xen-zcopy can safely be detached from DRM and moved from

drivers/gpu/drm/xen into drivers/xen/dma-buf-backend(?).

This driver then becomes a universal way to turn any shared buffer between Dom0/DomD

and DomU(s) into a dma-buf, e.g. one can create a dma-buf from any grant references

or represent a dma-buf as grant-references for export.

This way the driver can be used not only for DRM use-cases, but also for other

use-cases which may require zero copying between domains.

For example, the use-cases we are about to work in the nearest future will use

V4L, e.g. we plan to support cameras, codecs etc. and all these will benefit

from zero copying much. Potentially, even block/net devices may benefit,

but this needs some evaluation.

I would love to hear comments for authors of the hyper-dmabuf

and Xen community, as well as DRI-Devel and other interested parties.

Thank you,

Oleksandr

Quoted 55 lines · Oleksandr Andrushchenko

From: Oleksandr Andrushchenko <[email protected]>

Hello!

When using Xen PV DRM frontend driver then on backend side one will need to do copying of display buffers' contents (filled by the frontend's user-space) into buffers allocated at the backend side. Taking into account the size of display buffers and frames per seconds it may result in unneeded huge data bus occupation and performance loss.

This helper driver allows implementing zero-copying use-cases when using Xen para-virtualized frontend display driver by implementing a DRM/KMS helper driver running on backend's side. It utilizes PRIME buffers API to share frontend's buffers with physical device drivers on backend's side:

- a dumb buffer created on backend's side can be shared with the Xen PV frontend driver, so it directly writes into backend's domain memory (into the buffer exported from DRM/KMS driver of a physical display device)
- a dumb buffer allocated by the frontend can be imported into physical device DRM/KMS driver, thus allowing to achieve no copying as well

For that reason number of IOCTLs are introduced:
- DRM_XEN_ZCOPY_DUMB_FROM_REFS
This will create a DRM dumb buffer from grant references provided by the frontend
- DRM_XEN_ZCOPY_DUMB_TO_REFS
This will grant references to a dumb/display buffer's memory provided by the backend
- DRM_XEN_ZCOPY_DUMB_WAIT_FREE
This will block until the dumb buffer with the wait handle provided be freed

With this helper driver I was able to drop CPU usage from 17% to 3% on Renesas R-Car M3 board.

This was tested with Renesas' Wayland-KMS and backend running as DRM master.

Thank you,
Oleksandr

Oleksandr Andrushchenko (1):
drm/xen-zcopy: Add Xen zero-copy helper DRM driver

  Documentation/gpu/drivers.rst               |   1 +
  Documentation/gpu/xen-zcopy.rst             |  32 +
  drivers/gpu/drm/xen/Kconfig                 |  25 +
  drivers/gpu/drm/xen/Makefile                |   5 +
  drivers/gpu/drm/xen/xen_drm_zcopy.c         | 880 ++++++++++++++++++++++++++++
  drivers/gpu/drm/xen/xen_drm_zcopy_balloon.c | 154 +++++
  drivers/gpu/drm/xen/xen_drm_zcopy_balloon.h |  38 ++
  include/uapi/drm/xen_zcopy_drm.h            | 129 ++++
  8 files changed, 1264 insertions(+)

Quoted 6 lines

create mode 100644 Documentation/gpu/xen-zcopy.rst create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy.c create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.c create mode 100644 drivers/gpu/drm/xen/xen_drm_zcopy_balloon.h create mode 100644 include/uapi/drm/xen_zcopy_drm.h

[1]
https://lists.xenproject.org/archives/html/xen-devel/2018-02/msg01202.html

Quoted 8 lines

Quoted 4 lines

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Signature · 3 lines

Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Quoted 2 lines · Dongwon Kim

Quoted 1 line · Daniel Vetter

On Mon, Apr 16, 2018 at 12:29:05PM -0700, Dongwon Kim wrote:

3.2 Backend exports dma-buf to xen-front

In this case Dom0 pages are shared with DomU. As before, DomU can only write to these pages, not any other page from Dom0, so it can be still considered safe.
But, the following must be considered (highlighted in xen-front's Kernel documentation):
 - If guest domain dies then pages/grants received from the backend cannot be claimed back - think of it as memory lost to Dom0 (won't be used for any
   other guest)
 - Misbehaving guest may send too many requests to the backend exhausting its grant references and memory (consider this from security POV). As the backend runs in the trusted domain we also assume that it is trusted as well,
   e.g. must take measures to prevent DDoS attacks.

4. xen-front/backend/xen-zcopy synchronization

4.1. As I already said in 2) all the inter VM communication happens between xen-front and the backend, xen-zcopy is NOT involved in that. When xen-front wants to destroy a display buffer (dumb/dma-buf) it issues a XENDISPL_OP_DBUF_DESTROY command (opposite to XENDISPL_OP_DBUF_CREATE). This call is synchronous, so xen-front expects that backend does free the buffer pages on return.

4.2. Backend, on XENDISPL_OP_DBUF_DESTROY:
  - closes all dumb handles/fd's of the buffer according to [3]
  - issues DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE IOCTL to xen-zcopy to make sure
    the buffer is freed (think of it as it waits for dma-buf->release callback)

  - replies to xen-front that the buffer can be destroyed.
This way deletion of the buffer happens synchronously on both Dom0 and DomU sides. In case if DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE returns with time-out error
(BTW, wait time is a parameter of this IOCTL), Xen will defer grant reference
removal and will retry later until those are free.

Hope this helps understand how buffers are synchronously deleted in case of xen-zcopy with a single protocol command.

I think the above logic can also be re-used by the hyper-dmabuf driver with some additional work:

1. xen-zcopy can be split into 2 parts and extend:
1.1. Xen gntdev driver [4], [5] to allow creating dma-buf from grefs and vise versa,

implement "wait" ioctl (wait for dma-buf->release): currently these are DRM_XEN_ZCOPY_DUMB_FROM_REFS, DRM_XEN_ZCOPY_DUMB_TO_REFS and DRM_XEN_ZCOPY_DUMB_WAIT_FREE
1.2. Xen balloon driver [6] to allow allocating contiguous buffers (not needed
by current hyper-dmabuf, but is a must for xen-zcopy use-cases)

Quoted 5 lines · Oleksandr Andrushchenko

Quoted 2 lines · Dongwon Kim

Quoted 1 line · Daniel Vetter

On Mon, Apr 16, 2018 at 12:29:05PM -0700, Dongwon Kim wrote:

3.2 Backend exports dma-buf to xen-front

Quoted 14 lines

In this case Dom0 pages are shared with DomU. As before, DomU can only write to these pages, not any other page from Dom0, so it can be still considered safe.
But, the following must be considered (highlighted in xen-front's Kernel documentation):
 - If guest domain dies then pages/grants received from the backend cannot be claimed back - think of it as memory lost to Dom0 (won't be used for
any
   other guest)
 - Misbehaving guest may send too many requests to the backend exhausting its grant references and memory (consider this from security POV). As the backend runs in the trusted domain we also assume that it is trusted as
well,
   e.g. must take measures to prevent DDoS attacks.

I cannot parse the above sentence:

"As the backend runs in the trusted domain we also assume that it is trusted as well, e.g. must take measures to prevent DDoS attacks."

What's the relation between being trusted and protecting from DoS attacks?

In any case, all? PV protocols are implemented with the frontend sharing pages to the backend, and I think there's a reason why this model is used, and it should continue to be used.

Having to add logic in the backend to prevent such attacks means that:

- We need more code in the backend, which increases complexity and chances of bugs.
- Such code/logic could be wrong, thus allowing DoS.

Quoted 9 lines

4. xen-front/backend/xen-zcopy synchronization

4.1. As I already said in 2) all the inter VM communication happens between xen-front and the backend, xen-zcopy is NOT involved in that. When xen-front wants to destroy a display buffer (dumb/dma-buf) it issues a XENDISPL_OP_DBUF_DESTROY command (opposite to XENDISPL_OP_DBUF_CREATE). This call is synchronous, so xen-front expects that backend does free the buffer pages on return.

Quoted 6 lines

4.2. Backend, on XENDISPL_OP_DBUF_DESTROY:
  - closes all dumb handles/fd's of the buffer according to [3]
  - issues DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE IOCTL to xen-zcopy to make
sure
    the buffer is freed (think of it as it waits for dma-buf->release
callback)

So this zcopy thing keeps some kind of track of the memory usage? Why can't the user-space backend keep track of the buffer usage?

Quoted 14 lines

  - replies to xen-front that the buffer can be destroyed.
This way deletion of the buffer happens synchronously on both Dom0 and DomU sides. In case if DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE returns with time-out error
(BTW, wait time is a parameter of this IOCTL), Xen will defer grant reference
removal and will retry later until those are free.

Hope this helps understand how buffers are synchronously deleted in case of xen-zcopy with a single protocol command.

I think the above logic can also be re-used by the hyper-dmabuf driver with some additional work:

Quoted 3 lines

1. xen-zcopy can be split into 2 parts and extend:
1.1. Xen gntdev driver [4], [5] to allow creating dma-buf from grefs and vise versa,

I don't know much about the dma-buf implementation in Linux, but gntdev is a user-space device, and AFAICT user-space applications don't have any notion of dma buffers. How are such buffers useful for user-space? Why can't this just be called memory?

Also, (with my FreeBSD maintainer hat) how is this going to translate to other OSes? So far the operations performed by the gntdev device are mostly OS-agnostic because this just map/unmap memory, and in fact they are implemented by Linux and FreeBSD.

Quoted 6 lines

implement "wait" ioctl (wait for dma-buf->release): currently these are DRM_XEN_ZCOPY_DUMB_FROM_REFS, DRM_XEN_ZCOPY_DUMB_TO_REFS and DRM_XEN_ZCOPY_DUMB_WAIT_FREE
1.2. Xen balloon driver [6] to allow allocating contiguous buffers (not needed
by current hyper-dmabuf, but is a must for xen-zcopy use-cases)

I think this needs clarifying. In which memory space do you need those regions to be contiguous?

Do they need to be contiguous in host physical memory, or guest physical memory?

If it's in guest memory space, isn't there any generic interface that you can use?

If it's in host physical memory space, why do you need this buffer to be contiguous in host physical memory space? The IOMMU should hide all this.

Thanks, Roger.

Quoted 20 lines · Roger Pau Monné

Quoted 5 lines · Oleksandr Andrushchenko

Quoted 2 lines · Dongwon Kim

Quoted 1 line · Daniel Vetter

On Mon, Apr 16, 2018 at 12:29:05PM -0700, Dongwon Kim wrote:

3.2 Backend exports dma-buf to xen-front

Quoted 14 lines

In this case Dom0 pages are shared with DomU. As before, DomU can only write to these pages, not any other page from Dom0, so it can be still considered safe.
But, the following must be considered (highlighted in xen-front's Kernel documentation):
 - If guest domain dies then pages/grants received from the backend cannot be claimed back - think of it as memory lost to Dom0 (won't be used for
any
   other guest)
 - Misbehaving guest may send too many requests to the backend exhausting its grant references and memory (consider this from security POV). As the backend runs in the trusted domain we also assume that it is trusted as
well,
   e.g. must take measures to prevent DDoS attacks.

Quoted 7 lines

I cannot parse the above sentence:

"As the backend runs in the trusted domain we also assume that it is trusted as well, e.g. must take measures to prevent DDoS attacks."

What's the relation between being trusted and protecting from DoS attacks?

I mean that we trust the backend that it can prevent Dom0 from crashing in case DomU's frontend misbehaves, e.g. if the frontend sends too many memory requests etc.

Quoted 3 lines

In any case, all? PV protocols are implemented with the frontend sharing pages to the backend, and I think there's a reason why this model is used, and it should continue to be used.

This is the first use-case above. But there are real-world use-cases (embedded in my case) when physically contiguous memory needs to be shared, one of the possible ways to achieve this is to share contiguous memory from Dom0 to DomU (the second use-case above)

Quoted 3 lines

Having to add logic in the backend to prevent such attacks means that:

Quoted 3 lines

- We need more code in the backend, which increases complexity and chances of bugs.
- Such code/logic could be wrong, thus allowing DoS.

You can live without this code at all, but this is then up to backend which may make Dom0 down because of DomU's frontend doing evil things

Quoted 9 lines

Quoted 9 lines

4. xen-front/backend/xen-zcopy synchronization

4.1. As I already said in 2) all the inter VM communication happens between xen-front and the backend, xen-zcopy is NOT involved in that. When xen-front wants to destroy a display buffer (dumb/dma-buf) it issues a XENDISPL_OP_DBUF_DESTROY command (opposite to XENDISPL_OP_DBUF_CREATE). This call is synchronous, so xen-front expects that backend does free the buffer pages on return.

Quoted 8 lines

Quoted 6 lines

4.2. Backend, on XENDISPL_OP_DBUF_DESTROY:
  - closes all dumb handles/fd's of the buffer according to [3]
  - issues DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE IOCTL to xen-zcopy to make
sure
    the buffer is freed (think of it as it waits for dma-buf->release
callback)

So this zcopy thing keeps some kind of track of the memory usage? Why can't the user-space backend keep track of the buffer usage?

Because there is no dma-buf UAPI which allows to track the buffer life cycle (e.g. wait until dma-buf's .release callback is called)

Quoted 14 lines

Quoted 14 lines

  - replies to xen-front that the buffer can be destroyed.
This way deletion of the buffer happens synchronously on both Dom0 and DomU sides. In case if DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE returns with time-out error
(BTW, wait time is a parameter of this IOCTL), Xen will defer grant reference
removal and will retry later until those are free.

Hope this helps understand how buffers are synchronously deleted in case of xen-zcopy with a single protocol command.

I think the above logic can also be re-used by the hyper-dmabuf driver with some additional work:

Quoted 7 lines

Quoted 3 lines

1. xen-zcopy can be split into 2 parts and extend:
1.1. Xen gntdev driver [4], [5] to allow creating dma-buf from grefs and vise versa,

I don't know much about the dma-buf implementation in Linux, but gntdev is a user-space device, and AFAICT user-space applications don't have any notion of dma buffers. How are such buffers useful for user-space? Why can't this just be called memory?

A dma-buf is seen by user-space as a file descriptor and you can pass it to different drivers then. For example, you can share a buffer used by a display driver for scanout with a GPU, to compose a picture into it:
1. User-space (US) allocates a display buffer from display driver
2. US asks display driver to export the dma-buf which backs up that buffer, US gets buffer's fd: dma_buf_fd
3. US asks GPU driver to import a buffer and provides it with dma_buf_fd
4. GPU renders contents into display buffer (dma_buf_fd)

Finally, this is indeed some memory, but a bit more [1]

Quoted 5 lines

Also, (with my FreeBSD maintainer hat) how is this going to translate to other OSes? So far the operations performed by the gntdev device are mostly OS-agnostic because this just map/unmap memory, and in fact they are implemented by Linux and FreeBSD.

At the moment I can only see Linux implementation and it seems to be perfectly ok as we do not change Xen's APIs etc. and only use the existing ones (remember, we only extend gntdev/balloon drivers, all the changes in the Linux kernel)
As the second note I can also think that we do not extend gntdev/balloon drivers and have re-worked xen-zcopy driver be a separate entity, say drivers/xen/dma-buf

Quoted 8 lines

Quoted 6 lines

implement "wait" ioctl (wait for dma-buf->release): currently these are DRM_XEN_ZCOPY_DUMB_FROM_REFS, DRM_XEN_ZCOPY_DUMB_TO_REFS and DRM_XEN_ZCOPY_DUMB_WAIT_FREE
1.2. Xen balloon driver [6] to allow allocating contiguous buffers (not needed
by current hyper-dmabuf, but is a must for xen-zcopy use-cases)

I think this needs clarifying. In which memory space do you need those regions to be contiguous?

Use-case: Dom0 has a HW driver which only works with contig memory and I want DomU to be able to directly write into that memory, thus implementing zero copying

Quoted 3 lines

Do they need to be contiguous in host physical memory, or guest physical memory?

Host

Quoted 4 lines

If it's in guest memory space, isn't there any generic interface that you can use?

Quoted 3 lines

If it's in host physical memory space, why do you need this buffer to be contiguous in host physical memory space? The IOMMU should hide all this.

There are drivers/HW which can only work with contig memory and if it is backed by an IOMMU then still it has to be contig in IPA space (real device doesn't know that it is actually IPA contig, not PA)

-----Original Message-----
From: Xen-devel [mailto:[email protected]] On Behalf Of Roger Pau Monné
Sent: 18 April 2018 11:11
To: Oleksandr Andrushchenko <[email protected]>
Cc: [email protected]; Artem Mygaiev <[email protected]>; Dongwon Kim <[email protected]>; [email protected]; [email protected]; [email protected]; dri- [email protected]; Potrola, MateuszX <[email protected]>; [email protected]; [email protected]; [email protected]; Matt Roper <[email protected]>
Subject: Re: [Xen-devel] [PATCH 0/1] drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 2 lines · Oleksandr Andrushchenko

Quoted 1 line · Roger Pau Monné

On Wed, Apr 18, 2018 at 09:38:39AM +0300, Oleksandr Andrushchenko

Quoted 6 lines · wrote:

Quoted 6 lines

Quoted 6 lines · Dongwon Kim

Quoted 2 lines · Daniel Vetter

Quoted 1 line · Dongwon Kim

3.2 Backend exports dma-buf to xen-front

In this case Dom0 pages are shared with DomU. As before, DomU can

only write

Quoted 1 line

Quoted 1 line

Quoted 1 line

to these pages, not any other page from Dom0, so it can be still

considered

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

safe.
But, the following must be considered (highlighted in xen-front's Kernel documentation):
 - If guest domain dies then pages/grants received from the backend

cannot

Quoted 1 line

Quoted 1 line

Quoted 1 line

   be claimed back - think of it as memory lost to Dom0 (won't be used

for

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

any
   other guest)
 - Misbehaving guest may send too many requests to the backend

exhausting

Quoted 1 line

Quoted 1 line

Quoted 1 line

   its grant references and memory (consider this from security POV).

As the

Quoted 1 line

Quoted 1 line

Quoted 1 line

   backend runs in the trusted domain we also assume that it is trusted

as

Quoted 12 lines

Quoted 9 lines

Quoted 2 lines

well,
   e.g. must take measures to prevent DDoS attacks.

I cannot parse the above sentence:

"As the backend runs in the trusted domain we also assume that it is trusted as well, e.g. must take measures to prevent DDoS attacks."

What's the relation between being trusted and protecting from DoS attacks?

I mean that we trust the backend that it can prevent Dom0 from crashing in case DomU's frontend misbehaves, e.g. if the frontend sends too many memory requests etc.

Quoted 7 lines

Quoted 3 lines

In any case, all? PV protocols are implemented with the frontend sharing pages to the backend, and I think there's a reason why this model is used, and it should continue to be used.

This is the first use-case above. But there are real-world use-cases (embedded in my case) when physically contiguous memory needs to be shared, one of the possible ways to achieve this is to share contiguous memory from Dom0 to DomU (the second use-case

above)

Quoted 8 lines

Quoted 6 lines

Having to add logic in the backend to prevent such attacks means that:

- We need more code in the backend, which increases complexity and chances of bugs.
- Such code/logic could be wrong, thus allowing DoS.

You can live without this code at all, but this is then up to backend which may make Dom0 down because of DomU's frontend doing

evil

Quoted 1 line

things

IMO we should design protocols that do not allow such attacks instead of having to defend against them.

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

4. xen-front/backend/xen-zcopy synchronization

4.1. As I already said in 2) all the inter VM communication happens

between

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

xen-front and the backend, xen-zcopy is NOT involved in that. When xen-front wants to destroy a display buffer (dumb/dma-buf) it

issues a

Quoted 1 line

Quoted 1 line

Quoted 1 line

XENDISPL_OP_DBUF_DESTROY command (opposite to

XENDISPL_OP_DBUF_CREATE).

Quoted 1 line

Quoted 1 line

Quoted 1 line

This call is synchronous, so xen-front expects that backend does free

the

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

buffer pages on return.

4.2. Backend, on XENDISPL_OP_DBUF_DESTROY:
  - closes all dumb handles/fd's of the buffer according to [3]
  - issues DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE IOCTL to xen-

zcopy to make

Quoted 9 lines

Quoted 5 lines

Quoted 3 lines

sure
    the buffer is freed (think of it as it waits for dma-buf->release
callback)

So this zcopy thing keeps some kind of track of the memory usage? Why can't the user-space backend keep track of the buffer usage?

Because there is no dma-buf UAPI which allows to track the buffer life cycle (e.g. wait until dma-buf's .release callback is called)

Quoted 2 lines

Quoted 2 lines

  - replies to xen-front that the buffer can be destroyed.
This way deletion of the buffer happens synchronously on both Dom0

and DomU

Quoted 1 line

Quoted 1 line

Quoted 1 line

sides. In case if DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE returns

with time-out

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

error
(BTW, wait time is a parameter of this IOCTL), Xen will defer grant reference
removal and will retry later until those are free.

Hope this helps understand how buffers are synchronously deleted in

case

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

of xen-zcopy with a single protocol command.

I think the above logic can also be re-used by the hyper-dmabuf driver

with

Quoted 9 lines

Quoted 9 lines

Quoted 5 lines

some additional work:

1. xen-zcopy can be split into 2 parts and extend:
1.1. Xen gntdev driver [4], [5] to allow creating dma-buf from grefs and vise versa,

I don't know much about the dma-buf implementation in Linux, but gntdev is a user-space device, and AFAICT user-space applications don't have any notion of dma buffers. How are such buffers useful for user-space? Why can't this just be called memory?

Quoted 9 lines

A dma-buf is seen by user-space as a file descriptor and you can pass it to different drivers then. For example, you can share a buffer used by a display driver for scanout with a GPU, to compose a picture into it:
1. User-space (US) allocates a display buffer from display driver
2. US asks display driver to export the dma-buf which backs up that buffer, US gets buffer's fd: dma_buf_fd
3. US asks GPU driver to import a buffer and provides it with dma_buf_fd
4. GPU renders contents into display buffer (dma_buf_fd)

After speaking with Oleksandr on IRC, I think the main usage of the gntdev extension is to:

1. Create a dma-buf from a set of grant references.
2. Share dma-buf and get a list of grant references.

I think this set of operations could be broken into:

1.1 Map grant references into user-space using the gntdev. 1.2 Create a dma-buf out of a set of user-space virtual addresses.

2.1 Map a dma-buf into user-space.
2.2 Get grefs out of the user-space addresses where the dma-buf is
mapped.

So it seems like what's actually missing is a way to:

- Create a dma-buf from a list of user-space virtual addresses.
- Allow to map a dma-buf into user-space, so it can then be used with the gntdev.

I think this is generic enough that it could be implemented by a device not tied to Xen. AFAICT the hyper_dma guys also wanted something similar to this.

Quoted 6 lines

Finally, this is indeed some memory, but a bit more [1]

Quoted 5 lines

Also, (with my FreeBSD maintainer hat) how is this going to translate to other OSes? So far the operations performed by the gntdev device are mostly OS-agnostic because this just map/unmap memory, and in fact they are implemented by Linux and FreeBSD.

Quoted 9 lines

At the moment I can only see Linux implementation and it seems to be perfectly ok as we do not change Xen's APIs etc. and only use the existing ones (remember, we only extend gntdev/balloon drivers, all the changes in the Linux kernel)
As the second note I can also think that we do not extend gntdev/balloon drivers and have re-worked xen-zcopy driver be a separate entity, say drivers/xen/dma-buf

Quoted 2 lines

Quoted 2 lines

implement "wait" ioctl (wait for dma-buf->release): currently these are DRM_XEN_ZCOPY_DUMB_FROM_REFS,

DRM_XEN_ZCOPY_DUMB_TO_REFS and

Quoted 6 lines

Quoted 6 lines

Quoted 4 lines

DRM_XEN_ZCOPY_DUMB_WAIT_FREE
1.2. Xen balloon driver [6] to allow allocating contiguous buffers (not needed
by current hyper-dmabuf, but is a must for xen-zcopy use-cases)

I think this needs clarifying. In which memory space do you need those regions to be contiguous?

Quoted 13 lines

Use-case: Dom0 has a HW driver which only works with contig memory and I want DomU to be able to directly write into that memory, thus implementing zero copying

Quoted 3 lines

Do they need to be contiguous in host physical memory, or guest physical memory?

Host

Quoted 6 lines

If it's in guest memory space, isn't there any generic interface that you can use?

If it's in host physical memory space, why do you need this buffer to be contiguous in host physical memory space? The IOMMU should hide

all

Quoted 4 lines

Quoted 1 line

this.

There are drivers/HW which can only work with contig memory and if it is backed by an IOMMU then still it has to be contig in IPA space (real device doesn't know that it is actually IPA contig, not PA)

What's IPA contig?

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Quoted 192 lines

-----Original Message-----
From: Xen-devel [mailto:[email protected]] On Behalf Of Roger Pau Monné
Sent: 18 April 2018 11:11
To: Oleksandr Andrushchenko <[email protected]>
Cc: [email protected]; Artem Mygaiev <[email protected]>; Dongwon Kim <[email protected]>; [email protected]; [email protected]; [email protected]; dri- [email protected]; Potrola, MateuszX <[email protected]>; [email protected]; [email protected]; [email protected]; Matt Roper <[email protected]>
Subject: Re: [Xen-devel] [PATCH 0/1] drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 2 lines · Oleksandr Andrushchenko

Quoted 1 line · Roger Pau Monné

On Wed, Apr 18, 2018 at 09:38:39AM +0300, Oleksandr Andrushchenko

Quoted 6 lines · wrote:

Quoted 6 lines

Quoted 6 lines · Dongwon Kim

Quoted 2 lines · Daniel Vetter

Quoted 1 line · Dongwon Kim

3.2 Backend exports dma-buf to xen-front

In this case Dom0 pages are shared with DomU. As before, DomU can

only write

Quoted 1 line

Quoted 1 line

Quoted 1 line

to these pages, not any other page from Dom0, so it can be still

considered

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

safe.
But, the following must be considered (highlighted in xen-front's Kernel documentation):
 - If guest domain dies then pages/grants received from the backend

cannot

Quoted 1 line

Quoted 1 line

Quoted 1 line

   be claimed back - think of it as memory lost to Dom0 (won't be used

for

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

any
   other guest)
 - Misbehaving guest may send too many requests to the backend

exhausting

Quoted 1 line

Quoted 1 line

Quoted 1 line

   its grant references and memory (consider this from security POV).

As the

Quoted 1 line

Quoted 1 line

Quoted 1 line

   backend runs in the trusted domain we also assume that it is trusted

as

Quoted 12 lines

Quoted 9 lines

Quoted 2 lines

well,
   e.g. must take measures to prevent DDoS attacks.

I cannot parse the above sentence:

"As the backend runs in the trusted domain we also assume that it is trusted as well, e.g. must take measures to prevent DDoS attacks."

What's the relation between being trusted and protecting from DoS attacks?

I mean that we trust the backend that it can prevent Dom0 from crashing in case DomU's frontend misbehaves, e.g. if the frontend sends too many memory requests etc.

Quoted 7 lines

Quoted 3 lines

In any case, all? PV protocols are implemented with the frontend sharing pages to the backend, and I think there's a reason why this model is used, and it should continue to be used.

This is the first use-case above. But there are real-world use-cases (embedded in my case) when physically contiguous memory needs to be shared, one of the possible ways to achieve this is to share contiguous memory from Dom0 to DomU (the second use-case

above)

Quoted 8 lines

Quoted 6 lines

Having to add logic in the backend to prevent such attacks means that:

- We need more code in the backend, which increases complexity and chances of bugs.
- Such code/logic could be wrong, thus allowing DoS.

You can live without this code at all, but this is then up to backend which may make Dom0 down because of DomU's frontend doing

evil

Quoted 1 line

things

IMO we should design protocols that do not allow such attacks instead of having to defend against them.

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

4. xen-front/backend/xen-zcopy synchronization

4.1. As I already said in 2) all the inter VM communication happens

between

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

xen-front and the backend, xen-zcopy is NOT involved in that. When xen-front wants to destroy a display buffer (dumb/dma-buf) it

issues a

Quoted 1 line

Quoted 1 line

Quoted 1 line

XENDISPL_OP_DBUF_DESTROY command (opposite to

XENDISPL_OP_DBUF_CREATE).

Quoted 1 line

Quoted 1 line

Quoted 1 line

This call is synchronous, so xen-front expects that backend does free

the

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

buffer pages on return.

4.2. Backend, on XENDISPL_OP_DBUF_DESTROY:
  - closes all dumb handles/fd's of the buffer according to [3]
  - issues DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE IOCTL to xen-

zcopy to make

Quoted 9 lines

Quoted 5 lines

Quoted 3 lines

sure
    the buffer is freed (think of it as it waits for dma-buf->release
callback)

So this zcopy thing keeps some kind of track of the memory usage? Why can't the user-space backend keep track of the buffer usage?

Because there is no dma-buf UAPI which allows to track the buffer life cycle (e.g. wait until dma-buf's .release callback is called)

Quoted 2 lines

Quoted 2 lines

  - replies to xen-front that the buffer can be destroyed.
This way deletion of the buffer happens synchronously on both Dom0

and DomU

Quoted 1 line

Quoted 1 line

Quoted 1 line

sides. In case if DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE returns

with time-out

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

error
(BTW, wait time is a parameter of this IOCTL), Xen will defer grant reference
removal and will retry later until those are free.

Hope this helps understand how buffers are synchronously deleted in

case

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

of xen-zcopy with a single protocol command.

I think the above logic can also be re-used by the hyper-dmabuf driver

with

Quoted 9 lines

Quoted 9 lines

Quoted 5 lines

some additional work:

1. xen-zcopy can be split into 2 parts and extend:
1.1. Xen gntdev driver [4], [5] to allow creating dma-buf from grefs and vise versa,

I don't know much about the dma-buf implementation in Linux, but gntdev is a user-space device, and AFAICT user-space applications don't have any notion of dma buffers. How are such buffers useful for user-space? Why can't this just be called memory?

Quoted 9 lines

A dma-buf is seen by user-space as a file descriptor and you can pass it to different drivers then. For example, you can share a buffer used by a display driver for scanout with a GPU, to compose a picture into it:
1. User-space (US) allocates a display buffer from display driver
2. US asks display driver to export the dma-buf which backs up that buffer, US gets buffer's fd: dma_buf_fd
3. US asks GPU driver to import a buffer and provides it with dma_buf_fd
4. GPU renders contents into display buffer (dma_buf_fd)

After speaking with Oleksandr on IRC, I think the main usage of the gntdev extension is to:

1. Create a dma-buf from a set of grant references.
2. Share dma-buf and get a list of grant references.

I think this set of operations could be broken into:

1.1 Map grant references into user-space using the gntdev. 1.2 Create a dma-buf out of a set of user-space virtual addresses.

2.1 Map a dma-buf into user-space.
2.2 Get grefs out of the user-space addresses where the dma-buf is
mapped.

So it seems like what's actually missing is a way to:

- Create a dma-buf from a list of user-space virtual addresses.
- Allow to map a dma-buf into user-space, so it can then be used with the gntdev.

I think this is generic enough that it could be implemented by a device not tied to Xen. AFAICT the hyper_dma guys also wanted something similar to this.

Quoted 5 lines

Finally, this is indeed some memory, but a bit more [1]

Quoted 4 lines

Also, (with my FreeBSD maintainer hat) how is this going to translate to other OSes? So far the operations performed by the gntdev device are mostly OS-agnostic because this just map/unmap memory, and in fact they are implemented by Linux and FreeBSD.

Quoted 9 lines

At the moment I can only see Linux implementation and it seems to be perfectly ok as we do not change Xen's APIs etc. and only use the existing ones (remember, we only extend gntdev/balloon drivers, all the changes in the Linux kernel)
As the second note I can also think that we do not extend gntdev/balloon drivers and have re-worked xen-zcopy driver be a separate entity, say drivers/xen/dma-buf

Quoted 2 lines

Quoted 2 lines

implement "wait" ioctl (wait for dma-buf->release): currently these are DRM_XEN_ZCOPY_DUMB_FROM_REFS,

DRM_XEN_ZCOPY_DUMB_TO_REFS and

Quoted 17 lines

Quoted 6 lines

Quoted 4 lines

DRM_XEN_ZCOPY_DUMB_WAIT_FREE
1.2. Xen balloon driver [6] to allow allocating contiguous buffers (not needed
by current hyper-dmabuf, but is a must for xen-zcopy use-cases)

I think this needs clarifying. In which memory space do you need those regions to be contiguous?

Use-case: Dom0 has a HW driver which only works with contig memory and I want DomU to be able to directly write into that memory, thus implementing zero copying

Quoted 2 lines

Do they need to be contiguous in host physical memory, or guest physical memory?

Host

Quoted 5 lines

If it's in guest memory space, isn't there any generic interface that you can use?

If it's in host physical memory space, why do you need this buffer to be contiguous in host physical memory space? The IOMMU should hide

Quoted 6 lines

all

Quoted 4 lines

Quoted 1 line

this.

There are drivers/HW which can only work with contig memory and if it is backed by an IOMMU then still it has to be contig in IPA space (real device doesn't know that it is actually IPA contig, not PA)

What's IPA contig?

I assume 'IPA' means 'IOMMU Physical Address'. I wonder whether this means what I've termed 'Bus Address' elsewhere?

Paul

Quoted 6 lines

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

-----Original Message-----
From: Oleksandr Andrushchenko [mailto:[email protected]]
Sent: 18 April 2018 11:21
To: Paul Durrant <[email protected]>; Roger Pau Monne <[email protected]>
Cc: [email protected]; Artem Mygaiev <[email protected]>; Dongwon Kim <[email protected]>; [email protected]; [email protected]; [email protected]; dri- [email protected]; Potrola, MateuszX <[email protected]>; [email protected]; [email protected]; [email protected]; Matt Roper <[email protected]>
Subject: Re: [Xen-devel] [PATCH 0/1] drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 2 lines · Paul Durrant

Quoted 2 lines

-----Original Message-----
From: Xen-devel [mailto:[email protected]] On

Behalf

Quoted 6 lines

Quoted 6 lines

Of Roger Pau Monné
Sent: 18 April 2018 11:11
To: Oleksandr Andrushchenko <[email protected]>
Cc: [email protected]; Artem Mygaiev <[email protected]>; Dongwon Kim <[email protected]>; [email protected]; [email protected]; [email protected];

dri-

Quoted 15 lines

Quoted 15 lines

[email protected]; Potrola, MateuszX <[email protected]>; [email protected]; [email protected]; [email protected]; Matt Roper <[email protected]>
Subject: Re: [Xen-devel] [PATCH 0/1] drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 2 lines · Oleksandr Andrushchenko

Quoted 1 line · Roger Pau Monné

On Wed, Apr 18, 2018 at 09:38:39AM +0300, Oleksandr Andrushchenko

Quoted 3 lines · wrote:

Quoted 3 lines

Quoted 3 lines · Dongwon Kim

Quoted 2 lines · Daniel Vetter

Quoted 1 line · Dongwon Kim

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

3.2 Backend exports dma-buf to xen-front

In this case Dom0 pages are shared with DomU. As before, DomU can

only write

Quoted 1 line

Quoted 1 line

Quoted 1 line

to these pages, not any other page from Dom0, so it can be still

considered

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

safe.
But, the following must be considered (highlighted in xen-front's

Kernel

Quoted 4 lines

Quoted 4 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

documentation):
 - If guest domain dies then pages/grants received from the backend

cannot

Quoted 1 line

Quoted 1 line

Quoted 1 line

   be claimed back - think of it as memory lost to Dom0 (won't be used

Quoted 8 lines

Quoted 8 lines

for

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

any
   other guest)
 - Misbehaving guest may send too many requests to the backend

exhausting

Quoted 1 line

Quoted 1 line

Quoted 1 line

   its grant references and memory (consider this from security POV).

As the

Quoted 1 line

Quoted 1 line

Quoted 1 line

   backend runs in the trusted domain we also assume that it is

trusted

Quoted 21 lines

Quoted 21 lines

as

Quoted 12 lines

Quoted 9 lines

Quoted 2 lines

well,
   e.g. must take measures to prevent DDoS attacks.

I cannot parse the above sentence:

"As the backend runs in the trusted domain we also assume that it is trusted as well, e.g. must take measures to prevent DDoS attacks."

What's the relation between being trusted and protecting from DoS attacks?

I mean that we trust the backend that it can prevent Dom0 from crashing in case DomU's frontend misbehaves, e.g. if the frontend sends too many memory requests etc.

Quoted 7 lines

Quoted 3 lines

In any case, all? PV protocols are implemented with the frontend sharing pages to the backend, and I think there's a reason why this model is used, and it should continue to be used.

This is the first use-case above. But there are real-world use-cases (embedded in my case) when physically contiguous memory needs to be shared, one of the possible ways to achieve this is to share contiguous memory from Dom0 to DomU (the second use-case

above)

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 6 lines

Having to add logic in the backend to prevent such attacks means that:

- We need more code in the backend, which increases complexity and chances of bugs.
- Such code/logic could be wrong, thus allowing DoS.

You can live without this code at all, but this is then up to backend which may make Dom0 down because of DomU's frontend

doing

Quoted 18 lines

Quoted 18 lines

evil

Quoted 1 line

things

IMO we should design protocols that do not allow such attacks instead of having to defend against them.

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

4. xen-front/backend/xen-zcopy synchronization

4.1. As I already said in 2) all the inter VM communication happens

between

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

xen-front and the backend, xen-zcopy is NOT involved in that. When xen-front wants to destroy a display buffer (dumb/dma-buf) it

issues a

Quoted 1 line

Quoted 1 line

Quoted 1 line

XENDISPL_OP_DBUF_DESTROY command (opposite to

XENDISPL_OP_DBUF_CREATE).

Quoted 1 line

Quoted 1 line

Quoted 1 line

This call is synchronous, so xen-front expects that backend does free

the

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

buffer pages on return.

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

4.2. Backend, on XENDISPL_OP_DBUF_DESTROY:
  - closes all dumb handles/fd's of the buffer according to [3]
  - issues DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE IOCTL to xen-

zcopy to make

Quoted 4 lines

Quoted 4 lines

Quoted 3 lines

sure
    the buffer is freed (think of it as it waits for dma-buf->release
callback)

So this zcopy thing keeps some kind of track of the memory usage?

Why

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 1 line

can't the user-space backend keep track of the buffer usage?

Because there is no dma-buf UAPI which allows to track the buffer life

cycle

Quoted 13 lines

Quoted 13 lines

Quoted 3 lines

(e.g. wait until dma-buf's .release callback is called)

Quoted 2 lines

Quoted 2 lines

  - replies to xen-front that the buffer can be destroyed.
This way deletion of the buffer happens synchronously on both Dom0

and DomU

Quoted 1 line

Quoted 1 line

Quoted 1 line

sides. In case if DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE returns

with time-out

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

error
(BTW, wait time is a parameter of this IOCTL), Xen will defer grant reference
removal and will retry later until those are free.

Hope this helps understand how buffers are synchronously deleted in

case

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

of xen-zcopy with a single protocol command.

I think the above logic can also be re-used by the hyper-dmabuf driver

with

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

some additional work:

1. xen-zcopy can be split into 2 parts and extend:
1.1. Xen gntdev driver [4], [5] to allow creating dma-buf from grefs

and

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

Quoted 5 lines

Quoted 1 line

vise versa,

I don't know much about the dma-buf implementation in Linux, but gntdev is a user-space device, and AFAICT user-space applications don't have any notion of dma buffers. How are such buffers useful for user-space? Why can't this just be called memory?

A dma-buf is seen by user-space as a file descriptor and you can pass it to different drivers then. For example, you can share a buffer used by a display driver for scanout with a GPU, to compose a picture into it:
1. User-space (US) allocates a display buffer from display driver
2. US asks display driver to export the dma-buf which backs up that

buffer,

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

US gets buffer's fd: dma_buf_fd
3. US asks GPU driver to import a buffer and provides it with

dma_buf_fd

Quoted 22 lines

Quoted 22 lines

Quoted 1 line

4. GPU renders contents into display buffer (dma_buf_fd)

After speaking with Oleksandr on IRC, I think the main usage of the gntdev extension is to:

1. Create a dma-buf from a set of grant references.
2. Share dma-buf and get a list of grant references.

I think this set of operations could be broken into:

1.1 Map grant references into user-space using the gntdev. 1.2 Create a dma-buf out of a set of user-space virtual addresses.

2.1 Map a dma-buf into user-space.
2.2 Get grefs out of the user-space addresses where the dma-buf is
mapped.

So it seems like what's actually missing is a way to:

- Create a dma-buf from a list of user-space virtual addresses.
- Allow to map a dma-buf into user-space, so it can then be used with the gntdev.

Quoted 8 lines

Quoted 8 lines

I think this is generic enough that it could be implemented by a device not tied to Xen. AFAICT the hyper_dma guys also wanted something similar to this.

Quoted 4 lines

Finally, this is indeed some memory, but a bit more [1]

Quoted 3 lines

Also, (with my FreeBSD maintainer hat) how is this going to translate to other OSes? So far the operations performed by the gntdev device are mostly OS-agnostic because this just map/unmap memory, and in

fact

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 1 line

they are implemented by Linux and FreeBSD.

At the moment I can only see Linux implementation and it seems to be perfectly ok as we do not change Xen's APIs etc. and only use the existing ones (remember, we only extend gntdev/balloon drivers, all the changes in the Linux kernel)
As the second note I can also think that we do not extend

gntdev/balloon

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

drivers and have re-worked xen-zcopy driver be a separate entity, say drivers/xen/dma-buf

Quoted 1 line

Quoted 1 line

implement "wait" ioctl (wait for dma-buf->release): currently these

are

Quoted 24 lines

Quoted 24 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

DRM_XEN_ZCOPY_DUMB_FROM_REFS,

DRM_XEN_ZCOPY_DUMB_TO_REFS and

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

DRM_XEN_ZCOPY_DUMB_WAIT_FREE
1.2. Xen balloon driver [6] to allow allocating contiguous buffers (not needed
by current hyper-dmabuf, but is a must for xen-zcopy use-cases)

Quoted 13 lines

Quoted 2 lines

I think this needs clarifying. In which memory space do you need those regions to be contiguous?

Use-case: Dom0 has a HW driver which only works with contig memory and I want DomU to be able to directly write into that memory, thus implementing zero copying

Quoted 2 lines

Do they need to be contiguous in host physical memory, or guest physical memory?

Host

Quoted 5 lines

If it's in guest memory space, isn't there any generic interface that you can use?

If it's in host physical memory space, why do you need this buffer to be contiguous in host physical memory space? The IOMMU should hide

all

Quoted 4 lines

Quoted 1 line

this.

There are drivers/HW which can only work with contig memory and if it is backed by an IOMMU then still it has to be contig in IPA space (real device doesn't know that it is actually IPA contig, not PA)

Quoted 2 lines

Quoted 1 line

What's IPA contig?

I assume 'IPA' means 'IOMMU Physical Address'. I wonder whether this

means what I've termed 'Bus Address' elsewhere?
sorry for not being clear here: I mean that the device sees contiguous range of
Intermediate Phys Addresses

Quoted 8 lines

Paul

Quoted 6 lines

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Quoted 226 lines

-----Original Message-----
From: Oleksandr Andrushchenko [mailto:[email protected]]
Sent: 18 April 2018 11:21
To: Paul Durrant <[email protected]>; Roger Pau Monne <[email protected]>
Cc: [email protected]; Artem Mygaiev <[email protected]>; Dongwon Kim <[email protected]>; [email protected]; [email protected]; [email protected]; dri- [email protected]; Potrola, MateuszX <[email protected]>; [email protected]; [email protected]; [email protected]; Matt Roper <[email protected]>
Subject: Re: [Xen-devel] [PATCH 0/1] drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 2 lines · Paul Durrant

Quoted 2 lines

-----Original Message-----
From: Xen-devel [mailto:[email protected]] On

Behalf

Quoted 6 lines

Quoted 6 lines

Of Roger Pau Monné
Sent: 18 April 2018 11:11
To: Oleksandr Andrushchenko <[email protected]>
Cc: [email protected]; Artem Mygaiev <[email protected]>; Dongwon Kim <[email protected]>; [email protected]; [email protected]; [email protected];

dri-

Quoted 15 lines

Quoted 15 lines

[email protected]; Potrola, MateuszX <[email protected]>; [email protected]; [email protected]; [email protected]; Matt Roper <[email protected]>
Subject: Re: [Xen-devel] [PATCH 0/1] drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 2 lines · Oleksandr Andrushchenko

Quoted 1 line · Roger Pau Monné

On Wed, Apr 18, 2018 at 09:38:39AM +0300, Oleksandr Andrushchenko

Quoted 3 lines · wrote:

Quoted 3 lines

Quoted 3 lines · Dongwon Kim

Quoted 2 lines · Daniel Vetter

Quoted 1 line · Dongwon Kim

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

3.2 Backend exports dma-buf to xen-front

In this case Dom0 pages are shared with DomU. As before, DomU can

only write

Quoted 1 line

Quoted 1 line

Quoted 1 line

to these pages, not any other page from Dom0, so it can be still

considered

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

safe.
But, the following must be considered (highlighted in xen-front's

Kernel

Quoted 4 lines

Quoted 4 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

documentation):
 - If guest domain dies then pages/grants received from the backend

cannot

Quoted 1 line

Quoted 1 line

Quoted 1 line

   be claimed back - think of it as memory lost to Dom0 (won't be used

Quoted 8 lines

Quoted 8 lines

for

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

any
   other guest)
 - Misbehaving guest may send too many requests to the backend

exhausting

Quoted 1 line

Quoted 1 line

Quoted 1 line

   its grant references and memory (consider this from security POV).

As the

Quoted 1 line

Quoted 1 line

Quoted 1 line

   backend runs in the trusted domain we also assume that it is

trusted

Quoted 21 lines

Quoted 21 lines

as

Quoted 12 lines

Quoted 9 lines

Quoted 2 lines

well,
   e.g. must take measures to prevent DDoS attacks.

I cannot parse the above sentence:

"As the backend runs in the trusted domain we also assume that it is trusted as well, e.g. must take measures to prevent DDoS attacks."

What's the relation between being trusted and protecting from DoS attacks?

I mean that we trust the backend that it can prevent Dom0 from crashing in case DomU's frontend misbehaves, e.g. if the frontend sends too many memory requests etc.

Quoted 7 lines

Quoted 3 lines

In any case, all? PV protocols are implemented with the frontend sharing pages to the backend, and I think there's a reason why this model is used, and it should continue to be used.

This is the first use-case above. But there are real-world use-cases (embedded in my case) when physically contiguous memory needs to be shared, one of the possible ways to achieve this is to share contiguous memory from Dom0 to DomU (the second use-case

above)

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 6 lines

Having to add logic in the backend to prevent such attacks means that:

- We need more code in the backend, which increases complexity and chances of bugs.
- Such code/logic could be wrong, thus allowing DoS.

You can live without this code at all, but this is then up to backend which may make Dom0 down because of DomU's frontend

doing

Quoted 18 lines

Quoted 18 lines

evil

Quoted 1 line

things

IMO we should design protocols that do not allow such attacks instead of having to defend against them.

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

4. xen-front/backend/xen-zcopy synchronization

4.1. As I already said in 2) all the inter VM communication happens

between

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

xen-front and the backend, xen-zcopy is NOT involved in that. When xen-front wants to destroy a display buffer (dumb/dma-buf) it

issues a

Quoted 1 line

Quoted 1 line

Quoted 1 line

XENDISPL_OP_DBUF_DESTROY command (opposite to

XENDISPL_OP_DBUF_CREATE).

Quoted 1 line

Quoted 1 line

Quoted 1 line

This call is synchronous, so xen-front expects that backend does free

the

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

buffer pages on return.

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

4.2. Backend, on XENDISPL_OP_DBUF_DESTROY:
  - closes all dumb handles/fd's of the buffer according to [3]
  - issues DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE IOCTL to xen-

zcopy to make

Quoted 4 lines

Quoted 4 lines

Quoted 3 lines

sure
    the buffer is freed (think of it as it waits for dma-buf->release
callback)

So this zcopy thing keeps some kind of track of the memory usage?

Why

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 1 line

can't the user-space backend keep track of the buffer usage?

Because there is no dma-buf UAPI which allows to track the buffer life

cycle

Quoted 13 lines

Quoted 13 lines

Quoted 3 lines

(e.g. wait until dma-buf's .release callback is called)

Quoted 2 lines

Quoted 2 lines

  - replies to xen-front that the buffer can be destroyed.
This way deletion of the buffer happens synchronously on both Dom0

and DomU

Quoted 1 line

Quoted 1 line

Quoted 1 line

sides. In case if DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE returns

with time-out

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

error
(BTW, wait time is a parameter of this IOCTL), Xen will defer grant reference
removal and will retry later until those are free.

Hope this helps understand how buffers are synchronously deleted in

case

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

of xen-zcopy with a single protocol command.

I think the above logic can also be re-used by the hyper-dmabuf driver

with

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

some additional work:

1. xen-zcopy can be split into 2 parts and extend:
1.1. Xen gntdev driver [4], [5] to allow creating dma-buf from grefs

and

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

Quoted 5 lines

Quoted 1 line

vise versa,

I don't know much about the dma-buf implementation in Linux, but gntdev is a user-space device, and AFAICT user-space applications don't have any notion of dma buffers. How are such buffers useful for user-space? Why can't this just be called memory?

A dma-buf is seen by user-space as a file descriptor and you can pass it to different drivers then. For example, you can share a buffer used by a display driver for scanout with a GPU, to compose a picture into it:
1. User-space (US) allocates a display buffer from display driver
2. US asks display driver to export the dma-buf which backs up that

buffer,

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

US gets buffer's fd: dma_buf_fd
3. US asks GPU driver to import a buffer and provides it with

dma_buf_fd

Quoted 22 lines

Quoted 22 lines

Quoted 1 line

4. GPU renders contents into display buffer (dma_buf_fd)

After speaking with Oleksandr on IRC, I think the main usage of the gntdev extension is to:

1. Create a dma-buf from a set of grant references.
2. Share dma-buf and get a list of grant references.

I think this set of operations could be broken into:

1.1 Map grant references into user-space using the gntdev. 1.2 Create a dma-buf out of a set of user-space virtual addresses.

2.1 Map a dma-buf into user-space.
2.2 Get grefs out of the user-space addresses where the dma-buf is
mapped.

So it seems like what's actually missing is a way to:

- Create a dma-buf from a list of user-space virtual addresses.
- Allow to map a dma-buf into user-space, so it can then be used with the gntdev.

Quoted 8 lines

Quoted 8 lines

I think this is generic enough that it could be implemented by a device not tied to Xen. AFAICT the hyper_dma guys also wanted something similar to this.

Quoted 4 lines

Finally, this is indeed some memory, but a bit more [1]

Quoted 3 lines

Also, (with my FreeBSD maintainer hat) how is this going to translate to other OSes? So far the operations performed by the gntdev device are mostly OS-agnostic because this just map/unmap memory, and in

fact

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 1 line

they are implemented by Linux and FreeBSD.

At the moment I can only see Linux implementation and it seems to be perfectly ok as we do not change Xen's APIs etc. and only use the existing ones (remember, we only extend gntdev/balloon drivers, all the changes in the Linux kernel)
As the second note I can also think that we do not extend

gntdev/balloon

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

drivers and have re-worked xen-zcopy driver be a separate entity, say drivers/xen/dma-buf

Quoted 1 line

Quoted 1 line

implement "wait" ioctl (wait for dma-buf->release): currently these

are

Quoted 24 lines

Quoted 24 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

DRM_XEN_ZCOPY_DUMB_FROM_REFS,

DRM_XEN_ZCOPY_DUMB_TO_REFS and

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

DRM_XEN_ZCOPY_DUMB_WAIT_FREE
1.2. Xen balloon driver [6] to allow allocating contiguous buffers (not needed
by current hyper-dmabuf, but is a must for xen-zcopy use-cases)

Quoted 13 lines

Quoted 2 lines

I think this needs clarifying. In which memory space do you need those regions to be contiguous?

Use-case: Dom0 has a HW driver which only works with contig memory and I want DomU to be able to directly write into that memory, thus implementing zero copying

Quoted 2 lines

Do they need to be contiguous in host physical memory, or guest physical memory?

Host

Quoted 5 lines

If it's in guest memory space, isn't there any generic interface that you can use?

If it's in host physical memory space, why do you need this buffer to be contiguous in host physical memory space? The IOMMU should hide

all

Quoted 4 lines

Quoted 1 line

this.

There are drivers/HW which can only work with contig memory and if it is backed by an IOMMU then still it has to be contig in IPA space (real device doesn't know that it is actually IPA contig, not PA)

Quoted 6 lines

Quoted 2 lines

Quoted 1 line

What's IPA contig?

I assume 'IPA' means 'IOMMU Physical Address'. I wonder whether this

means what I've termed 'Bus Address' elsewhere?
sorry for not being clear here: I mean that the device sees contiguous range of
Intermediate Phys Addresses

Still not clear (to me at least) what that means. Are you talking about the address space used by the device? If so, that is essentially virtual address space translated by the IOMMU and we have general termed this 'bus address space'.

Paul

Quoted 8 lines

Quoted 8 lines

Paul

Quoted 6 lines

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Quoted 226 lines

-----Original Message-----
From: Oleksandr Andrushchenko [mailto:[email protected]]
Sent: 18 April 2018 11:21
To: Paul Durrant <[email protected]>; Roger Pau Monne <[email protected]>
Cc: [email protected]; Artem Mygaiev <[email protected]>; Dongwon Kim <[email protected]>; [email protected]; [email protected]; [email protected]; dri- [email protected]; Potrola, MateuszX <[email protected]>; [email protected]; [email protected]; [email protected]; Matt Roper <[email protected]>
Subject: Re: [Xen-devel] [PATCH 0/1] drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 2 lines · Paul Durrant

Quoted 2 lines

-----Original Message-----
From: Xen-devel [mailto:[email protected]] On

Behalf

Quoted 6 lines

Quoted 6 lines

Of Roger Pau Monné
Sent: 18 April 2018 11:11
To: Oleksandr Andrushchenko <[email protected]>
Cc: [email protected]; Artem Mygaiev <[email protected]>; Dongwon Kim <[email protected]>; [email protected]; [email protected]; [email protected];

dri-

Quoted 15 lines

Quoted 15 lines

[email protected]; Potrola, MateuszX <[email protected]>; [email protected]; [email protected]; [email protected]; Matt Roper <[email protected]>
Subject: Re: [Xen-devel] [PATCH 0/1] drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 2 lines · Oleksandr Andrushchenko

Quoted 1 line · Roger Pau Monné

On Wed, Apr 18, 2018 at 09:38:39AM +0300, Oleksandr Andrushchenko

Quoted 3 lines · wrote:

Quoted 3 lines

Quoted 3 lines · Dongwon Kim

Quoted 2 lines · Daniel Vetter

Quoted 1 line · Dongwon Kim

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

3.2 Backend exports dma-buf to xen-front

In this case Dom0 pages are shared with DomU. As before, DomU can

only write

Quoted 1 line

Quoted 1 line

Quoted 1 line

to these pages, not any other page from Dom0, so it can be still

considered

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

safe.
But, the following must be considered (highlighted in xen-front's

Kernel

Quoted 4 lines

Quoted 4 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

documentation):
 - If guest domain dies then pages/grants received from the backend

cannot

Quoted 1 line

Quoted 1 line

Quoted 1 line

   be claimed back - think of it as memory lost to Dom0 (won't be used

Quoted 8 lines

Quoted 8 lines

for

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

any
   other guest)
 - Misbehaving guest may send too many requests to the backend

exhausting

Quoted 1 line

Quoted 1 line

Quoted 1 line

   its grant references and memory (consider this from security POV).

As the

Quoted 1 line

Quoted 1 line

Quoted 1 line

   backend runs in the trusted domain we also assume that it is

trusted

Quoted 21 lines

Quoted 21 lines

as

Quoted 12 lines

Quoted 9 lines

Quoted 2 lines

well,
   e.g. must take measures to prevent DDoS attacks.

I cannot parse the above sentence:

"As the backend runs in the trusted domain we also assume that it is trusted as well, e.g. must take measures to prevent DDoS attacks."

What's the relation between being trusted and protecting from DoS attacks?

I mean that we trust the backend that it can prevent Dom0 from crashing in case DomU's frontend misbehaves, e.g. if the frontend sends too many memory requests etc.

Quoted 7 lines

Quoted 3 lines

In any case, all? PV protocols are implemented with the frontend sharing pages to the backend, and I think there's a reason why this model is used, and it should continue to be used.

This is the first use-case above. But there are real-world use-cases (embedded in my case) when physically contiguous memory needs to be shared, one of the possible ways to achieve this is to share contiguous memory from Dom0 to DomU (the second use-case

above)

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 6 lines

Having to add logic in the backend to prevent such attacks means that:

- We need more code in the backend, which increases complexity and chances of bugs.
- Such code/logic could be wrong, thus allowing DoS.

You can live without this code at all, but this is then up to backend which may make Dom0 down because of DomU's frontend

doing

Quoted 18 lines

Quoted 18 lines

evil

Quoted 1 line

things

IMO we should design protocols that do not allow such attacks instead of having to defend against them.

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

4. xen-front/backend/xen-zcopy synchronization

4.1. As I already said in 2) all the inter VM communication happens

between

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

xen-front and the backend, xen-zcopy is NOT involved in that. When xen-front wants to destroy a display buffer (dumb/dma-buf) it

issues a

Quoted 1 line

Quoted 1 line

Quoted 1 line

XENDISPL_OP_DBUF_DESTROY command (opposite to

XENDISPL_OP_DBUF_CREATE).

Quoted 1 line

Quoted 1 line

Quoted 1 line

This call is synchronous, so xen-front expects that backend does free

the

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

buffer pages on return.

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

4.2. Backend, on XENDISPL_OP_DBUF_DESTROY:
  - closes all dumb handles/fd's of the buffer according to [3]
  - issues DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE IOCTL to xen-

zcopy to make

Quoted 4 lines

Quoted 4 lines

Quoted 3 lines

sure
    the buffer is freed (think of it as it waits for dma-buf->release
callback)

So this zcopy thing keeps some kind of track of the memory usage?

Why

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 1 line

can't the user-space backend keep track of the buffer usage?

Because there is no dma-buf UAPI which allows to track the buffer life

cycle

Quoted 13 lines

Quoted 13 lines

Quoted 3 lines

(e.g. wait until dma-buf's .release callback is called)

Quoted 2 lines

Quoted 2 lines

  - replies to xen-front that the buffer can be destroyed.
This way deletion of the buffer happens synchronously on both Dom0

and DomU

Quoted 1 line

Quoted 1 line

Quoted 1 line

sides. In case if DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE returns

with time-out

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

error
(BTW, wait time is a parameter of this IOCTL), Xen will defer grant reference
removal and will retry later until those are free.

Hope this helps understand how buffers are synchronously deleted in

case

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

of xen-zcopy with a single protocol command.

I think the above logic can also be re-used by the hyper-dmabuf driver

with

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

some additional work:

1. xen-zcopy can be split into 2 parts and extend:
1.1. Xen gntdev driver [4], [5] to allow creating dma-buf from grefs

and

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

Quoted 5 lines

Quoted 1 line

vise versa,

I don't know much about the dma-buf implementation in Linux, but gntdev is a user-space device, and AFAICT user-space applications don't have any notion of dma buffers. How are such buffers useful for user-space? Why can't this just be called memory?

A dma-buf is seen by user-space as a file descriptor and you can pass it to different drivers then. For example, you can share a buffer used by a display driver for scanout with a GPU, to compose a picture into it:
1. User-space (US) allocates a display buffer from display driver
2. US asks display driver to export the dma-buf which backs up that

buffer,

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

US gets buffer's fd: dma_buf_fd
3. US asks GPU driver to import a buffer and provides it with

dma_buf_fd

Quoted 22 lines

Quoted 22 lines

Quoted 1 line

4. GPU renders contents into display buffer (dma_buf_fd)

After speaking with Oleksandr on IRC, I think the main usage of the gntdev extension is to:

1. Create a dma-buf from a set of grant references.
2. Share dma-buf and get a list of grant references.

I think this set of operations could be broken into:

1.1 Map grant references into user-space using the gntdev. 1.2 Create a dma-buf out of a set of user-space virtual addresses.

2.1 Map a dma-buf into user-space.
2.2 Get grefs out of the user-space addresses where the dma-buf is
mapped.

So it seems like what's actually missing is a way to:

- Create a dma-buf from a list of user-space virtual addresses.
- Allow to map a dma-buf into user-space, so it can then be used with the gntdev.

Quoted 8 lines

Quoted 8 lines

I think this is generic enough that it could be implemented by a device not tied to Xen. AFAICT the hyper_dma guys also wanted something similar to this.

Quoted 4 lines

Finally, this is indeed some memory, but a bit more [1]

Quoted 3 lines

Also, (with my FreeBSD maintainer hat) how is this going to translate to other OSes? So far the operations performed by the gntdev device are mostly OS-agnostic because this just map/unmap memory, and in

fact

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 1 line

they are implemented by Linux and FreeBSD.

At the moment I can only see Linux implementation and it seems to be perfectly ok as we do not change Xen's APIs etc. and only use the existing ones (remember, we only extend gntdev/balloon drivers, all the changes in the Linux kernel)
As the second note I can also think that we do not extend

gntdev/balloon

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

drivers and have re-worked xen-zcopy driver be a separate entity, say drivers/xen/dma-buf

Quoted 1 line

Quoted 1 line

implement "wait" ioctl (wait for dma-buf->release): currently these

are

Quoted 24 lines

Quoted 24 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

DRM_XEN_ZCOPY_DUMB_FROM_REFS,

DRM_XEN_ZCOPY_DUMB_TO_REFS and

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

DRM_XEN_ZCOPY_DUMB_WAIT_FREE
1.2. Xen balloon driver [6] to allow allocating contiguous buffers (not needed
by current hyper-dmabuf, but is a must for xen-zcopy use-cases)

Quoted 13 lines

Quoted 2 lines

I think this needs clarifying. In which memory space do you need those regions to be contiguous?

Use-case: Dom0 has a HW driver which only works with contig memory and I want DomU to be able to directly write into that memory, thus implementing zero copying

Quoted 2 lines

Do they need to be contiguous in host physical memory, or guest physical memory?

Host

Quoted 5 lines

If it's in guest memory space, isn't there any generic interface that you can use?

If it's in host physical memory space, why do you need this buffer to be contiguous in host physical memory space? The IOMMU should hide

all

Quoted 4 lines

Quoted 1 line

this.

There are drivers/HW which can only work with contig memory and if it is backed by an IOMMU then still it has to be contig in IPA space (real device doesn't know that it is actually IPA contig, not PA)

Quoted 6 lines

Quoted 2 lines

Quoted 1 line

What's IPA contig?

I assume 'IPA' means 'IOMMU Physical Address'. I wonder whether this

means what I've termed 'Bus Address' elsewhere?
sorry for not being clear here: I mean that the device sees contiguous range of
Intermediate Phys Addresses

Still not clear (to me at least) what that means. Are you talking about the address space used by the device? If so, that is essentially virtual address space translated by the IOMMU and we have general termed this 'bus address space'.

Paul

Quoted 8 lines

Quoted 8 lines

Paul

Quoted 6 lines

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

-----Original Message-----
From: Oleksandr Andrushchenko [mailto:[email protected]]
Sent: 18 April 2018 11:21
To: Paul Durrant <[email protected]>; Roger Pau Monne <[email protected]>
Cc: [email protected]; Artem Mygaiev <[email protected]>; Dongwon Kim <[email protected]>; [email protected]; [email protected]; [email protected]; dri- [email protected]; Potrola, MateuszX <[email protected]>; [email protected]; [email protected]; [email protected]; Matt Roper <[email protected]>
Subject: Re: [Xen-devel] [PATCH 0/1] drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 2 lines · Paul Durrant

Quoted 2 lines

-----Original Message-----
From: Xen-devel [mailto:[email protected]] On

Behalf

Quoted 6 lines

Quoted 6 lines

Of Roger Pau Monné
Sent: 18 April 2018 11:11
To: Oleksandr Andrushchenko <[email protected]>
Cc: [email protected]; Artem Mygaiev <[email protected]>; Dongwon Kim <[email protected]>; [email protected]; [email protected]; [email protected];

dri-

Quoted 15 lines

Quoted 15 lines

[email protected]; Potrola, MateuszX <[email protected]>; [email protected]; [email protected]; [email protected]; Matt Roper <[email protected]>
Subject: Re: [Xen-devel] [PATCH 0/1] drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 2 lines · Oleksandr Andrushchenko

Quoted 1 line · Roger Pau Monné

On Wed, Apr 18, 2018 at 09:38:39AM +0300, Oleksandr Andrushchenko

Quoted 3 lines · wrote:

Quoted 3 lines

Quoted 3 lines · Dongwon Kim

Quoted 2 lines · Daniel Vetter

Quoted 1 line · Dongwon Kim

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

3.2 Backend exports dma-buf to xen-front

In this case Dom0 pages are shared with DomU. As before, DomU can

only write

Quoted 1 line

Quoted 1 line

Quoted 1 line

to these pages, not any other page from Dom0, so it can be still

considered

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

safe.
But, the following must be considered (highlighted in xen-front's

Kernel

Quoted 4 lines

Quoted 4 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

documentation):
 - If guest domain dies then pages/grants received from the backend

cannot

Quoted 1 line

Quoted 1 line

Quoted 1 line

   be claimed back - think of it as memory lost to Dom0 (won't be used

Quoted 8 lines

Quoted 8 lines

for

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

any
   other guest)
 - Misbehaving guest may send too many requests to the backend

exhausting

Quoted 1 line

Quoted 1 line

Quoted 1 line

   its grant references and memory (consider this from security POV).

As the

Quoted 1 line

Quoted 1 line

Quoted 1 line

   backend runs in the trusted domain we also assume that it is

trusted

Quoted 21 lines

Quoted 21 lines

as

Quoted 12 lines

Quoted 9 lines

Quoted 2 lines

well,
   e.g. must take measures to prevent DDoS attacks.

I cannot parse the above sentence:

"As the backend runs in the trusted domain we also assume that it is trusted as well, e.g. must take measures to prevent DDoS attacks."

What's the relation between being trusted and protecting from DoS attacks?

I mean that we trust the backend that it can prevent Dom0 from crashing in case DomU's frontend misbehaves, e.g. if the frontend sends too many memory requests etc.

Quoted 7 lines

Quoted 3 lines

In any case, all? PV protocols are implemented with the frontend sharing pages to the backend, and I think there's a reason why this model is used, and it should continue to be used.

This is the first use-case above. But there are real-world use-cases (embedded in my case) when physically contiguous memory needs to be shared, one of the possible ways to achieve this is to share contiguous memory from Dom0 to DomU (the second use-case

above)

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 6 lines

Having to add logic in the backend to prevent such attacks means that:

- We need more code in the backend, which increases complexity and chances of bugs.
- Such code/logic could be wrong, thus allowing DoS.

You can live without this code at all, but this is then up to backend which may make Dom0 down because of DomU's frontend

doing

Quoted 18 lines

Quoted 18 lines

evil

Quoted 1 line

things

IMO we should design protocols that do not allow such attacks instead of having to defend against them.

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

4. xen-front/backend/xen-zcopy synchronization

4.1. As I already said in 2) all the inter VM communication happens

between

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

xen-front and the backend, xen-zcopy is NOT involved in that. When xen-front wants to destroy a display buffer (dumb/dma-buf) it

issues a

Quoted 1 line

Quoted 1 line

Quoted 1 line

XENDISPL_OP_DBUF_DESTROY command (opposite to

XENDISPL_OP_DBUF_CREATE).

Quoted 1 line

Quoted 1 line

Quoted 1 line

This call is synchronous, so xen-front expects that backend does free

the

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

buffer pages on return.

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

4.2. Backend, on XENDISPL_OP_DBUF_DESTROY:
  - closes all dumb handles/fd's of the buffer according to [3]
  - issues DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE IOCTL to xen-

zcopy to make

Quoted 4 lines

Quoted 4 lines

Quoted 3 lines

sure
    the buffer is freed (think of it as it waits for dma-buf->release
callback)

So this zcopy thing keeps some kind of track of the memory usage?

Why

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 1 line

can't the user-space backend keep track of the buffer usage?

Because there is no dma-buf UAPI which allows to track the buffer life

cycle

Quoted 13 lines

Quoted 13 lines

Quoted 3 lines

(e.g. wait until dma-buf's .release callback is called)

Quoted 2 lines

Quoted 2 lines

  - replies to xen-front that the buffer can be destroyed.
This way deletion of the buffer happens synchronously on both Dom0

and DomU

Quoted 1 line

Quoted 1 line

Quoted 1 line

sides. In case if DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE returns

with time-out

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

error
(BTW, wait time is a parameter of this IOCTL), Xen will defer grant reference
removal and will retry later until those are free.

Hope this helps understand how buffers are synchronously deleted in

case

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

of xen-zcopy with a single protocol command.

I think the above logic can also be re-used by the hyper-dmabuf driver

with

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

some additional work:

1. xen-zcopy can be split into 2 parts and extend:
1.1. Xen gntdev driver [4], [5] to allow creating dma-buf from grefs

and

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

Quoted 5 lines

Quoted 1 line

vise versa,

I don't know much about the dma-buf implementation in Linux, but gntdev is a user-space device, and AFAICT user-space applications don't have any notion of dma buffers. How are such buffers useful for user-space? Why can't this just be called memory?

A dma-buf is seen by user-space as a file descriptor and you can pass it to different drivers then. For example, you can share a buffer used by a display driver for scanout with a GPU, to compose a picture into it:
1. User-space (US) allocates a display buffer from display driver
2. US asks display driver to export the dma-buf which backs up that

buffer,

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

US gets buffer's fd: dma_buf_fd
3. US asks GPU driver to import a buffer and provides it with

dma_buf_fd

Quoted 22 lines

Quoted 22 lines

Quoted 1 line

4. GPU renders contents into display buffer (dma_buf_fd)

After speaking with Oleksandr on IRC, I think the main usage of the gntdev extension is to:

1. Create a dma-buf from a set of grant references.
2. Share dma-buf and get a list of grant references.

I think this set of operations could be broken into:

1.1 Map grant references into user-space using the gntdev. 1.2 Create a dma-buf out of a set of user-space virtual addresses.

2.1 Map a dma-buf into user-space.
2.2 Get grefs out of the user-space addresses where the dma-buf is
mapped.

So it seems like what's actually missing is a way to:

- Create a dma-buf from a list of user-space virtual addresses.
- Allow to map a dma-buf into user-space, so it can then be used with the gntdev.

Quoted 8 lines

Quoted 8 lines

I think this is generic enough that it could be implemented by a device not tied to Xen. AFAICT the hyper_dma guys also wanted something similar to this.

Quoted 4 lines

Finally, this is indeed some memory, but a bit more [1]

Quoted 3 lines

Also, (with my FreeBSD maintainer hat) how is this going to translate to other OSes? So far the operations performed by the gntdev device are mostly OS-agnostic because this just map/unmap memory, and in

fact

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 1 line

they are implemented by Linux and FreeBSD.

At the moment I can only see Linux implementation and it seems to be perfectly ok as we do not change Xen's APIs etc. and only use the existing ones (remember, we only extend gntdev/balloon drivers, all the changes in the Linux kernel)
As the second note I can also think that we do not extend

gntdev/balloon

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

drivers and have re-worked xen-zcopy driver be a separate entity, say drivers/xen/dma-buf

Quoted 1 line

Quoted 1 line

implement "wait" ioctl (wait for dma-buf->release): currently these

are

Quoted 24 lines

Quoted 24 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

DRM_XEN_ZCOPY_DUMB_FROM_REFS,

DRM_XEN_ZCOPY_DUMB_TO_REFS and

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

DRM_XEN_ZCOPY_DUMB_WAIT_FREE
1.2. Xen balloon driver [6] to allow allocating contiguous buffers (not needed
by current hyper-dmabuf, but is a must for xen-zcopy use-cases)

Quoted 13 lines

Quoted 2 lines

I think this needs clarifying. In which memory space do you need those regions to be contiguous?

Use-case: Dom0 has a HW driver which only works with contig memory and I want DomU to be able to directly write into that memory, thus implementing zero copying

Quoted 2 lines

Do they need to be contiguous in host physical memory, or guest physical memory?

Host

Quoted 5 lines

If it's in guest memory space, isn't there any generic interface that you can use?

If it's in host physical memory space, why do you need this buffer to be contiguous in host physical memory space? The IOMMU should hide

all

Quoted 4 lines

Quoted 1 line

this.

There are drivers/HW which can only work with contig memory and if it is backed by an IOMMU then still it has to be contig in IPA space (real device doesn't know that it is actually IPA contig, not PA)

Quoted 2 lines

Quoted 1 line

What's IPA contig?

I assume 'IPA' means 'IOMMU Physical Address'. I wonder whether this

means what I've termed 'Bus Address' elsewhere?
sorry for not being clear here: I mean that the device sees contiguous range of
Intermediate Phys Addresses

Quoted 8 lines

Paul

Quoted 6 lines

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Quoted 192 lines

-----Original Message-----
From: Xen-devel [mailto:[email protected]] On Behalf Of Roger Pau Monné
Sent: 18 April 2018 11:11
To: Oleksandr Andrushchenko <[email protected]>
Cc: [email protected]; Artem Mygaiev <[email protected]>; Dongwon Kim <[email protected]>; [email protected]; [email protected]; [email protected]; dri- [email protected]; Potrola, MateuszX <[email protected]>; [email protected]; [email protected]; [email protected]; Matt Roper <[email protected]>
Subject: Re: [Xen-devel] [PATCH 0/1] drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 2 lines · Oleksandr Andrushchenko

Quoted 1 line · Roger Pau Monné

On Wed, Apr 18, 2018 at 09:38:39AM +0300, Oleksandr Andrushchenko

Quoted 6 lines · wrote:

Quoted 6 lines

Quoted 6 lines · Dongwon Kim

Quoted 2 lines · Daniel Vetter

Quoted 1 line · Dongwon Kim

3.2 Backend exports dma-buf to xen-front

In this case Dom0 pages are shared with DomU. As before, DomU can

only write

Quoted 1 line

Quoted 1 line

Quoted 1 line

to these pages, not any other page from Dom0, so it can be still

considered

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

safe.
But, the following must be considered (highlighted in xen-front's Kernel documentation):
 - If guest domain dies then pages/grants received from the backend

cannot

Quoted 1 line

Quoted 1 line

Quoted 1 line

   be claimed back - think of it as memory lost to Dom0 (won't be used

for

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

any
   other guest)
 - Misbehaving guest may send too many requests to the backend

exhausting

Quoted 1 line

Quoted 1 line

Quoted 1 line

   its grant references and memory (consider this from security POV).

As the

Quoted 1 line

Quoted 1 line

Quoted 1 line

   backend runs in the trusted domain we also assume that it is trusted

as

Quoted 12 lines

Quoted 9 lines

Quoted 2 lines

well,
   e.g. must take measures to prevent DDoS attacks.

I cannot parse the above sentence:

"As the backend runs in the trusted domain we also assume that it is trusted as well, e.g. must take measures to prevent DDoS attacks."

What's the relation between being trusted and protecting from DoS attacks?

I mean that we trust the backend that it can prevent Dom0 from crashing in case DomU's frontend misbehaves, e.g. if the frontend sends too many memory requests etc.

Quoted 7 lines

Quoted 3 lines

In any case, all? PV protocols are implemented with the frontend sharing pages to the backend, and I think there's a reason why this model is used, and it should continue to be used.

This is the first use-case above. But there are real-world use-cases (embedded in my case) when physically contiguous memory needs to be shared, one of the possible ways to achieve this is to share contiguous memory from Dom0 to DomU (the second use-case

above)

Quoted 8 lines

Quoted 6 lines

Having to add logic in the backend to prevent such attacks means that:

- We need more code in the backend, which increases complexity and chances of bugs.
- Such code/logic could be wrong, thus allowing DoS.

You can live without this code at all, but this is then up to backend which may make Dom0 down because of DomU's frontend doing

evil

Quoted 1 line

things

IMO we should design protocols that do not allow such attacks instead of having to defend against them.

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

4. xen-front/backend/xen-zcopy synchronization

4.1. As I already said in 2) all the inter VM communication happens

between

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

xen-front and the backend, xen-zcopy is NOT involved in that. When xen-front wants to destroy a display buffer (dumb/dma-buf) it

issues a

Quoted 1 line

Quoted 1 line

Quoted 1 line

XENDISPL_OP_DBUF_DESTROY command (opposite to

XENDISPL_OP_DBUF_CREATE).

Quoted 1 line

Quoted 1 line

Quoted 1 line

This call is synchronous, so xen-front expects that backend does free

the

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

buffer pages on return.

4.2. Backend, on XENDISPL_OP_DBUF_DESTROY:
  - closes all dumb handles/fd's of the buffer according to [3]
  - issues DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE IOCTL to xen-

zcopy to make

Quoted 9 lines

Quoted 5 lines

Quoted 3 lines

sure
    the buffer is freed (think of it as it waits for dma-buf->release
callback)

So this zcopy thing keeps some kind of track of the memory usage? Why can't the user-space backend keep track of the buffer usage?

Because there is no dma-buf UAPI which allows to track the buffer life cycle (e.g. wait until dma-buf's .release callback is called)

Quoted 2 lines

Quoted 2 lines

  - replies to xen-front that the buffer can be destroyed.
This way deletion of the buffer happens synchronously on both Dom0

and DomU

Quoted 1 line

Quoted 1 line

Quoted 1 line

sides. In case if DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE returns

with time-out

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

error
(BTW, wait time is a parameter of this IOCTL), Xen will defer grant reference
removal and will retry later until those are free.

Hope this helps understand how buffers are synchronously deleted in

case

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

of xen-zcopy with a single protocol command.

I think the above logic can also be re-used by the hyper-dmabuf driver

with

Quoted 9 lines

Quoted 9 lines

Quoted 5 lines

some additional work:

1. xen-zcopy can be split into 2 parts and extend:
1.1. Xen gntdev driver [4], [5] to allow creating dma-buf from grefs and vise versa,

I don't know much about the dma-buf implementation in Linux, but gntdev is a user-space device, and AFAICT user-space applications don't have any notion of dma buffers. How are such buffers useful for user-space? Why can't this just be called memory?

Quoted 9 lines

A dma-buf is seen by user-space as a file descriptor and you can pass it to different drivers then. For example, you can share a buffer used by a display driver for scanout with a GPU, to compose a picture into it:
1. User-space (US) allocates a display buffer from display driver
2. US asks display driver to export the dma-buf which backs up that buffer, US gets buffer's fd: dma_buf_fd
3. US asks GPU driver to import a buffer and provides it with dma_buf_fd
4. GPU renders contents into display buffer (dma_buf_fd)

After speaking with Oleksandr on IRC, I think the main usage of the gntdev extension is to:

1. Create a dma-buf from a set of grant references.
2. Share dma-buf and get a list of grant references.

I think this set of operations could be broken into:

1.1 Map grant references into user-space using the gntdev. 1.2 Create a dma-buf out of a set of user-space virtual addresses.

2.1 Map a dma-buf into user-space.
2.2 Get grefs out of the user-space addresses where the dma-buf is
mapped.

So it seems like what's actually missing is a way to:

- Create a dma-buf from a list of user-space virtual addresses.
- Allow to map a dma-buf into user-space, so it can then be used with the gntdev.

I think this is generic enough that it could be implemented by a device not tied to Xen. AFAICT the hyper_dma guys also wanted something similar to this.

Quoted 5 lines

Finally, this is indeed some memory, but a bit more [1]

Quoted 4 lines

Also, (with my FreeBSD maintainer hat) how is this going to translate to other OSes? So far the operations performed by the gntdev device are mostly OS-agnostic because this just map/unmap memory, and in fact they are implemented by Linux and FreeBSD.

Quoted 9 lines

At the moment I can only see Linux implementation and it seems to be perfectly ok as we do not change Xen's APIs etc. and only use the existing ones (remember, we only extend gntdev/balloon drivers, all the changes in the Linux kernel)
As the second note I can also think that we do not extend gntdev/balloon drivers and have re-worked xen-zcopy driver be a separate entity, say drivers/xen/dma-buf

Quoted 2 lines

Quoted 2 lines

implement "wait" ioctl (wait for dma-buf->release): currently these are DRM_XEN_ZCOPY_DUMB_FROM_REFS,

DRM_XEN_ZCOPY_DUMB_TO_REFS and

Quoted 17 lines

Quoted 6 lines

Quoted 4 lines

DRM_XEN_ZCOPY_DUMB_WAIT_FREE
1.2. Xen balloon driver [6] to allow allocating contiguous buffers (not needed
by current hyper-dmabuf, but is a must for xen-zcopy use-cases)

I think this needs clarifying. In which memory space do you need those regions to be contiguous?

Use-case: Dom0 has a HW driver which only works with contig memory and I want DomU to be able to directly write into that memory, thus implementing zero copying

Quoted 2 lines

Do they need to be contiguous in host physical memory, or guest physical memory?

Host

Quoted 5 lines

If it's in guest memory space, isn't there any generic interface that you can use?

If it's in host physical memory space, why do you need this buffer to be contiguous in host physical memory space? The IOMMU should hide

Quoted 6 lines

all

Quoted 4 lines

Quoted 1 line

this.

There are drivers/HW which can only work with contig memory and if it is backed by an IOMMU then still it has to be contig in IPA space (real device doesn't know that it is actually IPA contig, not PA)

What's IPA contig?

I assume 'IPA' means 'IOMMU Physical Address'. I wonder whether this means what I've termed 'Bus Address' elsewhere?

Paul

Quoted 6 lines

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Quoted 152 lines

-----Original Message-----
From: Xen-devel [mailto:[email protected]] On Behalf Of Roger Pau Monné
Sent: 18 April 2018 11:11
To: Oleksandr Andrushchenko <[email protected]>
Cc: [email protected]; Artem Mygaiev <[email protected]>; Dongwon Kim <[email protected]>; [email protected]; [email protected]; [email protected]; dri- [email protected]; Potrola, MateuszX <[email protected]>; [email protected]; [email protected]; [email protected]; Matt Roper <[email protected]>
Subject: Re: [Xen-devel] [PATCH 0/1] drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 2 lines · Oleksandr Andrushchenko

Quoted 1 line · Roger Pau Monné

On Wed, Apr 18, 2018 at 09:38:39AM +0300, Oleksandr Andrushchenko

Quoted 6 lines · wrote:

Quoted 6 lines

Quoted 6 lines · Dongwon Kim

Quoted 2 lines · Daniel Vetter

Quoted 1 line · Dongwon Kim

3.2 Backend exports dma-buf to xen-front

In this case Dom0 pages are shared with DomU. As before, DomU can

only write

Quoted 1 line

Quoted 1 line

Quoted 1 line

to these pages, not any other page from Dom0, so it can be still

considered

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

safe.
But, the following must be considered (highlighted in xen-front's Kernel documentation):
 - If guest domain dies then pages/grants received from the backend

cannot

Quoted 1 line

Quoted 1 line

Quoted 1 line

   be claimed back - think of it as memory lost to Dom0 (won't be used

for

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

any
   other guest)
 - Misbehaving guest may send too many requests to the backend

exhausting

Quoted 1 line

Quoted 1 line

Quoted 1 line

   its grant references and memory (consider this from security POV).

As the

Quoted 1 line

Quoted 1 line

Quoted 1 line

   backend runs in the trusted domain we also assume that it is trusted

as

Quoted 12 lines

Quoted 9 lines

Quoted 2 lines

well,
   e.g. must take measures to prevent DDoS attacks.

I cannot parse the above sentence:

"As the backend runs in the trusted domain we also assume that it is trusted as well, e.g. must take measures to prevent DDoS attacks."

What's the relation between being trusted and protecting from DoS attacks?

I mean that we trust the backend that it can prevent Dom0 from crashing in case DomU's frontend misbehaves, e.g. if the frontend sends too many memory requests etc.

Quoted 7 lines

Quoted 3 lines

In any case, all? PV protocols are implemented with the frontend sharing pages to the backend, and I think there's a reason why this model is used, and it should continue to be used.

This is the first use-case above. But there are real-world use-cases (embedded in my case) when physically contiguous memory needs to be shared, one of the possible ways to achieve this is to share contiguous memory from Dom0 to DomU (the second use-case

above)

Quoted 8 lines

Quoted 6 lines

Having to add logic in the backend to prevent such attacks means that:

- We need more code in the backend, which increases complexity and chances of bugs.
- Such code/logic could be wrong, thus allowing DoS.

You can live without this code at all, but this is then up to backend which may make Dom0 down because of DomU's frontend doing

evil

Quoted 1 line

things

IMO we should design protocols that do not allow such attacks instead of having to defend against them.

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

4. xen-front/backend/xen-zcopy synchronization

4.1. As I already said in 2) all the inter VM communication happens

between

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

xen-front and the backend, xen-zcopy is NOT involved in that. When xen-front wants to destroy a display buffer (dumb/dma-buf) it

issues a

Quoted 1 line

Quoted 1 line

Quoted 1 line

XENDISPL_OP_DBUF_DESTROY command (opposite to

XENDISPL_OP_DBUF_CREATE).

Quoted 1 line

Quoted 1 line

Quoted 1 line

This call is synchronous, so xen-front expects that backend does free

the

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

buffer pages on return.

4.2. Backend, on XENDISPL_OP_DBUF_DESTROY:
  - closes all dumb handles/fd's of the buffer according to [3]
  - issues DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE IOCTL to xen-

zcopy to make

Quoted 9 lines

Quoted 5 lines

Quoted 3 lines

sure
    the buffer is freed (think of it as it waits for dma-buf->release
callback)

So this zcopy thing keeps some kind of track of the memory usage? Why can't the user-space backend keep track of the buffer usage?

Because there is no dma-buf UAPI which allows to track the buffer life cycle (e.g. wait until dma-buf's .release callback is called)

Quoted 2 lines

Quoted 2 lines

  - replies to xen-front that the buffer can be destroyed.
This way deletion of the buffer happens synchronously on both Dom0

and DomU

Quoted 1 line

Quoted 1 line

Quoted 1 line

sides. In case if DRM_IOCTL_XEN_ZCOPY_DUMB_WAIT_FREE returns

with time-out

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

error
(BTW, wait time is a parameter of this IOCTL), Xen will defer grant reference
removal and will retry later until those are free.

Hope this helps understand how buffers are synchronously deleted in

case

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

of xen-zcopy with a single protocol command.

I think the above logic can also be re-used by the hyper-dmabuf driver

with

Quoted 9 lines

Quoted 9 lines

Quoted 5 lines

some additional work:

1. xen-zcopy can be split into 2 parts and extend:
1.1. Xen gntdev driver [4], [5] to allow creating dma-buf from grefs and vise versa,

I don't know much about the dma-buf implementation in Linux, but gntdev is a user-space device, and AFAICT user-space applications don't have any notion of dma buffers. How are such buffers useful for user-space? Why can't this just be called memory?

Quoted 9 lines

A dma-buf is seen by user-space as a file descriptor and you can pass it to different drivers then. For example, you can share a buffer used by a display driver for scanout with a GPU, to compose a picture into it:
1. User-space (US) allocates a display buffer from display driver
2. US asks display driver to export the dma-buf which backs up that buffer, US gets buffer's fd: dma_buf_fd
3. US asks GPU driver to import a buffer and provides it with dma_buf_fd
4. GPU renders contents into display buffer (dma_buf_fd)

After speaking with Oleksandr on IRC, I think the main usage of the gntdev extension is to:

1. Create a dma-buf from a set of grant references.
2. Share dma-buf and get a list of grant references.

I think this set of operations could be broken into:

1.1 Map grant references into user-space using the gntdev. 1.2 Create a dma-buf out of a set of user-space virtual addresses.

2.1 Map a dma-buf into user-space.
2.2 Get grefs out of the user-space addresses where the dma-buf is
mapped.

So it seems like what's actually missing is a way to:

Quoted 7 lines

- Create a dma-buf from a list of user-space virtual addresses.
- Allow to map a dma-buf into user-space, so it can then be used with the gntdev.

I think this is generic enough that it could be implemented by a device not tied to Xen. AFAICT the hyper_dma guys also wanted something similar to this.

Quoted 33 lines

Quoted 5 lines

Finally, this is indeed some memory, but a bit more [1]

Quoted 4 lines

Also, (with my FreeBSD maintainer hat) how is this going to translate to other OSes? So far the operations performed by the gntdev device are mostly OS-agnostic because this just map/unmap memory, and in fact they are implemented by Linux and FreeBSD.

Quoted 9 lines

At the moment I can only see Linux implementation and it seems to be perfectly ok as we do not change Xen's APIs etc. and only use the existing ones (remember, we only extend gntdev/balloon drivers, all the changes in the Linux kernel)
As the second note I can also think that we do not extend gntdev/balloon drivers and have re-worked xen-zcopy driver be a separate entity, say drivers/xen/dma-buf

Quoted 2 lines

Quoted 2 lines

implement "wait" ioctl (wait for dma-buf->release): currently these are DRM_XEN_ZCOPY_DUMB_FROM_REFS,

DRM_XEN_ZCOPY_DUMB_TO_REFS and

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

DRM_XEN_ZCOPY_DUMB_WAIT_FREE
1.2. Xen balloon driver [6] to allow allocating contiguous buffers (not needed
by current hyper-dmabuf, but is a must for xen-zcopy use-cases)

Quoted 13 lines

Quoted 2 lines

I think this needs clarifying. In which memory space do you need those regions to be contiguous?

Use-case: Dom0 has a HW driver which only works with contig memory and I want DomU to be able to directly write into that memory, thus implementing zero copying

Quoted 2 lines

Do they need to be contiguous in host physical memory, or guest physical memory?

Host

Quoted 5 lines

If it's in guest memory space, isn't there any generic interface that you can use?

If it's in host physical memory space, why do you need this buffer to be contiguous in host physical memory space? The IOMMU should hide

all

Quoted 5 lines

Quoted 4 lines

Quoted 1 line

this.

There are drivers/HW which can only work with contig memory and if it is backed by an IOMMU then still it has to be contig in IPA space (real device doesn't know that it is actually IPA contig, not PA)

What's IPA contig?

I assume 'IPA' means 'IOMMU Physical Address'. I wonder whether this means what I've termed 'Bus Address' elsewhere?

Paul

Quoted 6 lines

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Quoted 39 lines · Paul Durrant

Quoted 39 lines

-----Original Message-----
From: Xen-devel [mailto:[email protected]] On Behalf Of Roger Pau Monné
Sent: 18 April 2018 11:11
To: Oleksandr Andrushchenko <[email protected]>
Cc: [email protected]; Artem Mygaiev <[email protected]>; Dongwon Kim <[email protected]>; [email protected]; [email protected]; [email protected]; dri- [email protected]; Potrola, MateuszX <[email protected]>; [email protected]; [email protected]; [email protected]; Matt Roper <[email protected]>
Subject: Re: [Xen-devel] [PATCH 0/1] drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 1 line · Oleksandr Andrushchenko

On 04/18/2018 10:35 AM, Roger Pau Monné wrote:

After speaking with Oleksandr on IRC, I think the main usage of the gntdev extension is to:

1. Create a dma-buf from a set of grant references.
2. Share dma-buf and get a list of grant references.

I think this set of operations could be broken into:

1.1 Map grant references into user-space using the gntdev. 1.2 Create a dma-buf out of a set of user-space virtual addresses.

2.1 Map a dma-buf into user-space.
2.2 Get grefs out of the user-space addresses where the dma-buf is
mapped.

So it seems like what's actually missing is a way to:

- Create a dma-buf from a list of user-space virtual addresses.
- Allow to map a dma-buf into user-space, so it can then be used with the gntdev.

Quoted 3 lines

Quoted 3 lines

I think this is generic enough that it could be implemented by a device not tied to Xen. AFAICT the hyper_dma guys also wanted something similar to this.

Ok, so just to summarize, xen-zcopy/hyper-dmabuf as they are now, are no go from your POV?

Instead, we have to make all that fancy stuff
with VAs <-> device-X and have that device-X driver live out of drivers/xen as it is not a Xen specific driver?

Quoted 40 lines · Oleksandr Andrushchenko

Quoted 39 lines · Paul Durrant

Quoted 39 lines

-----Original Message-----
From: Xen-devel [mailto:[email protected]] On Behalf Of Roger Pau Monné
Sent: 18 April 2018 11:11
To: Oleksandr Andrushchenko <[email protected]>
Cc: [email protected]; Artem Mygaiev <[email protected]>; Dongwon Kim <[email protected]>; [email protected]; [email protected]; [email protected]; dri- [email protected]; Potrola, MateuszX <[email protected]>; [email protected]; [email protected]; [email protected]; Matt Roper <[email protected]>
Subject: Re: [Xen-devel] [PATCH 0/1] drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 1 line · Oleksandr Andrushchenko

On 04/18/2018 10:35 AM, Roger Pau Monné wrote:

After speaking with Oleksandr on IRC, I think the main usage of the gntdev extension is to:

1. Create a dma-buf from a set of grant references.
2. Share dma-buf and get a list of grant references.

I think this set of operations could be broken into:

1.1 Map grant references into user-space using the gntdev. 1.2 Create a dma-buf out of a set of user-space virtual addresses.

2.1 Map a dma-buf into user-space.
2.2 Get grefs out of the user-space addresses where the dma-buf is
mapped.

So it seems like what's actually missing is a way to:

- Create a dma-buf from a list of user-space virtual addresses.
- Allow to map a dma-buf into user-space, so it can then be used with the gntdev.

Quoted 5 lines

Quoted 3 lines

Quoted 3 lines

I think this is generic enough that it could be implemented by a device not tied to Xen. AFAICT the hyper_dma guys also wanted something similar to this.

Ok, so just to summarize, xen-zcopy/hyper-dmabuf as they are now, are no go from your POV?

My opinion is that there seems to be a more generic way to implement this, and thus I would prefer that one.

Quoted 3 lines

Instead, we have to make all that fancy stuff
with VAs <-> device-X and have that device-X driver live out of drivers/xen as it is not a Xen specific driver?

That would be my preference if feasible, simply because it can be reused by other use-cases that need to create dma-bufs in user-space.

In any case I just knew about dma-bufs this morning, there might be things that I'm missing.

Roger.

Quoted 41 lines · Roger Pau Monné

Quoted 40 lines · Oleksandr Andrushchenko

Quoted 39 lines · Paul Durrant

Quoted 39 lines

-----Original Message-----
From: Xen-devel [mailto:[email protected]] On Behalf Of Roger Pau Monné
Sent: 18 April 2018 11:11
To: Oleksandr Andrushchenko <[email protected]>
Cc: [email protected]; Artem Mygaiev <[email protected]>; Dongwon Kim <[email protected]>; [email protected]; [email protected]; [email protected]; dri- [email protected]; Potrola, MateuszX <[email protected]>; [email protected]; [email protected]; [email protected]; Matt Roper <[email protected]>
Subject: Re: [Xen-devel] [PATCH 0/1] drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 1 line · Oleksandr Andrushchenko

On 04/18/2018 10:35 AM, Roger Pau Monné wrote:

After speaking with Oleksandr on IRC, I think the main usage of the gntdev extension is to:

1. Create a dma-buf from a set of grant references.
2. Share dma-buf and get a list of grant references.

I think this set of operations could be broken into:

1.1 Map grant references into user-space using the gntdev. 1.2 Create a dma-buf out of a set of user-space virtual addresses.

2.1 Map a dma-buf into user-space.
2.2 Get grefs out of the user-space addresses where the dma-buf is
mapped.

So it seems like what's actually missing is a way to:

- Create a dma-buf from a list of user-space virtual addresses.
- Allow to map a dma-buf into user-space, so it can then be used with the gntdev.

Quoted 5 lines

Quoted 5 lines

Quoted 3 lines

Quoted 3 lines

I think this is generic enough that it could be implemented by a device not tied to Xen. AFAICT the hyper_dma guys also wanted something similar to this.

Ok, so just to summarize, xen-zcopy/hyper-dmabuf as they are now, are no go from your POV?

Quoted 8 lines

My opinion is that there seems to be a more generic way to implement this, and thus I would prefer that one.

Quoted 3 lines

Instead, we have to make all that fancy stuff
with VAs <-> device-X and have that device-X driver live out of drivers/xen as it is not a Xen specific driver?

That would be my preference if feasible, simply because it can be reused by other use-cases that need to create dma-bufs in user-space.

There is a use-case I have: a display unit on my target has a DMA controller which can't do scatter-gather, e.g. it only expects a single starting address of the buffer.
In order to create a dma-buf from grefs in this case
I allocate memory with dma_alloc_xxx and then balloon pages of the buffer and finally map grefs onto this DMA buffer.
This way I can give this shared buffer to the display unit as its bus addresses are contiguous.

With the proposed solution (gntdev + device-X) I won't be able to achieve this,
as I have no control over from where gntdev/balloon drivers get the pages (even more, those can easily be out of DMA address space of the display unit).

Thus, even if implemented, I can't use this approach.

Quoted 5 lines

In any case I just knew about dma-bufs this morning, there might be things that I'm missing.

Roger.

Quoted 42 lines · Oleksandr Andrushchenko

Quoted 41 lines · Roger Pau Monné

Quoted 40 lines · Oleksandr Andrushchenko

Quoted 39 lines · Paul Durrant

Quoted 39 lines

-----Original Message-----
From: Xen-devel [mailto:[email protected]] On Behalf Of Roger Pau Monné
Sent: 18 April 2018 11:11
To: Oleksandr Andrushchenko <[email protected]>
Cc: [email protected]; Artem Mygaiev <[email protected]>; Dongwon Kim <[email protected]>; [email protected]; [email protected]; [email protected]; dri- [email protected]; Potrola, MateuszX <[email protected]>; [email protected]; [email protected]; [email protected]; Matt Roper <[email protected]>
Subject: Re: [Xen-devel] [PATCH 0/1] drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 1 line · Oleksandr Andrushchenko

On 04/18/2018 10:35 AM, Roger Pau Monné wrote:

After speaking with Oleksandr on IRC, I think the main usage of the gntdev extension is to:

1. Create a dma-buf from a set of grant references.
2. Share dma-buf and get a list of grant references.

I think this set of operations could be broken into:

1.1 Map grant references into user-space using the gntdev. 1.2 Create a dma-buf out of a set of user-space virtual addresses.

2.1 Map a dma-buf into user-space.
2.2 Get grefs out of the user-space addresses where the dma-buf is
mapped.

So it seems like what's actually missing is a way to:

- Create a dma-buf from a list of user-space virtual addresses.
- Allow to map a dma-buf into user-space, so it can then be used with the gntdev.

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 3 lines

Quoted 3 lines

I think this is generic enough that it could be implemented by a device not tied to Xen. AFAICT the hyper_dma guys also wanted something similar to this.

Ok, so just to summarize, xen-zcopy/hyper-dmabuf as they are now, are no go from your POV?

FYI,

our use-case is "surface sharing" or "graphic obj sharing" where a client application in one guest renders and export this render target(e.g. EGL surface) as dma-buf. This dma-buf is then exported to another guest/host via hyper_dmabuf drv where a compositor is running. This importing domain creates a dmabuf with shared reference then it is imported as EGL image that later can be used as texture object via EGL api.

Mapping dmabuf to the userspace or vice versa
might be possible with modifying user space drivers/applications but it is an unnecessary extra step from our perspective.

Also, we want to keep all objects
in the kernel level.

Quoted 28 lines

Quoted 8 lines

My opinion is that there seems to be a more generic way to implement this, and thus I would prefer that one.

Quoted 3 lines

Instead, we have to make all that fancy stuff
with VAs <-> device-X and have that device-X driver live out of drivers/xen as it is not a Xen specific driver?

That would be my preference if feasible, simply because it can be reused by other use-cases that need to create dma-bufs in user-space.

There is a use-case I have: a display unit on my target has a DMA controller which can't do scatter-gather, e.g. it only expects a single starting address of the buffer.
In order to create a dma-buf from grefs in this case
I allocate memory with dma_alloc_xxx and then balloon pages of the buffer and finally map grefs onto this DMA buffer.
This way I can give this shared buffer to the display unit as its bus addresses are contiguous.

With the proposed solution (gntdev + device-X) I won't be able to achieve this,
as I have no control over from where gntdev/balloon drivers get the pages (even more, those can easily be out of DMA address space of the display unit).

Thus, even if implemented, I can't use this approach.

Quoted 4 lines

In any case I just knew about dma-bufs this morning, there might be things that I'm missing.

Roger.

Quoted 42 lines · Oleksandr Andrushchenko

Quoted 41 lines · Roger Pau Monné

Quoted 40 lines · Oleksandr Andrushchenko

Quoted 39 lines · Paul Durrant

Quoted 39 lines

-----Original Message-----
From: Xen-devel [mailto:[email protected]] On Behalf Of Roger Pau Monné
Sent: 18 April 2018 11:11
To: Oleksandr Andrushchenko <[email protected]>
Cc: [email protected]; Artem Mygaiev <[email protected]>; Dongwon Kim <[email protected]>; [email protected]; [email protected]; [email protected]; dri- [email protected]; Potrola, MateuszX <[email protected]>; [email protected]; [email protected]; [email protected]; Matt Roper <[email protected]>
Subject: Re: [Xen-devel] [PATCH 0/1] drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 1 line · Oleksandr Andrushchenko

On 04/18/2018 10:35 AM, Roger Pau Monné wrote:

After speaking with Oleksandr on IRC, I think the main usage of the gntdev extension is to:

1. Create a dma-buf from a set of grant references.
2. Share dma-buf and get a list of grant references.

I think this set of operations could be broken into:

1.1 Map grant references into user-space using the gntdev. 1.2 Create a dma-buf out of a set of user-space virtual addresses.

2.1 Map a dma-buf into user-space.
2.2 Get grefs out of the user-space addresses where the dma-buf is
mapped.

So it seems like what's actually missing is a way to:

- Create a dma-buf from a list of user-space virtual addresses.
- Allow to map a dma-buf into user-space, so it can then be used with the gntdev.

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 3 lines

Quoted 3 lines

I think this is generic enough that it could be implemented by a device not tied to Xen. AFAICT the hyper_dma guys also wanted something similar to this.

Ok, so just to summarize, xen-zcopy/hyper-dmabuf as they are now, are no go from your POV?

FYI,

our use-case is "surface sharing" or "graphic obj sharing" where a client application in one guest renders and export this render target(e.g. EGL surface) as dma-buf. This dma-buf is then exported to another guest/host via hyper_dmabuf drv where a compositor is running. This importing domain creates a dmabuf with shared reference then it is imported as EGL image that later can be used as texture object via EGL api.

Mapping dmabuf to the userspace or vice versa
might be possible with modifying user space drivers/applications but it is an unnecessary extra step from our perspective.

Also, we want to keep all objects
in the kernel level.

Quoted 28 lines

Quoted 8 lines

My opinion is that there seems to be a more generic way to implement this, and thus I would prefer that one.

Quoted 3 lines

Instead, we have to make all that fancy stuff
with VAs <-> device-X and have that device-X driver live out of drivers/xen as it is not a Xen specific driver?

That would be my preference if feasible, simply because it can be reused by other use-cases that need to create dma-bufs in user-space.

There is a use-case I have: a display unit on my target has a DMA controller which can't do scatter-gather, e.g. it only expects a single starting address of the buffer.
In order to create a dma-buf from grefs in this case
I allocate memory with dma_alloc_xxx and then balloon pages of the buffer and finally map grefs onto this DMA buffer.
This way I can give this shared buffer to the display unit as its bus addresses are contiguous.

With the proposed solution (gntdev + device-X) I won't be able to achieve this,
as I have no control over from where gntdev/balloon drivers get the pages (even more, those can easily be out of DMA address space of the display unit).

Thus, even if implemented, I can't use this approach.

Quoted 4 lines

In any case I just knew about dma-bufs this morning, there might be things that I'm missing.

Roger.

Quoted 41 lines · Roger Pau Monné

Quoted 40 lines · Oleksandr Andrushchenko

Quoted 39 lines · Paul Durrant

Quoted 39 lines

-----Original Message-----
From: Xen-devel [mailto:[email protected]] On Behalf Of Roger Pau Monné
Sent: 18 April 2018 11:11
To: Oleksandr Andrushchenko <[email protected]>
Cc: [email protected]; Artem Mygaiev <[email protected]>; Dongwon Kim <[email protected]>; [email protected]; [email protected]; [email protected]; dri- [email protected]; Potrola, MateuszX <[email protected]>; [email protected]; [email protected]; [email protected]; Matt Roper <[email protected]>
Subject: Re: [Xen-devel] [PATCH 0/1] drm/xen-zcopy: Add Xen zero-copy helper DRM driver

Quoted 1 line · Oleksandr Andrushchenko

On 04/18/2018 10:35 AM, Roger Pau Monné wrote:

After speaking with Oleksandr on IRC, I think the main usage of the gntdev extension is to:

1. Create a dma-buf from a set of grant references.
2. Share dma-buf and get a list of grant references.

I think this set of operations could be broken into:

1.1 Map grant references into user-space using the gntdev. 1.2 Create a dma-buf out of a set of user-space virtual addresses.

2.1 Map a dma-buf into user-space.
2.2 Get grefs out of the user-space addresses where the dma-buf is
mapped.

So it seems like what's actually missing is a way to:

- Create a dma-buf from a list of user-space virtual addresses.
- Allow to map a dma-buf into user-space, so it can then be used with the gntdev.

Quoted 5 lines

Quoted 5 lines

Quoted 3 lines

Quoted 3 lines

I think this is generic enough that it could be implemented by a device not tied to Xen. AFAICT the hyper_dma guys also wanted something similar to this.

Ok, so just to summarize, xen-zcopy/hyper-dmabuf as they are now, are no go from your POV?

Quoted 8 lines

My opinion is that there seems to be a more generic way to implement this, and thus I would prefer that one.

Quoted 3 lines

Instead, we have to make all that fancy stuff
with VAs <-> device-X and have that device-X driver live out of drivers/xen as it is not a Xen specific driver?

That would be my preference if feasible, simply because it can be reused by other use-cases that need to create dma-bufs in user-space.

There is a use-case I have: a display unit on my target has a DMA controller which can't do scatter-gather, e.g. it only expects a single starting address of the buffer.
In order to create a dma-buf from grefs in this case
I allocate memory with dma_alloc_xxx and then balloon pages of the buffer and finally map grefs onto this DMA buffer.
This way I can give this shared buffer to the display unit as its bus addresses are contiguous.