drm/doc: device hot-unplug for userspace
Subsystems (3)
PATCH 1/1 drm/doc: device hot-unplug for userspace
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and from https://lists.freedesktop.org/archives/dri-devel/2020-May/265484.html .
---
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 75 ++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)
Documentation/gpu/drm-uapi.rst +75 -0
@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================
@@ -162,6 +164,79 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May 13). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some cases specific +ioctls returning EIO. + +This goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was + in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. + +- If underlying memory disappears, the mmaps are replaced with harmless + zero pages where access does not raise SIGBUS. Reads return zeros, + writes are ignored. + +- dmabuf which point to memory that has disappeared are rewritten to + point to harmless zero pages, similar to mmaps. Imports still succeed + both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing any dmabuf. + +- Render ioctls return EIO which is then handled in userspace drivers, + e.g. Mesa, to have the device disappearance handled in the way + specified for each API (OpenGL, GL ES: GL_KHR_robustness; + Vulkan: VK_ERROR_DEVICE_LOST; etc.) + +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like Mesa produces. Signal +handlers are not composable, you can't have different handlers for GPU1 +and GPU2 from different vendors, and a third handler for mmapped regular +files. Threads cause additional pain with signal handling as well. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes
Signature · 6 lines
2.20.1 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Thanks for the summary, does put things in order and makes it easier to comprehend all the TODOs, some questions bellow
Quoted 69 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Ce8e13dc4c85648e5fcd408d7fbdc5f2b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637254796242596783&sdata=bA%2FAy3VGvzNqmV1kGigNROSZQEws2E98JibDxvEICNs%3D&reserved=0 .Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>---
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 75 ++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)Documentation/gpu/drm-uapi.rst +34
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,79 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May 13). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some cases specific +ioctls returning EIO. + +This goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 208
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+ in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success.
Why wouldn't we return -EIO for the atommic commit IOTCL/legasy pflip and modeset ioctls here same way as you suggested returning -EIO for render ioctl ?
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 220
Quoted 12 lines
Documentation/gpu/drm-uapi.rst +12
-0+ +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. + +- If underlying memory disappears, the mmaps are replaced with harmless + zero pages where access does not raise SIGBUS. Reads return zeros, + writes are ignored. + +- dmabuf which point to memory that has disappeared are rewritten to + point to harmless zero pages, similar to mmaps. Imports still succeed + both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing any dmabuf.
Same as above, I would expect drmPrimeHandleToFD and drmPrimeFDToHandle to return -EIO in case device is detached
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 225
Quoted 5 lines
Documentation/gpu/drm-uapi.rst +5
-0+ +- Render ioctls return EIO which is then handled in userspace drivers, + e.g. Mesa, to have the device disappearance handled in the way + specified for each API (OpenGL, GL ES: GL_KHR_robustness; + Vulkan: VK_ERROR_DEVICE_LOST; etc.)
As far as I understood from our discussion we would reject any IOCTL aimed at a device which is gone and not only render ioctls.
Andrey
Quoted 7 lines
Documentation/gpu/drm-uapi.rst +7
-0+ +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like Mesa produces. Signal +handlers are not composable, you can't have different handlers for GPU1 +and GPU2 from different vendors, and a third handler for mmapped regular +files. Threads cause additional pain with signal handling as well.
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 242
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +7
-0+ +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 80 lines · On Tue, 19 May 2020 10:37:12 -0400 Andrey Grodzovsky…
Thanks for the summary, does put things in order and makes it easier to comprehend all the TODOs, some questions bellow
Quoted 76 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Ce8e13dc4c85648e5fcd408d7fbdc5f2b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637254796242596783&sdata=bA%2FAy3VGvzNqmV1kGigNROSZQEws2E98JibDxvEICNs%3D&reserved=0 .Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>---
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 75 ++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)Documentation/gpu/drm-uapi.rst +41
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,79 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May 13). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some cases specific +ioctls returning EIO. + +This goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was + in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected. +
Quoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 208
Quoted 3 lines
Documentation/gpu/drm-uapi.rst +3
-0+- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success.Why wouldn't we return
-EIOfor the atommic commit IOTCL/legasy pflip and modeset ioctls here same way as you suggested returning-EIOfor render ioctl ?
Hi,
that is more of a question for Daniel Vetter than me. I believe he is worried that userspace will get the error handling horribly wrong anyway, because it needs to be handled in every single display server project. Render ioctl errors OTOH need to be handled only in the corresponding Mesa or other userspace driver, and for render there are API (OpenGL/EGL, Vulkan) specs that say how it must be handled to fill the API contract. Because of the API contract, those are more likely to have reasonable error handling in place already.
I first thought it would be obvious for at least atomic commits to fail appropriately, but then thinking again, it will only introduce new failures that are currently very hard to test for (VKMS to the rescue), and userspace would need to have code to correctly bail out for EIO rather than attempt fallbacks. The uevent telling the device is gone is going to come anyway. In between the device disappearance and uevent handling, it doesn't matter much what happens: the device is abruptly gone and the user will see a glitch anyway - he knows, he just yanked out the cable.
To me this decision is just to make life in userspace easier by not introducing a whole new type of error. However, if the kernel did return EIO instead, it would be impossible for me to see that as a kernel regression because the old behaviour was probably explosions everywhere anyway.
I heard that UDL and maybe some other drivers in the kernel already handle hot-unplug somehow. Maybe those are what would regress if the kernel returned EIO?
Quoted 8 lines
Quoted 8 lines
Documentation/gpu/drm-uapi.rst +8
-0+ +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. + +- If underlying memory disappears, the mmaps are replaced with harmless + zero pages where access does not raise SIGBUS. Reads return zeros, + writes are ignored. +
Quoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 220
Quoted 4 lines
Documentation/gpu/drm-uapi.rst +4
-0+- dmabuf which point to memory that has disappeared are rewritten to + point to harmless zero pages, similar to mmaps. Imports still succeed + both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing any dmabuf.Same as above, I would expect drmPrimeHandleToFD and drmPrimeFDToHandle to return
-EIOin case device is detached
I imagined a use case behind this: a Wayland compositor attempting to import a client's dmabuf. The GPU the client was using is hot-unplugged, meaning the dmabufs lose their storage. It is up to the client to handle its rendering GPU disappearance on its own, but also the compositor should not disconnect it because the GPU disappeared. It's not the client's fault the GPU disappeared.
In Wayland, failure to use a wl_buffer is considered a protocol error, and protocol errors are always fatal: the client is disconnected. The rationale is that the compositor must always be able to present the client buffer somehow. If the compositor cannot, then the client did not obey the protocol.
The fallback presentation path in a compositor is usually importing the dmabuf to EGL, to be sampled from OpenGL. Normally the protocol guarantees that this works, so any failure to do so is a protocol violation. But if the GPU used by the client suddenly disappears and the imports start to fail, that is interpreted as a protocol violation unless the compositor can see why the import failed. Since the import is done via EGL, getting the right error code plumbed through from libdrm functions to the EGL caller would be a hassle. I don't see any error code in EGL_EXT_image_dma_buf_import reserved for "the dmabuf storage was hot-unplugged", and I doubt there is anything exclusively for only that in the EGL base spec either.
The cost of lying that the import worked is that the compositor will paint black or transparent where the window was supposed to be. It's a graphical glitch that is contrary to the Wayland design principles, but in this case a glitch is unavoidable: even if the compositor knew this buffer is now bad, what would it paint instead? It has nothing else to paint from. I'm assuming the compositor is using a different GPU than what disappeared.
Ideally, the client will eventually react to losing the GPU and either crash, quit, or switch its rendering to something that works which simply gives the compositor a new, working buffer without losing any window state in the process. If we risk the compositor disconnecting the client, then the client might not recover even if it wanted to.
Quoted 9 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 225
Quoted 5 lines
Documentation/gpu/drm-uapi.rst +5
-0+ +- Render ioctls return EIO which is then handled in userspace drivers, + e.g. Mesa, to have the device disappearance handled in the way + specified for each API (OpenGL, GL ES: GL_KHR_robustness; + Vulkan: VK_ERROR_DEVICE_LOST; etc.)As far as I understood from our discussion we would reject any IOCTL aimed at a device which is gone and not only render ioctls.
Daniel?
Thanks,
pq
Quoted 20 lines
Andrey
Quoted 16 lines
Documentation/gpu/drm-uapi.rst +14
-0+ +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like Mesa produces. Signal +handlers are not composable, you can't have different handlers for GPU1 +and GPU2 from different vendors, and a third handler for mmapped regular +files. Threads cause additional pain with signal handling as well. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + .. _drm_driver_ioctl:
Quoted 1 line
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 242
Quoted 1 line
Documentation/gpu/drm-uapi.rst +0
-0IOCTL Support on Device Nodes
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 94 lines · Pekka Paalanen
Quoted 80 lines · On Tue, 19 May 2020 10:37:12 -0400 Andrey Grodzovsky…
Thanks for the summary, does put things in order and makes it easier to comprehend all the TODOs, some questions bellow
Quoted 76 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Ce8e13dc4c85648e5fcd408d7fbdc5f2b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637254796242596783&sdata=bA%2FAy3VGvzNqmV1kGigNROSZQEws2E98JibDxvEICNs%3D&reserved=0 .Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>---
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 75 ++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)Documentation/gpu/drm-uapi.rst +41
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,79 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May 13). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some cases specific +ioctls returning EIO. + +This goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was + in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected. +Quoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 208
Quoted 3 lines
Documentation/gpu/drm-uapi.rst +3
-0+- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success.Why wouldn't we return
-EIOfor the atommic commit IOTCL/legasy pflip and modeset ioctls here same way as you suggested returning-EIOfor render ioctl ?Hi,
Quoted 8 lines
that is more of a question for Daniel Vetter than me. I believe he is worried that userspace will get the error handling horribly wrong anyway, because it needs to be handled in every single display server project. Render ioctl errors OTOH need to be handled only in the corresponding Mesa or other userspace driver, and for render there are API (OpenGL/EGL, Vulkan) specs that say how it must be handled to fill the API contract. Because of the API contract, those are more likely to have reasonable error handling in place already.
Yup pretty much. Atm compositors expect an -EINVAL (especially for TEST_ONLY), some cope with the semi-spurious -EBUSY we still throw around, but I expect most will just die if the get an -ENOMEM or -EIO or really anything.
I think one area where we've historically thrown some spurious errors is also vblank ioctls, since they don't take full locks and sometimes the kernel needs to sneak in a modeset for some reason (gpu recovery, dp link recovery, ...).
Either way I don't think there's anything the compositor can do than just ignore the error and carry on.
Quoted 16 lines
I first thought it would be obvious for at least atomic commits to fail appropriately, but then thinking again, it will only introduce new failures that are currently very hard to test for (VKMS to the rescue), and userspace would need to have code to correctly bail out for EIO rather than attempt fallbacks. The uevent telling the device is gone is going to come anyway. In between the device disappearance and uevent handling, it doesn't matter much what happens: the device is abruptly gone and the user will see a glitch anyway - he knows, he just yanked out the cable.
To me this decision is just to make life in userspace easier by not introducing a whole new type of error. However, if the kernel did return EIO instead, it would be impossible for me to see that as a kernel regression because the old behaviour was probably explosions everywhere anyway.
Quoted 3 lines
I heard that UDL and maybe some other drivers in the kernel already handle hot-unplug somehow. Maybe those are what would regress if the kernel returned EIO?
I think most of the not-explodey drivers currently just (mostly?) it all hw errors. You kinda have to if you actually want to deal with hotunplug, stuff can disappear for any moment without warning anyway. Starting to just skip everything is the simplest approach, plus in large parts of the modeset helpers we don't even return values - kernel drivers suck as much at handling the unexpected errors as userspace :-)
-Daniel
Quoted 96 lines
Quoted 18 lines
Quoted 4 lines
Documentation/gpu/drm-uapi.rst +2
-0+ +- Pending non-blocking KMS operations deliver the DRM events userspaceDocumentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 220
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+ is expecting. + +- If underlying memory disappears, the mmaps are replaced with harmless + zero pages where access does not raise SIGBUS. Reads return zeros, + writes are ignored. + +- dmabuf which point to memory that has disappeared are rewritten to + point to harmless zero pages, similar to mmaps. Imports still succeed + both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing any dmabuf.Same as above, I would expect drmPrimeHandleToFD and drmPrimeFDToHandle to return
-EIOin case device is detachedI imagined a use case behind this: a Wayland compositor attempting to import a client's dmabuf. The GPU the client was using is hot-unplugged, meaning the dmabufs lose their storage. It is up to the client to handle its rendering GPU disappearance on its own, but also the compositor should not disconnect it because the GPU disappeared. It's not the client's fault the GPU disappeared.
In Wayland, failure to use a
wl_bufferis considered a protocol error, and protocol errors are always fatal: the client is disconnected. The rationale is that the compositor must always be able to present the client buffer somehow. If the compositor cannot, then the client did not obey the protocol.The fallback presentation path in a compositor is usually importing the dmabuf to EGL, to be sampled from OpenGL. Normally the protocol guarantees that this works, so any failure to do so is a protocol violation. But if the GPU used by the client suddenly disappears and the imports start to fail, that is interpreted as a protocol violation unless the compositor can see why the import failed. Since the import is done via EGL, getting the right error code plumbed through from libdrm functions to the EGL caller would be a hassle. I don't see any error code in
EGL_EXT_image_dma_buf_importreserved for "the dmabuf storage was hot-unplugged", and I doubt there is anything exclusively for only that in the EGL base spec either.The cost of lying that the import worked is that the compositor will paint black or transparent where the window was supposed to be. It's a graphical glitch that is contrary to the Wayland design principles, but in this case a glitch is unavoidable: even if the compositor knew this buffer is now bad, what would it paint instead? It has nothing else to paint from. I'm assuming the compositor is using a different GPU than what disappeared.
Ideally, the client will eventually react to losing the GPU and either crash, quit, or switch its rendering to something that works which simply gives the compositor a new, working buffer without losing any window state in the process. If we risk the compositor disconnecting the client, then the client might not recover even if it wanted to.
Quoted 9 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 225
Quoted 5 lines
Documentation/gpu/drm-uapi.rst +5
-0+ +- Render ioctls return EIO which is then handled in userspace drivers, + e.g. Mesa, to have the device disappearance handled in the way + specified for each API (OpenGL, GL ES: GL_KHR_robustness; + Vulkan: VK_ERROR_DEVICE_LOST; etc.)As far as I understood from our discussion we would reject any IOCTL aimed at a device which is gone and not only render ioctls.
Daniel?
Thanks,
pqQuoted 21 lines
Andrey
Quoted 17 lines
Documentation/gpu/drm-uapi.rst +14
-0+ +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like Mesa produces. Signal +handlers are not composable, you can't have different handlers for GPU1 +and GPU2 from different vendors, and a third handler for mmapped regular +files. Threads cause additional pain with signal handling as well. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 106 lines · Daniel Vetter
Quoted 92 lines · Pekka Paalanen
Quoted 79 lines · On Tue, 19 May 2020 10:37:12 -0400 Andrey Grodzovsky…
Thanks for the summary, does put things in order and makes it easier to comprehend all the TODOs, some questions bellow
Quoted 75 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7CAndrey.Grodzovsky%40amd.com%7Ca4da241ff1e54610d95508d7fcbbcc11%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637255755828209496&sdata=YDFoP2g3z3IiB77sRvAmPB%2Fix%2FV0Mh78YcCSAAlhXdg%3D&reserved=0 .Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>---
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 75 ++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)Documentation/gpu/drm-uapi.rst +40
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,79 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May 13). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some cases specific +ioctls returning EIO. + +This goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was + in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected.Quoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 208
Quoted 4 lines
Documentation/gpu/drm-uapi.rst +4
-0+ +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success.Why wouldn't we return
-EIOfor the atommic commit IOTCL/legasy pflip and modeset ioctls here same way as you suggested returning-EIOfor render ioctl ?Hi,
Quoted 8 lines
that is more of a question for Daniel Vetter than me. I believe he is worried that userspace will get the error handling horribly wrong anyway, because it needs to be handled in every single display server project. Render ioctl errors OTOH need to be handled only in the corresponding Mesa or other userspace driver, and for render there are API (OpenGL/EGL, Vulkan) specs that say how it must be handled to fill the API contract. Because of the API contract, those are more likely to have reasonable error handling in place already.
Yup pretty much. Atm compositors expect an
-EINVAL(especially forTEST_ONLY), some cope with the semi-spurious-EBUSYwe still throw around, but I expect most will just die if the get an-ENOMEMor-EIOor really anything.
Quoted 7 lines
I think one area where we've historically thrown some spurious errors is also vblank ioctls, since they don't take full locks and sometimes the kernel needs to sneak in a modeset for some reason (gpu recovery, dp link recovery, ...).
Either way I don't think there's anything the compositor can do than just ignore the error and carry on.
So currently drm_ioctl will just check for drm_dev_is_unplugged and return -ENODEV at the very beginning of the function (https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_ioctl.c#L825) meaning that currently we blanket failure any IOCTL in case the device is unplagued (ignoring the race condition if the device unplugged after the check). So do we want to remove this check and put it only for render ioctls (which are those ? e.g. for amdgpu there is AMDGPU_CS ioctl) but not for mode setting/dma_buf_import/dma_buf_export ioctls ? What about other types of ioctls which are non of the listed above ?
Andrey
Quoted 116 lines
Quoted 17 lines
I first thought it would be obvious for at least atomic commits to fail appropriately, but then thinking again, it will only introduce new failures that are currently very hard to test for (VKMS to the rescue), and userspace would need to have code to correctly bail out for EIO rather than attempt fallbacks. The uevent telling the device is gone is going to come anyway. In between the device disappearance and uevent handling, it doesn't matter much what happens: the device is abruptly gone and the user will see a glitch anyway - he knows, he just yanked out the cable.
To me this decision is just to make life in userspace easier by not introducing a whole new type of error. However, if the kernel did return EIO instead, it would be impossible for me to see that as a kernel regression because the old behaviour was probably explosions everywhere anyway.
Quoted 3 lines
I heard that UDL and maybe some other drivers in the kernel already handle hot-unplug somehow. Maybe those are what would regress if the kernel returned EIO?
I think most of the not-explodey drivers currently just (mostly?) it all hw errors. You kinda have to if you actually want to deal with hotunplug, stuff can disappear for any moment without warning anyway. Starting to just skip everything is the simplest approach, plus in large parts of the modeset helpers we don't even return values - kernel drivers suck as much at handling the unexpected errors as userspace :-)
-DanielQuoted 88 lines
Quoted 16 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 220
Quoted 13 lines
Documentation/gpu/drm-uapi.rst +12
-0+ +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. + +- If underlying memory disappears, the mmaps are replaced with harmless + zero pages where access does not raise SIGBUS. Reads return zeros, + writes are ignored. + +- dmabuf which point to memory that has disappeared are rewritten to + point to harmless zero pages, similar to mmaps. Imports still succeed + both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing any dmabuf.Same as above, I would expect drmPrimeHandleToFD and drmPrimeFDToHandle to return
-EIOin case device is detachedI imagined a use case behind this: a Wayland compositor attempting to import a client's dmabuf. The GPU the client was using is hot-unplugged, meaning the dmabufs lose their storage. It is up to the client to handle its rendering GPU disappearance on its own, but also the compositor should not disconnect it because the GPU disappeared. It's not the client's fault the GPU disappeared.
In Wayland, failure to use a
wl_bufferis considered a protocol error, and protocol errors are always fatal: the client is disconnected. The rationale is that the compositor must always be able to present the client buffer somehow. If the compositor cannot, then the client did not obey the protocol.The fallback presentation path in a compositor is usually importing the dmabuf to EGL, to be sampled from OpenGL. Normally the protocol guarantees that this works, so any failure to do so is a protocol violation. But if the GPU used by the client suddenly disappears and the imports start to fail, that is interpreted as a protocol violation unless the compositor can see why the import failed. Since the import is done via EGL, getting the right error code plumbed through from libdrm functions to the EGL caller would be a hassle. I don't see any error code in
EGL_EXT_image_dma_buf_importreserved for "the dmabuf storage was hot-unplugged", and I doubt there is anything exclusively for only that in the EGL base spec either.The cost of lying that the import worked is that the compositor will paint black or transparent where the window was supposed to be. It's a graphical glitch that is contrary to the Wayland design principles, but in this case a glitch is unavoidable: even if the compositor knew this buffer is now bad, what would it paint instead? It has nothing else to paint from. I'm assuming the compositor is using a different GPU than what disappeared.
Ideally, the client will eventually react to losing the GPU and either crash, quit, or switch its rendering to something that works which simply gives the compositor a new, working buffer without losing any window state in the process. If we risk the compositor disconnecting the client, then the client might not recover even if it wanted to.
Quoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 225
Quoted 5 lines
Documentation/gpu/drm-uapi.rst +5
-0+ +- Render ioctls return EIO which is then handled in userspace drivers, + e.g. Mesa, to have the device disappearance handled in the way + specified for each API (OpenGL, GL ES: GL_KHR_robustness; + Vulkan: VK_ERROR_DEVICE_LOST; etc.)As far as I understood from our discussion we would reject any IOCTL aimed at a device which is gone and not only render ioctls.
Daniel?
Thanks,
pqQuoted 19 lines
Andrey
Quoted 16 lines
Documentation/gpu/drm-uapi.rst +14
-0+ +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like Mesa produces. Signal +handlers are not composable, you can't have different handlers for GPU1 +and GPU2 from different vendors, and a third handler for mmapped regular +files. Threads cause additional pain with signal handling as well. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + .. _drm_driver_ioctl:
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 242
Quoted 1 line
Documentation/gpu/drm-uapi.rst +0
-0IOCTL Support on Device Nodes
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Ping on the question bellow
Andrey
Quoted 128 lines · Andrey Grodzovsky
Quoted 126 lines · Daniel Vetter
Quoted 107 lines · Pekka Paalanen
Quoted 94 lines · On Tue, 19 May 2020 10:37:12 -0400 Andrey Grodzovsky…
Thanks for the summary, does put things in order and makes it easier to comprehend all the TODOs, some questions bellow
Quoted 90 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7CAndrey.Grodzovsky%40amd.com%7Ca4da241ff1e54610d95508d7fcbbcc11%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637255755828209496&sdata=YDFoP2g3z3IiB77sRvAmPB%2Fix%2FV0Mh78YcCSAAlhXdg%3D&reserved=0 .Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>---
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
---
Documentation/gpu/drm-uapi.rst| 75
++++++++++++++++++++++++++++++++++1 file changed, 75 insertions(+)Documentation/gpu/drm-uapi.rst +2
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. +===================
Userland interfaces
===================Documentation/gpu/drm-uapi.rst +0
-0@@ -162,6 +164,79 @@ other hand, a driver requires shared statebetween clients which is
visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes.
+Device Hot-Unplug
+=================Documentation/gpu/drm-uapi.rst +6
-0+ +.. note:: + The following is the plan. Implementation is not there yet + (2020 May 13). + +Graphics devices (display and/or render) may be connected via USB(e.g.
+display adapters or docking stations) or Thunderbolt (e.g. eGPU). An endDocumentation/gpu/drm-uapi.rst +2
-0+user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash.Any
+damage from hot-unplugging a DRM device needs to be limited as much as
+possible and userspace must be given the chance to handle it if it wants
+to. Ideally, unplugging a DRM device still lets a desktop to continue
+running, but that is going to need explicit support throughout the wholeDocumentation/gpu/drm-uapi.rst +2
-0+graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications andlibraries.
Documentation/gpu/drm-uapi.rst +3
-0+ +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of adriver
Documentation/gpu/drm-uapi.rst +3
-0+from the physical device. + +In other words, from userspace perspective everything needs tokeep on
+working more or less, until userspace stops using the disappeared DRMDocumentation/gpu/drm-uapi.rst +2
-0+device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some casesspecific
Documentation/gpu/drm-uapi.rst +3
-0+ioctls returning EIO. + +This goal raises at least the following requirements for thekernel and
Documentation/gpu/drm-uapi.rst +3
-0+drivers: + +- The kernel must not hang, crash or oops, no matter whatuserspace was
+ in the middle of doing when the device disappeared.Documentation/gpu/drm-uapi.rst +5
-0+ +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected.Quoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 208
Quoted 4 lines
Documentation/gpu/drm-uapi.rst +4
-0+ +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success.Why wouldn't we return
-EIOfor the atommic commit IOTCL/legasy pflip and modeset ioctls here same way as you suggested returning-EIOfor render ioctl ?Hi,
Quoted 8 lines
that is more of a question for Daniel Vetter than me. I believe he is worried that userspace will get the error handling horribly wrong anyway, because it needs to be handled in every single display server project. Render ioctl errors OTOH need to be handled only in the corresponding Mesa or other userspace driver, and for render there are API (OpenGL/EGL, Vulkan) specs that say how it must be handled to fill the API contract. Because of the API contract, those are more likely to have reasonable error handling in place already.
Yup pretty much. Atm compositors expect an
-EINVAL(especially forTEST_ONLY), some cope with the semi-spurious-EBUSYwe still throw around, but I expect most will just die if the get an-ENOMEMor-EIOor really anything.I think one area where we've historically thrown some spurious errors is also vblank ioctls, since they don't take full locks and sometimes the kernel needs to sneak in a modeset for some reason (gpu recovery, dp link recovery, ...).
Quoted 135 lines
Quoted 2 lines
Either way I don't think there's anything the compositor can do than just ignore the error and carry on.
So currently
drm_ioctlwill just check fordrm_dev_is_unpluggedand return-ENODEVat the very beginning of the function (https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_ioctl.c#L825) meaning that currently we blanket failure any IOCTL in case the device is unplagued (ignoring the race condition if the device unplugged after the check). So do we want to remove this check and put it only for render ioctls (which are those ? e.g. for amdgpu there isAMDGPU_CSioctl) but not for mode setting/dma_buf_import/dma_buf_export ioctls ? What about other types of ioctls which are non of the listed above ?Andrey
Quoted 126 lines
Quoted 17 lines
I first thought it would be obvious for at least atomic commits to fail appropriately, but then thinking again, it will only introduce new failures that are currently very hard to test for (VKMS to the rescue), and userspace would need to have code to correctly bail out for EIO rather than attempt fallbacks. The uevent telling the device is gone is going to come anyway. In between the device disappearance and uevent handling, it doesn't matter much what happens: the device is abruptly gone and the user will see a glitch anyway - he knows, he just yanked out the cable.
To me this decision is just to make life in userspace easier by not introducing a whole new type of error. However, if the kernel did return EIO instead, it would be impossible for me to see that as a kernel regression because the old behaviour was probably explosions everywhere anyway.
Quoted 3 lines
I heard that UDL and maybe some other drivers in the kernel already handle hot-unplug somehow. Maybe those are what would regress if the kernel returned EIO?
I think most of the not-explodey drivers currently just (mostly?) it all hw errors. You kinda have to if you actually want to deal with hotunplug, stuff can disappear for any moment without warning anyway. Starting to just skip everything is the simplest approach, plus in large parts of the modeset helpers we don't even return values - kernel drivers suck as much at handling the unexpected errors as userspace :-)
-DanielQuoted 96 lines
Quoted 11 lines
Quoted 11 lines
+
+- Pending non-blocking KMS operations deliver the DRM events userspace
+ is expecting.
+
+- If underlying memory disappears, the mmaps are replaced with harmless
+ zero pages where access does not raise SIGBUS. Reads return zeros, + writes are ignored.
+Quoted 8 lines
Quoted 5 lines
+- dmabuf which point to memory that has disappeared are rewritten to + point to harmless zero pages, similar to mmaps. Imports still succeed
+ both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing any dmabuf.Same as above, I would expect drmPrimeHandleToFD and drmPrimeFDToHandle to return
-EIOin case device is detachedI imagined a use case behind this: a Wayland compositor attempting to import a client's dmabuf. The GPU the client was using is hot-unplugged, meaning the dmabufs lose their storage. It is up to the client to handle its rendering GPU disappearance on its own, but also the compositor should not disconnect it because the GPU disappeared. It's not the client's fault the GPU disappeared.
In Wayland, failure to use a
wl_bufferis considered a protocol error, and protocol errors are always fatal: the client is disconnected. The rationale is that the compositor must always be able to present the client buffer somehow. If the compositor cannot, then the client did not obey the protocol.The fallback presentation path in a compositor is usually importing the dmabuf to EGL, to be sampled from OpenGL. Normally the protocol guarantees that this works, so any failure to do so is a protocol violation. But if the GPU used by the client suddenly disappears and the imports start to fail, that is interpreted as a protocol violation unless the compositor can see why the import failed. Since the import is done via EGL, getting the right error code plumbed through from libdrm functions to the EGL caller would be a hassle. I don't see any error code in
EGL_EXT_image_dma_buf_importreserved for "the dmabuf storage was hot-unplugged", and I doubt there is anything exclusively for only that in the EGL base spec either.The cost of lying that the import worked is that the compositor will paint black or transparent where the window was supposed to be. It's a graphical glitch that is contrary to the Wayland design principles, but in this case a glitch is unavoidable: even if the compositor knew this buffer is now bad, what would it paint instead? It has nothing else to paint from. I'm assuming the compositor is using a different GPU than what disappeared.
Ideally, the client will eventually react to losing the GPU and either crash, quit, or switch its rendering to something that works which simply gives the compositor a new, working buffer without losing any window state in the process. If we risk the compositor disconnecting the client, then the client might not recover even if it wanted to.
Quoted 9 lines
Quoted 6 lines
+
+- Render ioctls return EIO which is then handled in userspace drivers,
+ e.g. Mesa, to have the device disappearance handled in the way + specified for each API (OpenGL, GL ES:GL_KHR_robustness; + Vulkan:VK_ERROR_DEVICE_LOST; etc.)As far as I understood from our discussion we would reject any IOCTL aimed at a device which is gone and not only render ioctls.
Daniel?
Thanks,
pqQuoted 23 lines
Andrey
Quoted 20 lines
Documentation/gpu/drm-uapi.rst +2
-0+ +Raising SIGBUS is not an option, because userspace cannotrealistically
Documentation/gpu/drm-uapi.rst +3
-0+handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like Mesa produces. Signal +handlers are not composable, you can't have different handlersfor GPU1
+and GPU2 from different vendors, and a third handler for mmapped regularDocumentation/gpu/drm-uapi.rst +3
-0+files. Threads cause additional pain with signal handling as well. + +Only after userspace has closed all relevant DRM device anddmabuf file
Documentation/gpu/drm-uapi.rst +5
-0+descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. +..
_drm_driver_ioctl:
IOCTL Support on Device Nodes
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 116 lines · Andrey Grodzovsky
Quoted 105 lines · Daniel Vetter
Quoted 91 lines · Pekka Paalanen
Quoted 78 lines · On Tue, 19 May 2020 10:37:12 -0400 Andrey Grodzovsky…
Thanks for the summary, does put things in order and makes it easier to comprehend all the TODOs, some questions bellow
Quoted 74 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7CAndrey.Grodzovsky%40amd.com%7Ca4da241ff1e54610d95508d7fcbbcc11%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637255755828209496&sdata=YDFoP2g3z3IiB77sRvAmPB%2Fix%2FV0Mh78YcCSAAlhXdg%3D&reserved=0 .Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>---
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 75 ++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)Documentation/gpu/drm-uapi.rst +40
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,79 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May 13). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some cases specific +ioctls returning EIO. + +This goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was + in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected.Quoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 208
Quoted 4 lines
Documentation/gpu/drm-uapi.rst +4
-0+ +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success.Why wouldn't we return
-EIOfor the atommic commit IOTCL/legasy pflip and modeset ioctls here same way as you suggested returning-EIOfor render ioctl ?Hi,
Quoted 8 lines
that is more of a question for Daniel Vetter than me. I believe he is worried that userspace will get the error handling horribly wrong anyway, because it needs to be handled in every single display server project. Render ioctl errors OTOH need to be handled only in the corresponding Mesa or other userspace driver, and for render there are API (OpenGL/EGL, Vulkan) specs that say how it must be handled to fill the API contract. Because of the API contract, those are more likely to have reasonable error handling in place already.
Yup pretty much. Atm compositors expect an
-EINVAL(especially forTEST_ONLY), some cope with the semi-spurious-EBUSYwe still throw around, but I expect most will just die if the get an-ENOMEMor-EIOor really anything.Quoted 7 lines
I think one area where we've historically thrown some spurious errors is also vblank ioctls, since they don't take full locks and sometimes the kernel needs to sneak in a modeset for some reason (gpu recovery, dp link recovery, ...).
Either way I don't think there's anything the compositor can do than just ignore the error and carry on.
Quoted 8 lines
So currently
drm_ioctlwill just check fordrm_dev_is_unpluggedand return-ENODEVat the very beginning of the function (https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_ioctl.c#L825)
meaning that currently we blanket failure any IOCTL in case the device is
unplagued (ignoring the race condition if the device unplugged after the
check). So do we want to remove this check and put it only for render ioctls
(which are those ? e.g. for amdgpu there isAMDGPU_CSioctl) but not for
mode setting/dma_buf_import/dma_buf_export ioctls ? What about other types
of ioctls which are non of the listed above ?
Hm right, and this goes back all the way to first usb udl support:
commit 2c07a21d6fb0be47fda696a618b726ea258ed1dd
Author: Dave Airlie <[email protected]> Date: Mon Feb 20 14:18:07 2012 +0000
drm: add core support for unplugging a device (v2)
So I guess we need to change the uapi docs to explain that. Not sure everyone can cope, but I guess they better do. Since the use-case back then was just udl, so simple legacy kms only, the damage was probably rather limited. I'm not sure we can get away with that now, where kms code has spread to funny places likey vulkan winsys code.
Or maybe we want a file priv flag you can set along the lines of "give me less shitty hotunplug semantics for ioctls". Or maybe we can just change the semantics, not crashing&burning shouldn't cause a regression :-)
For everything else (mmap, dma-buf fd, sync_file fd, syncobj fd) I think the discussion is still more or less accurate.
Pekka, any thoughts?
-Daniel
Quoted 122 lines
Andrey
Quoted 118 lines
Quoted 17 lines
I first thought it would be obvious for at least atomic commits to fail appropriately, but then thinking again, it will only introduce new failures that are currently very hard to test for (VKMS to the rescue), and userspace would need to have code to correctly bail out for EIO rather than attempt fallbacks. The uevent telling the device is gone is going to come anyway. In between the device disappearance and uevent handling, it doesn't matter much what happens: the device is abruptly gone and the user will see a glitch anyway - he knows, he just yanked out the cable.
To me this decision is just to make life in userspace easier by not introducing a whole new type of error. However, if the kernel did return EIO instead, it would be impossible for me to see that as a kernel regression because the old behaviour was probably explosions everywhere anyway.
Quoted 3 lines
I heard that UDL and maybe some other drivers in the kernel already handle hot-unplug somehow. Maybe those are what would regress if the kernel returned EIO?
I think most of the not-explodey drivers currently just (mostly?) it all hw errors. You kinda have to if you actually want to deal with hotunplug, stuff can disappear for any moment without warning anyway. Starting to just skip everything is the simplest approach, plus in large parts of the modeset helpers we don't even return values - kernel drivers suck as much at handling the unexpected errors as userspace :-)
-DanielQuoted 88 lines
Quoted 16 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 220
Quoted 13 lines
Documentation/gpu/drm-uapi.rst +12
-0+ +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. + +- If underlying memory disappears, the mmaps are replaced with harmless + zero pages where access does not raise SIGBUS. Reads return zeros, + writes are ignored. + +- dmabuf which point to memory that has disappeared are rewritten to + point to harmless zero pages, similar to mmaps. Imports still succeed + both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing any dmabuf.Same as above, I would expect drmPrimeHandleToFD and drmPrimeFDToHandle to return
-EIOin case device is detachedI imagined a use case behind this: a Wayland compositor attempting to import a client's dmabuf. The GPU the client was using is hot-unplugged, meaning the dmabufs lose their storage. It is up to the client to handle its rendering GPU disappearance on its own, but also the compositor should not disconnect it because the GPU disappeared. It's not the client's fault the GPU disappeared.
In Wayland, failure to use a
wl_bufferis considered a protocol error, and protocol errors are always fatal: the client is disconnected. The rationale is that the compositor must always be able to present the client buffer somehow. If the compositor cannot, then the client did not obey the protocol.The fallback presentation path in a compositor is usually importing the dmabuf to EGL, to be sampled from OpenGL. Normally the protocol guarantees that this works, so any failure to do so is a protocol violation. But if the GPU used by the client suddenly disappears and the imports start to fail, that is interpreted as a protocol violation unless the compositor can see why the import failed. Since the import is done via EGL, getting the right error code plumbed through from libdrm functions to the EGL caller would be a hassle. I don't see any error code in
EGL_EXT_image_dma_buf_importreserved for "the dmabuf storage was hot-unplugged", and I doubt there is anything exclusively for only that in the EGL base spec either.The cost of lying that the import worked is that the compositor will paint black or transparent where the window was supposed to be. It's a graphical glitch that is contrary to the Wayland design principles, but in this case a glitch is unavoidable: even if the compositor knew this buffer is now bad, what would it paint instead? It has nothing else to paint from. I'm assuming the compositor is using a different GPU than what disappeared.
Ideally, the client will eventually react to losing the GPU and either crash, quit, or switch its rendering to something that works which simply gives the compositor a new, working buffer without losing any window state in the process. If we risk the compositor disconnecting the client, then the client might not recover even if it wanted to.
Quoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 225
Quoted 5 lines
Documentation/gpu/drm-uapi.rst +5
-0+ +- Render ioctls return EIO which is then handled in userspace drivers, + e.g. Mesa, to have the device disappearance handled in the way + specified for each API (OpenGL, GL ES: GL_KHR_robustness; + Vulkan: VK_ERROR_DEVICE_LOST; etc.)As far as I understood from our discussion we would reject any IOCTL aimed at a device which is gone and not only render ioctls.
Daniel?
Thanks,
pqQuoted 19 lines
Andrey
Quoted 16 lines
Documentation/gpu/drm-uapi.rst +14
-0+ +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like Mesa produces. Signal +handlers are not composable, you can't have different handlers for GPU1 +and GPU2 from different vendors, and a third handler for mmapped regular +files. Threads cause additional pain with signal handling as well. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 140 lines · On Mon, 25 May 2020 16:28:04 +0200 Daniel Vetter <[email protected]> wrote:
Quoted 116 lines · Andrey Grodzovsky
Quoted 105 lines · Daniel Vetter
Quoted 91 lines · Pekka Paalanen
Quoted 78 lines · On Tue, 19 May 2020 10:37:12 -0400 Andrey Grodzovsky…
Thanks for the summary, does put things in order and makes it easier to comprehend all the TODOs, some questions bellow
Quoted 74 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7CAndrey.Grodzovsky%40amd.com%7Ca4da241ff1e54610d95508d7fcbbcc11%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637255755828209496&sdata=YDFoP2g3z3IiB77sRvAmPB%2Fix%2FV0Mh78YcCSAAlhXdg%3D&reserved=0 .Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>---
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 75 ++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)Documentation/gpu/drm-uapi.rst +40
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,79 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May 13). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some cases specific +ioctls returning EIO. + +This goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was + in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected.Quoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 208
Quoted 4 lines
Documentation/gpu/drm-uapi.rst +4
-0+ +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success.Why wouldn't we return
-EIOfor the atommic commit IOTCL/legasy pflip and modeset ioctls here same way as you suggested returning-EIOfor render ioctl ?Hi,
Quoted 8 lines
that is more of a question for Daniel Vetter than me. I believe he is worried that userspace will get the error handling horribly wrong anyway, because it needs to be handled in every single display server project. Render ioctl errors OTOH need to be handled only in the corresponding Mesa or other userspace driver, and for render there are API (OpenGL/EGL, Vulkan) specs that say how it must be handled to fill the API contract. Because of the API contract, those are more likely to have reasonable error handling in place already.
Yup pretty much. Atm compositors expect an
-EINVAL(especially forTEST_ONLY), some cope with the semi-spurious-EBUSYwe still throw around, but I expect most will just die if the get an-ENOMEMor-EIOor really anything.Quoted 7 lines
I think one area where we've historically thrown some spurious errors is also vblank ioctls, since they don't take full locks and sometimes the kernel needs to sneak in a modeset for some reason (gpu recovery, dp link recovery, ...).
Either way I don't think there's anything the compositor can do than just ignore the error and carry on.
Quoted 8 lines
So currently
drm_ioctlwill just check fordrm_dev_is_unpluggedand return-ENODEVat the very beginning of the function (https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_ioctl.c#L825)
meaning that currently we blanket failure any IOCTL in case the device is
unplagued (ignoring the race condition if the device unplugged after the
check). So do we want to remove this check and put it only for render ioctls
(which are those ? e.g. for amdgpu there isAMDGPU_CSioctl) but not for
mode setting/dma_buf_import/dma_buf_export ioctls ? What about other types
of ioctls which are non of the listed above ?Hm right, and this goes back all the way to first usb udl support:
commit
2c07a21d6fb0be47fda696a618b726ea258ed1ddAuthor: Dave Airlie <[email protected]> Date: Mon Feb 20 14:18:07 2012 +0000drm: add core support for unplugging a device (v2)
So I guess we need to change the uapi docs to explain that. Not sure everyone can cope, but I guess they better do. Since the use-case back then was just udl, so simple legacy kms only, the damage was probably rather limited. I'm not sure we can get away with that now, where kms code has spread to funny places likey vulkan winsys code.
Quoted 8 lines
Or maybe we want a file priv flag you can set along the lines of "give me less shitty hotunplug semantics for ioctls". Or maybe we can just change the semantics, not crashing&burning shouldn't cause a regression :-)
For everything else (mmap, dma-buf fd,
sync_filefd, syncobj fd) I think the discussion is still more or less accurate.Pekka, any thoughts?
Hi,
is ENODEV unique to this particular failure?
Returning errors instead of faking success was my first idea, but you already convinced me that faking is at least as good if not better. :-)
So as long as the error code returned is unique to hot-unplug or other "oops, the device is gone" conditions, I think I'm fine. Weston does not handle ENODEV any way, it never did, and it certainly cannot be called a kernel regression.
As a Weston developer, I don't mind adding checks for ENODEV. But if I don't have to, even better. Weston is going to need more code to handle DRM device unplug in any case.
Sorry, no preference from me. ;-)
I do agree that replacing ENODEV with fake success is hard to imagine regressing anything. It's something you can do in the kernel at any time easily, but going from fake success to error is going to be painful. Maybe don't change things until there is a good reason to?
We need a kernel that doesn't crash before we can properly test what would be best for userspace, fake or error.
Thanks,
pq
Quoted 125 lines
-Daniel
Quoted 122 lines
Andrey
Quoted 118 lines
Quoted 17 lines
I first thought it would be obvious for at least atomic commits to fail appropriately, but then thinking again, it will only introduce new failures that are currently very hard to test for (VKMS to the rescue), and userspace would need to have code to correctly bail out for EIO rather than attempt fallbacks. The uevent telling the device is gone is going to come anyway. In between the device disappearance and uevent handling, it doesn't matter much what happens: the device is abruptly gone and the user will see a glitch anyway - he knows, he just yanked out the cable.
To me this decision is just to make life in userspace easier by not introducing a whole new type of error. However, if the kernel did return EIO instead, it would be impossible for me to see that as a kernel regression because the old behaviour was probably explosions everywhere anyway.
Quoted 3 lines
I heard that UDL and maybe some other drivers in the kernel already handle hot-unplug somehow. Maybe those are what would regress if the kernel returned EIO?
I think most of the not-explodey drivers currently just (mostly?) it all hw errors. You kinda have to if you actually want to deal with hotunplug, stuff can disappear for any moment without warning anyway. Starting to just skip everything is the simplest approach, plus in large parts of the modeset helpers we don't even return values - kernel drivers suck as much at handling the unexpected errors as userspace :-)
-DanielQuoted 88 lines
Quoted 16 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 220
Quoted 13 lines
Documentation/gpu/drm-uapi.rst +12
-0+ +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. + +- If underlying memory disappears, the mmaps are replaced with harmless + zero pages where access does not raise SIGBUS. Reads return zeros, + writes are ignored. + +- dmabuf which point to memory that has disappeared are rewritten to + point to harmless zero pages, similar to mmaps. Imports still succeed + both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing any dmabuf.Same as above, I would expect drmPrimeHandleToFD and drmPrimeFDToHandle to return
-EIOin case device is detachedI imagined a use case behind this: a Wayland compositor attempting to import a client's dmabuf. The GPU the client was using is hot-unplugged, meaning the dmabufs lose their storage. It is up to the client to handle its rendering GPU disappearance on its own, but also the compositor should not disconnect it because the GPU disappeared. It's not the client's fault the GPU disappeared.
In Wayland, failure to use a
wl_bufferis considered a protocol error, and protocol errors are always fatal: the client is disconnected. The rationale is that the compositor must always be able to present the client buffer somehow. If the compositor cannot, then the client did not obey the protocol.The fallback presentation path in a compositor is usually importing the dmabuf to EGL, to be sampled from OpenGL. Normally the protocol guarantees that this works, so any failure to do so is a protocol violation. But if the GPU used by the client suddenly disappears and the imports start to fail, that is interpreted as a protocol violation unless the compositor can see why the import failed. Since the import is done via EGL, getting the right error code plumbed through from libdrm functions to the EGL caller would be a hassle. I don't see any error code in
EGL_EXT_image_dma_buf_importreserved for "the dmabuf storage was hot-unplugged", and I doubt there is anything exclusively for only that in the EGL base spec either.The cost of lying that the import worked is that the compositor will paint black or transparent where the window was supposed to be. It's a graphical glitch that is contrary to the Wayland design principles, but in this case a glitch is unavoidable: even if the compositor knew this buffer is now bad, what would it paint instead? It has nothing else to paint from. I'm assuming the compositor is using a different GPU than what disappeared.
Ideally, the client will eventually react to losing the GPU and either crash, quit, or switch its rendering to something that works which simply gives the compositor a new, working buffer without losing any window state in the process. If we risk the compositor disconnecting the client, then the client might not recover even if it wanted to.
Quoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 225
Quoted 5 lines
Documentation/gpu/drm-uapi.rst +5
-0+ +- Render ioctls return EIO which is then handled in userspace drivers, + e.g. Mesa, to have the device disappearance handled in the way + specified for each API (OpenGL, GL ES: GL_KHR_robustness; + Vulkan: VK_ERROR_DEVICE_LOST; etc.)As far as I understood from our discussion we would reject any IOCTL aimed at a device which is gone and not only render ioctls.
Daniel?
Thanks,
pqQuoted 19 lines
Andrey
Quoted 16 lines
Documentation/gpu/drm-uapi.rst +14
-0+ +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like Mesa produces. Signal +handlers are not composable, you can't have different handlers for GPU1 +and GPU2 from different vendors, and a third handler for mmapped regular +files. Threads cause additional pain with signal handling as well. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 147 lines · Pekka Paalanen
Quoted 144 lines · On Mon, 25 May 2020 16:28:04 +0200 Daniel Vetter <[email protected]> wrote:
Quoted 116 lines · Andrey Grodzovsky
Quoted 105 lines · Daniel Vetter
Quoted 91 lines · Pekka Paalanen
Quoted 78 lines · On Tue, 19 May 2020 10:37:12 -0400 Andrey Grodzovsky…
Thanks for the summary, does put things in order and makes it easier to comprehend all the TODOs, some questions bellow
Quoted 74 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7CAndrey.Grodzovsky%40amd.com%7Ca4da241ff1e54610d95508d7fcbbcc11%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637255755828209496&sdata=YDFoP2g3z3IiB77sRvAmPB%2Fix%2FV0Mh78YcCSAAlhXdg%3D&reserved=0 .Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>---
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 75 ++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)Documentation/gpu/drm-uapi.rst +40
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,79 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May 13). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some cases specific +ioctls returning EIO. + +This goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was + in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected.Quoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 207
Quoted 4 lines
Documentation/gpu/drm-uapi.rst +4
-0+ +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success.Why wouldn't we return
-EIOfor the atommic commit IOTCL/legasy pflip and modeset ioctls here same way as you suggested returning-EIOfor render ioctl ?Hi,
Quoted 8 lines
that is more of a question for Daniel Vetter than me. I believe he is worried that userspace will get the error handling horribly wrong anyway, because it needs to be handled in every single display server project. Render ioctl errors OTOH need to be handled only in the corresponding Mesa or other userspace driver, and for render there are API (OpenGL/EGL, Vulkan) specs that say how it must be handled to fill the API contract. Because of the API contract, those are more likely to have reasonable error handling in place already.
Yup pretty much. Atm compositors expect an
-EINVAL(especially forTEST_ONLY), some cope with the semi-spurious-EBUSYwe still throw around, but I expect most will just die if the get an-ENOMEMor-EIOor really anything.Quoted 7 lines
I think one area where we've historically thrown some spurious errors is also vblank ioctls, since they don't take full locks and sometimes the kernel needs to sneak in a modeset for some reason (gpu recovery, dp link recovery, ...).
Either way I don't think there's anything the compositor can do than just ignore the error and carry on.
Quoted 8 lines
So currently
drm_ioctlwill just check fordrm_dev_is_unpluggedand return-ENODEVat the very beginning of the function (https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_ioctl.c#L825)
meaning that currently we blanket failure any IOCTL in case the device is
unplagued (ignoring the race condition if the device unplugged after the
check). So do we want to remove this check and put it only for render ioctls
(which are those ? e.g. for amdgpu there isAMDGPU_CSioctl) but not for
mode setting/dma_buf_import/dma_buf_export ioctls ? What about other types
of ioctls which are non of the listed above ?Hm right, and this goes back all the way to first usb udl support:
commit
2c07a21d6fb0be47fda696a618b726ea258ed1ddAuthor: Dave Airlie <[email protected]> Date: Mon Feb 20 14:18:07 2012 +0000drm: add core support for unplugging a device (v2)
So I guess we need to change the uapi docs to explain that. Not sure everyone can cope, but I guess they better do. Since the use-case back then was just udl, so simple legacy kms only, the damage was probably rather limited. I'm not sure we can get away with that now, where kms code has spread to funny places likey vulkan winsys code.
Or maybe we want a file priv flag you can set along the lines of "give me less shitty hotunplug semantics for ioctls". Or maybe we can just change the semantics, not crashing&burning shouldn't cause a regression :-)
Quoted 8 lines
Quoted 4 lines
For everything else (mmap, dma-buf fd,
sync_filefd, syncobj fd) I think the discussion is still more or less accurate.Pekka, any thoughts?
Hi,
is ENODEV unique to this particular failure?
Not really sure, we'd need to audit all of drm ...
Quoted 14 lines
Returning errors instead of faking success was my first idea, but you already convinced me that faking is at least as good if not better. :-)
So as long as the error code returned is unique to hot-unplug or other "oops, the device is gone" conditions, I think I'm fine. Weston does not handle ENODEV any way, it never did, and it certainly cannot be called a kernel regression.
As a Weston developer, I don't mind adding checks for ENODEV. But if I don't have to, even better. Weston is going to need more code to handle DRM device unplug in any case.
Sorry, no preference from me. ;-)
Quoted 7 lines
I do agree that replacing ENODEV with fake success is hard to imagine regressing anything. It's something you can do in the kernel at any time easily, but going from fake success to error is going to be painful. Maybe don't change things until there is a good reason to?
We need a kernel that doesn't crash before we can properly test what would be best for userspace, fake or error.
One upshot of faking stuff and only bailing in low-level hw code is that it makes validating the races when you hotunplug easier - if we remove the early bail-out check even an ioctl later on will look like it raced with the hotunplug path in the kernel. So better assurance that things won't blow up badly.
Otoh the early bail out in the top-level ioctl code increases the odds that you'll survive even on a driver that's totally buggy.
So yeah I guess maybe we should just document that currently you get an -ENODEV and maybe have the option mentioned that we might change this going forward. See also
https://dri.freedesktop.org/docs/drm/gpu/drm-uapi.html#recommended-ioctl-return-values
I think that needs an update, at least clarify that "not present" includes "no longer present".
-Daniel
Quoted 132 lines
Thanks,
pqQuoted 125 lines
-Daniel
Quoted 122 lines
Andrey
Quoted 118 lines
Quoted 17 lines
I first thought it would be obvious for at least atomic commits to fail appropriately, but then thinking again, it will only introduce new failures that are currently very hard to test for (VKMS to the rescue), and userspace would need to have code to correctly bail out for EIO rather than attempt fallbacks. The uevent telling the device is gone is going to come anyway. In between the device disappearance and uevent handling, it doesn't matter much what happens: the device is abruptly gone and the user will see a glitch anyway - he knows, he just yanked out the cable.
To me this decision is just to make life in userspace easier by not introducing a whole new type of error. However, if the kernel did return EIO instead, it would be impossible for me to see that as a kernel regression because the old behaviour was probably explosions everywhere anyway.
Quoted 3 lines
I heard that UDL and maybe some other drivers in the kernel already handle hot-unplug somehow. Maybe those are what would regress if the kernel returned EIO?
I think most of the not-explodey drivers currently just (mostly?) it all hw errors. You kinda have to if you actually want to deal with hotunplug, stuff can disappear for any moment without warning anyway. Starting to just skip everything is the simplest approach, plus in large parts of the modeset helpers we don't even return values - kernel drivers suck as much at handling the unexpected errors as userspace :-)
-DanielQuoted 88 lines
Quoted 16 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 219
Quoted 13 lines
Documentation/gpu/drm-uapi.rst +12
-0+ +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. + +- If underlying memory disappears, the mmaps are replaced with harmless + zero pages where access does not raise SIGBUS. Reads return zeros, + writes are ignored. + +- dmabuf which point to memory that has disappeared are rewritten to + point to harmless zero pages, similar to mmaps. Imports still succeed + both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing any dmabuf.Same as above, I would expect drmPrimeHandleToFD and drmPrimeFDToHandle to return
-EIOin case device is detachedI imagined a use case behind this: a Wayland compositor attempting to import a client's dmabuf. The GPU the client was using is hot-unplugged, meaning the dmabufs lose their storage. It is up to the client to handle its rendering GPU disappearance on its own, but also the compositor should not disconnect it because the GPU disappeared. It's not the client's fault the GPU disappeared.
In Wayland, failure to use a
wl_bufferis considered a protocol error, and protocol errors are always fatal: the client is disconnected. The rationale is that the compositor must always be able to present the client buffer somehow. If the compositor cannot, then the client did not obey the protocol.The fallback presentation path in a compositor is usually importing the dmabuf to EGL, to be sampled from OpenGL. Normally the protocol guarantees that this works, so any failure to do so is a protocol violation. But if the GPU used by the client suddenly disappears and the imports start to fail, that is interpreted as a protocol violation unless the compositor can see why the import failed. Since the import is done via EGL, getting the right error code plumbed through from libdrm functions to the EGL caller would be a hassle. I don't see any error code in
EGL_EXT_image_dma_buf_importreserved for "the dmabuf storage was hot-unplugged", and I doubt there is anything exclusively for only that in the EGL base spec either.The cost of lying that the import worked is that the compositor will paint black or transparent where the window was supposed to be. It's a graphical glitch that is contrary to the Wayland design principles, but in this case a glitch is unavoidable: even if the compositor knew this buffer is now bad, what would it paint instead? It has nothing else to paint from. I'm assuming the compositor is using a different GPU than what disappeared.
Ideally, the client will eventually react to losing the GPU and either crash, quit, or switch its rendering to something that works which simply gives the compositor a new, working buffer without losing any window state in the process. If we risk the compositor disconnecting the client, then the client might not recover even if it wanted to.
Quoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 224
Quoted 5 lines
Documentation/gpu/drm-uapi.rst +5
-0+ +- Render ioctls return EIO which is then handled in userspace drivers, + e.g. Mesa, to have the device disappearance handled in the way + specified for each API (OpenGL, GL ES: GL_KHR_robustness; + Vulkan: VK_ERROR_DEVICE_LOST; etc.)As far as I understood from our discussion we would reject any IOCTL aimed at a device which is gone and not only render ioctls.
Daniel?
Thanks,
pqQuoted 19 lines
Andrey
Quoted 16 lines
Documentation/gpu/drm-uapi.rst +14
-0+ +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like Mesa produces. Signal +handlers are not composable, you can't have different handlers for GPU1 +and GPU2 from different vendors, and a third handler for mmapped regular +files. Threads cause additional pain with signal handling as well. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 19 lines · On Mon, 25 May 2020 17:09:55 +0200 Daniel Vetter <[email protected]> wrote:
Quoted 18 lines · Pekka Paalanen
Quoted 15 lines · On Mon, 25 May 2020 16:28:04 +0200 Daniel Vetter <[email protected]> wrote:
Quoted 14 lines · Andrey Grodzovsky
Quoted 12 lines · Daniel Vetter
Quoted 11 lines · Pekka Paalanen
Quoted 8 lines · On Tue, 19 May 2020 10:37:12 -0400 Andrey Grodzovsky…
Thanks for the summary, does put things in order and makes it easier to comprehend all the TODOs, some questions bellow
Quoted 4 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
...
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 8 lines
So currently
drm_ioctlwill just check fordrm_dev_is_unpluggedand return-ENODEVat the very beginning of the function (https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_ioctl.c#L825)
meaning that currently we blanket failure any IOCTL in case the device is
unplagued (ignoring the race condition if the device unplugged after the
check). So do we want to remove this check and put it only for render ioctls
(which are those ? e.g. for amdgpu there isAMDGPU_CSioctl) but not for
mode setting/dma_buf_import/dma_buf_export ioctls ? What about other types
of ioctls which are non of the listed above ?Hm right, and this goes back all the way to first usb udl support:
commit
2c07a21d6fb0be47fda696a618b726ea258ed1ddAuthor: Dave Airlie <[email protected]> Date: Mon Feb 20 14:18:07 2012 +0000drm: add core support for unplugging a device (v2)
So I guess we need to change the uapi docs to explain that. Not sure everyone can cope, but I guess they better do. Since the use-case back then was just udl, so simple legacy kms only, the damage was probably rather limited. I'm not sure we can get away with that now, where kms code has spread to funny places likey vulkan winsys code.
Or maybe we want a file priv flag you can set along the lines of "give me less shitty hotunplug semantics for ioctls". Or maybe we can just change the semantics, not crashing&burning shouldn't cause a regression :-)
For everything else (mmap, dma-buf fd,
sync_filefd, syncobj fd) I think the discussion is still more or less accurate.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Pekka, any thoughts?
Hi,
is ENODEV unique to this particular failure?
Not really sure, we'd need to audit all of drm ...
$ git ngrep ENODEV -- drivers/gpu/drm | wc -l 762
Yeah, grep is not enough.
Quoted 31 lines
Quoted 14 lines
Returning errors instead of faking success was my first idea, but you already convinced me that faking is at least as good if not better. :-)
So as long as the error code returned is unique to hot-unplug or other "oops, the device is gone" conditions, I think I'm fine. Weston does not handle ENODEV any way, it never did, and it certainly cannot be called a kernel regression.
As a Weston developer, I don't mind adding checks for ENODEV. But if I don't have to, even better. Weston is going to need more code to handle DRM device unplug in any case.
Sorry, no preference from me. ;-)
Quoted 7 lines
I do agree that replacing ENODEV with fake success is hard to imagine regressing anything. It's something you can do in the kernel at any time easily, but going from fake success to error is going to be painful. Maybe don't change things until there is a good reason to?
We need a kernel that doesn't crash before we can properly test what would be best for userspace, fake or error.
One upshot of faking stuff and only bailing in low-level hw code is that it makes validating the races when you hotunplug easier - if we remove the early bail-out check even an ioctl later on will look like it raced with the hotunplug path in the kernel. So better assurance that things won't blow up badly.
Otoh the early bail out in the top-level ioctl code increases the odds that you'll survive even on a driver that's totally buggy.
Quoted 8 lines
So yeah I guess maybe we should just document that currently you get an
-ENODEVand maybe have the option mentioned that we might change this going forward. See alsohttps://dri.freedesktop.org/docs/drm/gpu/drm-uapi.html#recommended-ioctl-return-values
I think that needs an update, at least clarify that "not present" includes "no longer present".
So if dmabuf import related ioctl returns ENODEV, it means a Wayland compositor should not punish the client for giving a bad buffer.
If a compositor uses EGL to import a dmabuf, then the EGL implementation will be calling dmabuf import related ioctls, does it not? But I'm fairly sure that EGL has no way to signal this particular condition to the caller uniquely. That means that either a compositor accepts buffers it never should have, or that it punishes clients for the DRM device disappearing.
See the below quote for a reminder:
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 7 lines
Review
Quoted 4 lines
+- dmabuf which point to memory that has disappeared are rewritten to
(quoted diff) +3
-0+ point to harmless zero pages, similar to mmaps. Imports still succeed + both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing any dmabuf.Same as above, I would expect drmPrimeHandleToFD and drmPrimeFDToHandle to return
-EIOin case device is detachedI imagined a use case behind this: a Wayland compositor attempting to import a client's dmabuf. The GPU the client was using is hot-unplugged, meaning the dmabufs lose their storage. It is up to the client to handle its rendering GPU disappearance on its own, but also the compositor should not disconnect it because the GPU disappeared. It's not the client's fault the GPU disappeared.
In Wayland, failure to use a
wl_bufferis considered a protocol error, and protocol errors are always fatal: the client is disconnected. The rationale is that the compositor must always be able to present the client buffer somehow. If the compositor cannot, then the client did not obey the protocol.The fallback presentation path in a compositor is usually importing the dmabuf to EGL, to be sampled from OpenGL. Normally the protocol guarantees that this works, so any failure to do so is a protocol violation. But if the GPU used by the client suddenly disappears and the imports start to fail, that is interpreted as a protocol violation unless the compositor can see why the import failed. Since the import is done via EGL, getting the right error code plumbed through from libdrm functions to the EGL caller would be a hassle. I don't see any error code in
EGL_EXT_image_dma_buf_importreserved for "the dmabuf storage was hot-unplugged", and I doubt there is anything exclusively for only that in the EGL base spec either.The cost of lying that the import worked is that the compositor will paint black or transparent where the window was supposed to be. It's a graphical glitch that is contrary to the Wayland design principles, but in this case a glitch is unavoidable: even if the compositor knew this buffer is now bad, what would it paint instead? It has nothing else to paint from. I'm assuming the compositor is using a different GPU than what disappeared.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Ideally, the client will eventually react to losing the GPU and either crash, quit, or switch its rendering to something that works which simply gives the compositor a new, working buffer without losing any window state in the process. If we risk the compositor disconnecting the client, then the client might not recover even if it wanted to.
This goes for both ways: importing a good dmabuf to a bad device, and a bad dmabuf to a good device. In neither case there should be a risk of erroneously disconnecting the Wayland client.
Hmm. Maybe Wayland compositors should ignore all EGL import failures that happen after the wl_buffer has been created (which implies that the dmabuf has been validated to work initially). When import fails at a later time, the compositor should just paint some error pattern instead of the window. That would let the kernel keep on returning errors.
Yeah, ok. I'll keep the ENODEV there in my next version. Let's see how that looks then.
Thanks,
pq
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 103 lines · Pekka Paalanen
Quoted 19 lines · On Mon, 25 May 2020 17:09:55 +0200 Daniel Vetter <[email protected]> wrote:
Quoted 18 lines · Pekka Paalanen
Quoted 15 lines · On Mon, 25 May 2020 16:28:04 +0200 Daniel Vetter <[email protected]> wrote:
Quoted 14 lines · Andrey Grodzovsky
Quoted 12 lines · Daniel Vetter
Quoted 11 lines · Pekka Paalanen
Quoted 8 lines · On Tue, 19 May 2020 10:37:12 -0400 Andrey Grodzovsky…
Thanks for the summary, does put things in order and makes it easier to comprehend all the TODOs, some questions bellow
Quoted 4 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
...
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 8 lines
So currently
drm_ioctlwill just check fordrm_dev_is_unpluggedand return-ENODEVat the very beginning of the function (https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/drm_ioctl.c#L825)
meaning that currently we blanket failure any IOCTL in case the device is
unplagued (ignoring the race condition if the device unplugged after the
check). So do we want to remove this check and put it only for render ioctls
(which are those ? e.g. for amdgpu there isAMDGPU_CSioctl) but not for
mode setting/dma_buf_import/dma_buf_export ioctls ? What about other types
of ioctls which are non of the listed above ?Hm right, and this goes back all the way to first usb udl support:
commit
2c07a21d6fb0be47fda696a618b726ea258ed1ddAuthor: Dave Airlie <[email protected]> Date: Mon Feb 20 14:18:07 2012 +0000drm: add core support for unplugging a device (v2)
So I guess we need to change the uapi docs to explain that. Not sure everyone can cope, but I guess they better do. Since the use-case back then was just udl, so simple legacy kms only, the damage was probably rather limited. I'm not sure we can get away with that now, where kms code has spread to funny places likey vulkan winsys code.
Or maybe we want a file priv flag you can set along the lines of "give me less shitty hotunplug semantics for ioctls". Or maybe we can just change the semantics, not crashing&burning shouldn't cause a regression :-)
For everything else (mmap, dma-buf fd,
sync_filefd, syncobj fd) I think the discussion is still more or less accurate.Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Pekka, any thoughts?
Hi,
is ENODEV unique to this particular failure?
Not really sure, we'd need to audit all of drm ...
$ git ngrep ENODEV -- drivers/gpu/drm | wc -l 762Yeah, grep is not enough.
Quoted 35 lines
Quoted 14 lines
Returning errors instead of faking success was my first idea, but you already convinced me that faking is at least as good if not better. :-)
So as long as the error code returned is unique to hot-unplug or other "oops, the device is gone" conditions, I think I'm fine. Weston does not handle ENODEV any way, it never did, and it certainly cannot be called a kernel regression.
As a Weston developer, I don't mind adding checks for ENODEV. But if I don't have to, even better. Weston is going to need more code to handle DRM device unplug in any case.
Sorry, no preference from me. ;-)
Quoted 7 lines
I do agree that replacing ENODEV with fake success is hard to imagine regressing anything. It's something you can do in the kernel at any time easily, but going from fake success to error is going to be painful. Maybe don't change things until there is a good reason to?
We need a kernel that doesn't crash before we can properly test what would be best for userspace, fake or error.
One upshot of faking stuff and only bailing in low-level hw code is that it makes validating the races when you hotunplug easier - if we remove the early bail-out check even an ioctl later on will look like it raced with the hotunplug path in the kernel. So better assurance that things won't blow up badly.
Otoh the early bail out in the top-level ioctl code increases the odds that you'll survive even on a driver that's totally buggy.
So yeah I guess maybe we should just document that currently you get an
-ENODEVand maybe have the option mentioned that we might change this going forward. See also
Quoted 7 lines
Quoted 4 lines
https://dri.freedesktop.org/docs/drm/gpu/drm-uapi.html#recommended-ioctl-return-values
I think that needs an update, at least clarify that "not present" includes "no longer present".
So if dmabuf import related ioctl returns ENODEV, it means a Wayland compositor should not punish the client for giving a bad buffer.
Atm this is only for the drm fd. I think right now dma-buf keep "working", where "working" = "probably results in some oopses". So maybe for dma-buf we need to put down the rule that they're not allowed to go boom, and then figure out how to do that, see below.
Quoted 66 lines
If a compositor uses EGL to import a dmabuf, then the EGL implementation will be calling dmabuf import related ioctls, does it not? But I'm fairly sure that EGL has no way to signal this particular condition to the caller uniquely. That means that either a compositor accepts buffers it never should have, or that it punishes clients for the DRM device disappearing.
See the below quote for a reminder:
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 7 lines
Review
Quoted 4 lines
+- dmabuf which point to memory that has disappeared are rewritten to
(quoted diff) +3
-0+ point to harmless zero pages, similar to mmaps. Imports still succeed + both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing any dmabuf.Same as above, I would expect drmPrimeHandleToFD and drmPrimeFDToHandle to return
-EIOin case device is detachedI imagined a use case behind this: a Wayland compositor attempting to import a client's dmabuf. The GPU the client was using is hot-unplugged, meaning the dmabufs lose their storage. It is up to the client to handle its rendering GPU disappearance on its own, but also the compositor should not disconnect it because the GPU disappeared. It's not the client's fault the GPU disappeared.
In Wayland, failure to use a
wl_bufferis considered a protocol error, and protocol errors are always fatal: the client is disconnected. The rationale is that the compositor must always be able to present the client buffer somehow. If the compositor cannot, then the client did not obey the protocol.The fallback presentation path in a compositor is usually importing the dmabuf to EGL, to be sampled from OpenGL. Normally the protocol guarantees that this works, so any failure to do so is a protocol violation. But if the GPU used by the client suddenly disappears and the imports start to fail, that is interpreted as a protocol violation unless the compositor can see why the import failed. Since the import is done via EGL, getting the right error code plumbed through from libdrm functions to the EGL caller would be a hassle. I don't see any error code in
EGL_EXT_image_dma_buf_importreserved for "the dmabuf storage was hot-unplugged", and I doubt there is anything exclusively for only that in the EGL base spec either.The cost of lying that the import worked is that the compositor will paint black or transparent where the window was supposed to be. It's a graphical glitch that is contrary to the Wayland design principles, but in this case a glitch is unavoidable: even if the compositor knew this buffer is now bad, what would it paint instead? It has nothing else to paint from. I'm assuming the compositor is using a different GPU than what disappeared.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Ideally, the client will eventually react to losing the GPU and either crash, quit, or switch its rendering to something that works which simply gives the compositor a new, working buffer without losing any window state in the process. If we risk the compositor disconnecting the client, then the client might not recover even if it wanted to.
This goes for both ways: importing a good dmabuf to a bad device, and a bad dmabuf to a good device. In neither case there should be a risk of erroneously disconnecting the Wayland client.
Hmm. Maybe Wayland compositors should ignore all EGL import failures that happen after the
wl_bufferhas been created (which implies that the dmabuf has been validated to work initially). When import fails at a later time, the compositor should just paint some error pattern instead of the window. That would let the kernel keep on returning errors.
Quoted 2 lines
Yeah, ok. I'll keep the ENODEV there in my next version. Let's see how that looks then.
tbh I have no idea what to do with dma-buf shared across drivers.
For dma-fence it's fairly simple: Force-complete them all, with an error code of ENODEV. But for dma-buf I have no idea. As long as the dma-buf sits in system memory it should keep working, plus/minus bugs in the exporter where it tries to look at device state that might no longer be there.
The real fun starts when the buffer is in vram, or when the mmap somehow goes through the device (but that's more a case for integrated gpu, and it's a bit hard to hotunplug those and consider that a real use-case). -Daniel
Quoted 4 lines
Thanks,
pq
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 1 line · On Thu, 28 May 2020 16:45:45 +0200 Daniel Vetter <[email protected]> wrote:
On Thu, May 28, 2020 at 03:27:57PM +0300, Pekka Paalanen wrote:
...
Quoted 18 lines
Quoted 9 lines
Hmm. Maybe Wayland compositors should ignore all EGL import failures that happen after the
wl_bufferhas been created (which implies that the dmabuf has been validated to work initially). When import fails at a later time, the compositor should just paint some error pattern instead of the window. That would let the kernel keep on returning errors.Yeah, ok. I'll keep the ENODEV there in my next version. Let's see how that looks then.
tbh I have no idea what to do with dma-buf shared across drivers.
For dma-fence it's fairly simple: Force-complete them all, with an error code of ENODEV. But for dma-buf I have no idea. As long as the dma-buf sits in system memory it should keep working, plus/minus bugs in the exporter where it tries to look at device state that might no longer be there.
Quoted 3 lines
The real fun starts when the buffer is in vram, or when the mmap somehow goes through the device (but that's more a case for integrated gpu, and it's a bit hard to hotunplug those and consider that a real use-case).
Is forced driver unbind not a real use-case, or would it not invalidate the memory references wrapped in a dmabuf?
In the proposal, I listed driver unbind as a use-case.
I don't mind if cross-driver shared dmabuf needed ten years of kernel internals development to reach a state where it won't explode anymore, but that's the goal I want to set today. I don't think we can tell userspace to never share dmabufs cross-device if the exporter device is hot-unpluggable, can we?
Thanks,
pq
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 76 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and from https://lists.freedesktop.org/archives/dri-devel/2020-May/265484.html .
Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>---
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 75 ++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)Documentation/gpu/drm-uapi.rst +41
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,79 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May 13). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some cases specific +ioctls returning EIO. + +This goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was + in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected. +
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 215
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. + +- If underlying memory disappears, the mmaps are replaced with harmless + zero pages where access does not raise SIGBUS. Reads return zeros, + writes are ignored.
Hm this is going to be expensive to implement, every write access would cause a minor fault to catch it and throw it away. That's not going to be fast is someone is caught trying to upload a giant texture at that moment :-/
Could we go with just "Reads and writes will still completely without errors, but have otherwise undefined behaviour". Then we could go with a "single shared page for every drmfd" (to avoid leaks), set up with rw permissions. That's dirt cheap, should be easy to implement and everything stays fast.
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 220
Quoted 5 lines
Documentation/gpu/drm-uapi.rst +5
-0+ +- dmabuf which point to memory that has disappeared are rewritten to + point to harmless zero pages, similar to mmaps. Imports still succeed + both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing any dmabuf.
Same here about mmaps.
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 225
Quoted 5 lines
Documentation/gpu/drm-uapi.rst +5
-0+ +- Render ioctls return EIO which is then handled in userspace drivers, + e.g. Mesa, to have the device disappearance handled in the way + specified for each API (OpenGL, GL ES: GL_KHR_robustness; + Vulkan: VK_ERROR_DEVICE_LOST; etc.)
I'd go further and spec that driver-specific render ioctl have driver specific behaviour, but overall must not result in crashes and failures expect as specified in the client apis you already list.
Some drivers might just go with an uevent and have no errors from ioctls, some might only want to have an error on a very specific ioctl, some maybe on all of them. Since no one except the hw specific userspace drivers will have to deal with that I think best to only specify the outcome we want.
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 232
Quoted 7 lines
Documentation/gpu/drm-uapi.rst +7
-0+ +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like Mesa produces. Signal +handlers are not composable, you can't have different handlers for GPU1 +and GPU2 from different vendors, and a third handler for mmapped regular +files. Threads cause additional pain with signal handling as well.
Hm maybe move that to the entire mmap discussion and why SIGBUS is a really bad idea?
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 238
Quoted 6 lines
Documentation/gpu/drm-uapi.rst +6
-0+ +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device.
Maybe we should add an explicit note that there's no guarantee about the new chardev minor this new device will get, it could both inherit the existing one (you can't open the old one anymore anyway) or get a new one?
Or does userspace want a guarantee, i.e. as long as there's still a handle open kernel guarantees to not recycle the chardev minor (which is what we currently do). In that case better to add that to your list of guarantees above.
I think overall a good start for documenting the uapi expectations of hotunplug, we'll definitely have to refine this as we go.
Cheers, Daniel
Quoted 7 lines
Documentation/gpu/drm-uapi.rst +1
-0+ .. _drm_driver_ioctl: IOCTL Support on Device NodesSignature · 1 line
2.20.1
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 8 lines · Daniel Vetter
Maybe we should add an explicit note that there's no guarantee about the new chardev minor this new device will get, it could both inherit the existing one (you can't open the old one anymore anyway) or get a new one?
Or does userspace want a guarantee, i.e. as long as there's still a handle open kernel guarantees to not recycle the chardev minor (which is what we currently do). In that case better to add that to your list of guarantees above.
The are race conditions to consider too, e.g.
- Compositor sends /dev/dri/card0 to a client
- card0 goes away
- Another device takes card0
- Client receives /dev/dri/card0 and then starts using it, but it's the wrong device
At first glance these seem like edge-cases that almost never happen. However I've seen these happen in practice with connectors, especially with docks.
One solution would be to number minor numbers like PIDs: don't recycle card0 before we've reached the upper minor number limit.
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 12 lines · Simon Ser
Quoted 8 lines · Daniel Vetter
Maybe we should add an explicit note that there's no guarantee about the new chardev minor this new device will get, it could both inherit the existing one (you can't open the old one anymore anyway) or get a new one?
Or does userspace want a guarantee, i.e. as long as there's still a handle open kernel guarantees to not recycle the chardev minor (which is what we currently do). In that case better to add that to your list of guarantees above.
Quoted 7 lines
The are race conditions to consider too, e.g.
- Compositor sends
/dev/dri/card0to a client
- card0 goes away
- Another device takes card0
- Client receives/dev/dri/card0and then starts using it, but it's the wrong device
Oh reminds me, what should we do about open() - that one will fail, the chardev is going away after all, not failing kinda doesn't make sense. And more tricky, about creating new leases?
I think reasonable semantics here would be that both of these "create a new open drm fd" operations can fail as soon as the device is unplugged. Userspace needs to be able to deal with that. -Daniel
Quoted 7 lines
At first glance these seem like edge-cases that almost never happen. However I've seen these happen in practice with connectors, especially with docks.
One solution would be to number minor numbers like PIDs: don't recycle card0 before we've reached the upper minor number limit.
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 21 lines · On Wed, 20 May 2020 16:19:00 +0200 Daniel Vetter <[email protected]> wrote:
Quoted 12 lines · Simon Ser
Quoted 8 lines · Daniel Vetter
Maybe we should add an explicit note that there's no guarantee about the new chardev minor this new device will get, it could both inherit the existing one (you can't open the old one anymore anyway) or get a new one?
Or does userspace want a guarantee, i.e. as long as there's still a handle open kernel guarantees to not recycle the chardev minor (which is what we currently do). In that case better to add that to your list of guarantees above.
Quoted 7 lines
The are race conditions to consider too, e.g.
- Compositor sends
/dev/dri/card0to a client
- card0 goes away
- Another device takes card0
- Client receives/dev/dri/card0and then starts using it, but it's the wrong device
Quoted 7 lines
Oh reminds me, what should we do about
open()- that one will fail, the chardev is going away after all, not failing kinda doesn't make sense. And more tricky, about creating new leases?I think reasonable semantics here would be that both of these "create a new open drm fd" operations can fail as soon as the device is unplugged. Userspace needs to be able to deal with that.
Hi,
yeah, we can make mmap read/write end result undefined, recycle char minors like pids, and let new open()s and new leases fail. Pretty much everything Daniel and Simon said make sense to me.
I'll spin a v2, but maybe next week.
What about the drm_ioctl() issue Andrey pointed out?
Thanks,
pq
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 34 lines · Pekka Paalanen
Quoted 21 lines · On Wed, 20 May 2020 16:19:00 +0200 Daniel Vetter <[email protected]> wrote:
Quoted 12 lines · Simon Ser
Quoted 8 lines · Daniel Vetter
Maybe we should add an explicit note that there's no guarantee about the new chardev minor this new device will get, it could both inherit the existing one (you can't open the old one anymore anyway) or get a new one?
Or does userspace want a guarantee, i.e. as long as there's still a handle open kernel guarantees to not recycle the chardev minor (which is what we currently do). In that case better to add that to your list of guarantees above.
Quoted 7 lines
The are race conditions to consider too, e.g.
- Compositor sends
/dev/dri/card0to a client
- card0 goes away
- Another device takes card0
- Client receives/dev/dri/card0and then starts using it, but it's the wrong deviceQuoted 7 lines
Oh reminds me, what should we do about
open()- that one will fail, the chardev is going away after all, not failing kinda doesn't make sense. And more tricky, about creating new leases?I think reasonable semantics here would be that both of these "create a new open drm fd" operations can fail as soon as the device is unplugged. Userspace needs to be able to deal with that.
Hi,
Quoted 7 lines
yeah, we can make mmap read/write end result undefined, recycle char minors like pids, and let new open()s and new leases fail. Pretty much everything Daniel and Simon said make sense to me.
I'll spin a v2, but maybe next week.
What about the
drm_ioctl()issue Andrey pointed out?
Dropped some thoughts there, tbh dunno, need some more discussions? -Daniel
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
PATCH v2 1/1 drm/doc: device hot-unplug for userspace
In reply to Pekka PaalanenFrom: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and from https://lists.freedesktop.org/archives/dri-devel/2020-May/265484.html .
---
v2:
- mmap reads/writes undefined (danvet)
- make render ioctl behaviour driver-specific (danvet)
- restructure the mmap paragraphs (danvet)
- chardev minor notes (Simon)
- open behaviour (danvet)
- DRM leasing behaviour (danvet)
- added links
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 102 +++++++++++++++++++++++++++++++++ 1 file changed, 102 insertions(+)
Documentation/gpu/drm-uapi.rst +102 -0
@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================
@@ -162,6 +164,106 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some cases +driver-specific ioctls returning EIO. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + +Similar to PIDs, chardev minor numbers are not recycled immediately. A +new DRM device always picks the next free minor number compared to the +previous one allocated, and wraps around when minor numbers are +exhausted. + +Requirements for UAPI +--------------------- + +The goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was + in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. + +- dmabuf which point to memory that has disappeared will continue to + be successfully imported if it would have succeeded before the + disappearance. + +- Attempting to import a dmabuf to a disappeared device will succeed if + it would have succeeded without the disappearance. + +- Some userspace APIs already define what should happen when the device + disappears (OpenGL, GL ES: `GL_KHR_robustness`_; `Vulkan`_: + VK_ERROR_DEVICE_LOST; etc.). DRM drivers are free to implement this + behaviour the way they see best, e.g. returning failures in + driver-specific ioctls and handling those in userspace drivers, or + rely on uevents, and so on. + +- open() on a device node whose underlying device has disappeared will + fail. + +- Attempting to create a DRM lease on a disappeared DRM device will + fail. Existing DRM leases remain. + +.. _GL_KHR_robustness: https://www.khronos.org/registry/OpenGL/extensions/KHR/KHR_robustness.txt +.. _Vulkan: https://www.khronos.org/vulkan/ + +Requirements for memory maps +---------------------------- + +Memory maps have further requirements. If the underlying memory +disappears, the mmap is modified such that reads and writes will still +complete successfully but the result is undefined. This applies to both +userspace mmap()'d memory and memory pointed to by dmabuf which might be +mapped to other devices. + +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like those that Mesa produces. +Signal handlers are not composable, you can't have different handlers +for GPU1 and GPU2 from different vendors, and a third handler for +mmapped regular files. Threads cause additional pain with signal +handling as well. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes
Signature · 6 lines
2.20.1 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 54 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cc9676f35bbdf4d5a052808d800a9b517%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637260076178891269&sdata=tbOTr7TfESohEgWspomM1sbMq4U4n7bOvdS6JlYifmM%3D&reserved=0 .Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>Cc Simon Ser <[email protected]>---
v2:
- mmap reads/writes undefined (danvet)
- make render ioctl behaviour driver-specific (danvet)
- restructure the mmap paragraphs (danvet)
- chardev minor notes (Simon)
- open behaviour (danvet)
- DRM leasing behaviour (danvet)
- added linksDisclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 102 +++++++++++++++++++++++++++++++++ 1 file changed, 102 insertions(+)Documentation/gpu/drm-uapi.rst +9
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,106 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May). +
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 183
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries.
So to support all the requirements in this document only kernel changes should be enough and no changes are required from user mode part of the stack ?
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is
Quoted 9 lines
Documentation/gpu/drm-uapi.rst +9
-0+ +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent
Is this uevent already implemented ? Can you point me to the code ?
Quoted 24 lines
or in some cases
Documentation/gpu/drm-uapi.rst +23
-0+driver-specific ioctls returning EIO. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + +Similar to PIDs, chardev minor numbers are not recycled immediately. A +new DRM device always picks the next free minor number compared to the +previous one allocated, and wraps around when minor numbers are +exhausted. + +Requirements for UAPI +--------------------- + +The goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was + in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fences
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 225
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+ force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting.
The 4 points above refer to mode setting/display attached card and are irrelevant for secondary GPU (e.g. DRI-PRIME scenario) or no display system in general. Maybe we can somehow highlight this in the document and I on the implementing side can then decide as a first step to concentrate on implementing the non display case as a first step or the only step. In general and correct me if I am wrong, render only GPUs (or compute only) are the majority of cases where you would want to be able to detach/attach GPU on the fly (e.g attach stronger secondary graphic card to a laptop to get high performance in a game or add/remove a GPU to/from a compute cluster)
Andrey
Quoted 34 lines
Documentation/gpu/drm-uapi.rst +34
-0+ +- dmabuf which point to memory that has disappeared will continue to + be successfully imported if it would have succeeded before the + disappearance. + +- Attempting to import a dmabuf to a disappeared device will succeed if + it would have succeeded without the disappearance. + +- Some userspace APIs already define what should happen when the device + disappears (OpenGL, GL ES: `GL_KHR_robustness`_; `Vulkan`_: + VK_ERROR_DEVICE_LOST; etc.). DRM drivers are free to implement this + behaviour the way they see best, e.g. returning failures in + driver-specific ioctls and handling those in userspace drivers, or + rely on uevents, and so on. + +- open() on a device node whose underlying device has disappeared will + fail. + +- Attempting to create a DRM lease on a disappeared DRM device will + fail. Existing DRM leases remain. + +.. _GL_KHR_robustness: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.khronos.org%2Fregistry%2FOpenGL%2Fextensions%2FKHR%2FKHR_robustness.txt&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cc9676f35bbdf4d5a052808d800a9b517%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637260076178891269&sdata=m%2FneRusoe6qGVU8Edk%2FncaD7eSJZXtPnA1IqLr7k%2Bos%3D&reserved=0 +.. _Vulkan: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.khronos.org%2Fvulkan%2F&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cc9676f35bbdf4d5a052808d800a9b517%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637260076178901265&sdata=WsfLduUBzRKlybOJb5PQViBWYu5DgleEeycmf76l3UU%3D&reserved=0 + +Requirements for memory maps +---------------------------- + +Memory maps have further requirements. If the underlying memory +disappears, the mmap is modified such that reads and writes will still +complete successfully but the result is undefined. This applies to both +userspace mmap()'d memory and memory pointed to by dmabuf which might be +mapped to other devices. + +Raising SIGBUS is not an option, because userspace cannot realistically
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 269
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +7
-0+handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like those that Mesa produces. +Signal handlers are not composable, you can't have different handlers +for GPU1 and GPU2 from different vendors, and a third handler for +mmapped regular files. Threads cause additional pain with signal +handling as well. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 121 lines · Andrey Grodzovsky
Quoted 53 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cc9676f35bbdf4d5a052808d800a9b517%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637260076178891269&sdata=tbOTr7TfESohEgWspomM1sbMq4U4n7bOvdS6JlYifmM%3D&reserved=0 .Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>Cc Simon Ser <[email protected]>---
v2:
- mmap reads/writes undefined (danvet)
- make render ioctl behaviour driver-specific (danvet)
- restructure the mmap paragraphs (danvet)
- chardev minor notes (Simon)
- open behaviour (danvet)
- DRM leasing behaviour (danvet)
- added linksDisclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 102 +++++++++++++++++++++++++++++++++ 1 file changed, 102 insertions(+)Documentation/gpu/drm-uapi.rst +9
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,106 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May). +Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 182
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries.So to support all the requirements in this document only kernel changes should be enough and no changes are required from user mode part of the stack ?
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is
Quoted 9 lines
Documentation/gpu/drm-uapi.rst +9
-0+ +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed ueventIs this uevent already implemented ? Can you point me to the code ?
Quoted 24 lines
or in some cases
Documentation/gpu/drm-uapi.rst +23
-0+driver-specific ioctls returning EIO. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + +Similar to PIDs, chardev minor numbers are not recycled immediately. A +new DRM device always picks the next free minor number compared to the +previous one allocated, and wraps around when minor numbers are +exhausted. + +Requirements for UAPI +--------------------- + +The goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was + in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fencesDocumentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 225
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+ force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting.The 4 points above refer to mode setting/display attached card and are
Quoted 8 lines
irrelevant for secondary GPU (e.g. DRI-PRIME scenario) or no display system in general. Maybe we can somehow highlight this in the document and I on the implementing side can then decide as a first step to concentrate on implementing the non display case as a first step or the only step. In general and correct me if I am wrong, render only GPUs (or compute only) are the majority of cases where you would want to be able to detach/attach GPU on the fly (e.g attach stronger secondary graphic card to a laptop to get high performance in a game or add/remove a GPU to/from a compute cluster)
Yeah maybe splitting this up into kms section, and rendering/cross driver section (the dma-buf/fence stuff is relevant for both display and rendering) would make some sense.
-Daniel
Quoted 47 lines
Andrey
Quoted 43 lines
Documentation/gpu/drm-uapi.rst +41
-0+ +- dmabuf which point to memory that has disappeared will continue to + be successfully imported if it would have succeeded before the + disappearance. + +- Attempting to import a dmabuf to a disappeared device will succeed if + it would have succeeded without the disappearance. + +- Some userspace APIs already define what should happen when the device + disappears (OpenGL, GL ES: `GL_KHR_robustness`_; `Vulkan`_: + VK_ERROR_DEVICE_LOST; etc.). DRM drivers are free to implement this + behaviour the way they see best, e.g. returning failures in + driver-specific ioctls and handling those in userspace drivers, or + rely on uevents, and so on. + +- open() on a device node whose underlying device has disappeared will + fail. + +- Attempting to create a DRM lease on a disappeared DRM device will + fail. Existing DRM leases remain. + +.. _GL_KHR_robustness: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.khronos.org%2Fregistry%2FOpenGL%2Fextensions%2FKHR%2FKHR_robustness.txt&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cc9676f35bbdf4d5a052808d800a9b517%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637260076178891269&sdata=m%2FneRusoe6qGVU8Edk%2FncaD7eSJZXtPnA1IqLr7k%2Bos%3D&reserved=0 +.. _Vulkan: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.khronos.org%2Fvulkan%2F&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cc9676f35bbdf4d5a052808d800a9b517%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637260076178901265&sdata=WsfLduUBzRKlybOJb5PQViBWYu5DgleEeycmf76l3UU%3D&reserved=0 + +Requirements for memory maps +---------------------------- + +Memory maps have further requirements. If the underlying memory +disappears, the mmap is modified such that reads and writes will still +complete successfully but the result is undefined. This applies to both +userspace mmap()'d memory and memory pointed to by dmabuf which might be +mapped to other devices. + +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like those that Mesa produces. +Signal handlers are not composable, you can't have different handlers +for GPU1 and GPU2 from different vendors, and a third handler for +mmapped regular files. Threads cause additional pain with signal +handling as well. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 131 lines · On Mon, 25 May 2020 16:30:17 +0200 Daniel Vetter <[email protected]> wrote:
Quoted 121 lines · Andrey Grodzovsky
Quoted 53 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cc9676f35bbdf4d5a052808d800a9b517%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637260076178891269&sdata=tbOTr7TfESohEgWspomM1sbMq4U4n7bOvdS6JlYifmM%3D&reserved=0 .Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>Cc Simon Ser <[email protected]>---
v2:
- mmap reads/writes undefined (danvet)
- make render ioctl behaviour driver-specific (danvet)
- restructure the mmap paragraphs (danvet)
- chardev minor notes (Simon)
- open behaviour (danvet)
- DRM leasing behaviour (danvet)
- added linksDisclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 102 +++++++++++++++++++++++++++++++++ 1 file changed, 102 insertions(+)Documentation/gpu/drm-uapi.rst +9
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,106 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May). +Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 182
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries.So to support all the requirements in this document only kernel changes should be enough and no changes are required from user mode part of the stack ?
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is
Quoted 9 lines
Documentation/gpu/drm-uapi.rst +9
-0+ +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed ueventIs this uevent already implemented ? Can you point me to the code ?
Quoted 24 lines
or in some cases
Documentation/gpu/drm-uapi.rst +23
-0+driver-specific ioctls returning EIO. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + +Similar to PIDs, chardev minor numbers are not recycled immediately. A +new DRM device always picks the next free minor number compared to the +previous one allocated, and wraps around when minor numbers are +exhausted. + +Requirements for UAPI +--------------------- + +The goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was + in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fencesDocumentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 225
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+ force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting.The 4 points above refer to mode setting/display attached card and are
Quoted 8 lines
irrelevant for secondary GPU (e.g. DRI-PRIME scenario) or no display system in general. Maybe we can somehow highlight this in the document and I on the implementing side can then decide as a first step to concentrate on implementing the non display case as a first step or the only step. In general and correct me if I am wrong, render only GPUs (or compute only) are the majority of cases where you would want to be able to detach/attach GPU on the fly (e.g attach stronger secondary graphic card to a laptop to get high performance in a game or add/remove a GPU to/from a compute cluster)
Quoted 3 lines
Yeah maybe splitting this up into kms section, and rendering/cross driver section (the dma-buf/fence stuff is relevant for both display and rendering) would make some sense.
Is that really something that needs spelling out?
Hmm. I guess the unwritten assumption on every "fake success" is the condition that it would have succeeded if the device was not unplugged.
Is the problem here that one might read this as needing to fake success for things that could never have worked at all? Like KMS on render-only device.
The dmabuf items below have the wording.
I think splitting stuff into KMS stuff, render stuff, KMS-and-render stuff, cross-device stuff, and mmaps gets a bit far. Or do you expect a lot more text in here? Maybe expanding each bullet point to a paragraph?
Thanks,
pq
Quoted 41 lines
-Daniel
Quoted 39 lines
Andrey
Quoted 35 lines
Documentation/gpu/drm-uapi.rst +35
-0+ +- dmabuf which point to memory that has disappeared will continue to + be successfully imported if it would have succeeded before the + disappearance. + +- Attempting to import a dmabuf to a disappeared device will succeed if + it would have succeeded without the disappearance. + +- Some userspace APIs already define what should happen when the device + disappears (OpenGL, GL ES: `GL_KHR_robustness`_; `Vulkan`_: + VK_ERROR_DEVICE_LOST; etc.). DRM drivers are free to implement this + behaviour the way they see best, e.g. returning failures in + driver-specific ioctls and handling those in userspace drivers, or + rely on uevents, and so on. + +- open() on a device node whose underlying device has disappeared will + fail. + +- Attempting to create a DRM lease on a disappeared DRM device will + fail. Existing DRM leases remain. + +.. _GL_KHR_robustness: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.khronos.org%2Fregistry%2FOpenGL%2Fextensions%2FKHR%2FKHR_robustness.txt&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cc9676f35bbdf4d5a052808d800a9b517%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637260076178891269&sdata=m%2FneRusoe6qGVU8Edk%2FncaD7eSJZXtPnA1IqLr7k%2Bos%3D&reserved=0 +.. _Vulkan: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.khronos.org%2Fvulkan%2F&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cc9676f35bbdf4d5a052808d800a9b517%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637260076178901265&sdata=WsfLduUBzRKlybOJb5PQViBWYu5DgleEeycmf76l3UU%3D&reserved=0 + +Requirements for memory maps +---------------------------- + +Memory maps have further requirements. If the underlying memory +disappears, the mmap is modified such that reads and writes will still +complete successfully but the result is undefined. This applies to both +userspace mmap()'d memory and memory pointed to by dmabuf which might be +mapped to other devices. + +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely
Quoted 9 lines
Quoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 269
Quoted 8 lines
Documentation/gpu/drm-uapi.rst +6
-0+difficult to use correctly from libraries like those that Mesa produces. +Signal handlers are not composable, you can't have different handlers +for GPU1 and GPU2 from different vendors, and a third handler for +mmapped regular files. Threads cause additional pain with signal +handling as well. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 62 lines · On Mon, 25 May 2020 09:51:30 -0400 Andrey Grodzovsky…
Quoted 60 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cc9676f35bbdf4d5a052808d800a9b517%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637260076178891269&sdata=tbOTr7TfESohEgWspomM1sbMq4U4n7bOvdS6JlYifmM%3D&reserved=0 .Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>Cc Simon Ser <[email protected]>---
v2:
- mmap reads/writes undefined (danvet)
- make render ioctl behaviour driver-specific (danvet)
- restructure the mmap paragraphs (danvet)
- chardev minor notes (Simon)
- open behaviour (danvet)
- DRM leasing behaviour (danvet)
- added linksDisclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 102 +++++++++++++++++++++++++++++++++ 1 file changed, 102 insertions(+)Documentation/gpu/drm-uapi.rst +15
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,106 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants
Quoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 183
Quoted 4 lines
Documentation/gpu/drm-uapi.rst +4
-0+to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries.So to support all the requirements in this document only kernel changes should be enough and no changes are required from user mode part of the stack ?
Hi,
my intention is that this document describes what the kernel delivers, or should deliver, to allow userspace to cope with hot-unplug if userspace wishes to do so. "Userspace" here includes userspace part of GPU drivers.
Userspace has a lot to develop to actually recover instead of just sit in the dark after the device disappears. Handling the uevent for DRM device removal or errors from GL/Vulkan is just the beginning of it. I would assume that userspace drivers have things to implement as well, before GL or Vulkan apps can recover instead of get stuck or crash.
Unplugging "secondary" DRM devices (mostly used for KMS to have more monitors lit) should be relatively easy to implement in display servers. Unplugging the GPU that a display server is using for rendering is going to be really difficult and will need client (application toolkit) support the very least, and perhaps even new window system protocol.
I imagine this will be incremental development: first the kernel stops crashing. Then display servers stop crashing. At some point userspace GPU drivers stop crashing. Then display servers learn to recover instead of sit in the dark, but disconnect most of their clients. Then maybe with the help of window system protocol additions, some major toolkits learn to not get killed. And so on.
Once all that works, the follow-up step could be some protocol to switch applications from one GPU to another in flight. That's off-topic here, but being able to handle GPU unplug is half of the switch.
Quoted 4 lines
Quoted 4 lines
Documentation/gpu/drm-uapi.rst +4
-0+ +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device.
Quoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is
Quoted 5 lines
Documentation/gpu/drm-uapi.rst +5
-0+ +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed ueventIs this uevent already implemented ? Can you point me to the code ?
I can't point to any kernel code, I'm just not familiar with it. But it's the same uevent all Linux devices emit. You unplug a USB mouse, this is the event that gets sent.
You can emulate it with 'udevadm trigger -c remove' IIRC, and it is the "remove" event you can match in udev rules.
KMS hotplug event is also a uevent, but I think it is "change" rather than "remove". Otherwise the same mechanism. Display servers already watch for uevents to learn about monitor hotplug, and some watch for DRM device added events too. But I don't think any really watch for DRM device removed events, because usually everything explodes first. I don't know, maybe X.org handles UDL unplugs?
Quoted 38 lines
Quoted 24 lines
or in some cases
Documentation/gpu/drm-uapi.rst +23
-0+driver-specific ioctls returning EIO. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + +Similar to PIDs, chardev minor numbers are not recycled immediately. A +new DRM device always picks the next free minor number compared to the +previous one allocated, and wraps around when minor numbers are +exhausted. + +Requirements for UAPI +--------------------- + +The goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was + in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fencesDocumentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 225
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+ force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting.The 4 points above refer to mode setting/display attached card and are irrelevant for secondary GPU (e.g. DRI-PRIME scenario) or no display
Quoted 8 lines
system in general. Maybe we can somehow highlight this in the document and I on the implementing side can then decide as a first step to concentrate on implementing the non display case as a first step or the only step. In general and correct me if I am wrong, render only GPUs (or compute only) are the majority of cases where you would want to be able to detach/attach GPU on the fly (e.g attach stronger secondary graphic card to a laptop to get high performance in a game or add/remove a GPU to/from a compute cluster)
I do think KMS-only (not rendering) devices are a major use case for
hot-unplug: docks, USB-display-adapters etc. I wrote this patch on behalf of DisplayLink after all.
Render-only GPUs are another important use case like you describe. And a dock might perhaps have both: a powerful GPU and a big screen connected.
Personally, I have no expectations other than a hope that some day at least the drivers that support hot-unpluggable hardware would implement all of this. :-)
I would assume it's fine to work piece by piece towards the goal on your own pace. This patch here is just for setting up the goal without a deadline. I'm no DRM maintainer or even a DRM developer.
Thanks,
pq
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 76 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Ce8e13dc4c85648e5fcd408d7fbdc5f2b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637254796242596783&sdata=bA%2FAy3VGvzNqmV1kGigNROSZQEws2E98JibDxvEICNs%3D&reserved=0 .Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>---
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 75 ++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)Documentation/gpu/drm-uapi.rst +41
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,79 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May 13). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some cases specific +ioctls returning EIO. + +This goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was + in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected. +
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 215
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. + +- If underlying memory disappears, the mmaps are replaced with harmless + zero pages where access does not raise SIGBUS. Reads return zeros, + writes are ignored.
Regarding this paragraph - what about exiting mappings ? In the first patchset we would actively invalidate all the existing CPU mappings to device memory and i think we still should do it otherwise we will see random crashes in applications as was before. I guess it's because TLBs and page tables are not updated to reflect the fact the device is gone.
Andrey
Quoted 17 lines
Documentation/gpu/drm-uapi.rst +17
-0+ +- dmabuf which point to memory that has disappeared are rewritten to + point to harmless zero pages, similar to mmaps. Imports still succeed + both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing any dmabuf. + +- Render ioctls return EIO which is then handled in userspace drivers, + e.g. Mesa, to have the device disappearance handled in the way + specified for each API (OpenGL, GL ES: GL_KHR_robustness; + Vulkan: VK_ERROR_DEVICE_LOST; etc.) + +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like Mesa produces. Signal +handlers are not composable, you can't have different handlers for GPU1 +and GPU2 from different vendors, and a third handler for mmapped regular +files. Threads cause additional pain with signal handling as well.
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 242
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +7
-0+ +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 86 lines · On Tue, 26 May 2020 10:30:20 -0400 Andrey Grodzovsky…
Quoted 85 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Ce8e13dc4c85648e5fcd408d7fbdc5f2b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637254796242596783&sdata=bA%2FAy3VGvzNqmV1kGigNROSZQEws2E98JibDxvEICNs%3D&reserved=0 .Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>---
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 75 ++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)Documentation/gpu/drm-uapi.rst +50
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,79 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May 13). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some cases specific +ioctls returning EIO. + +This goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was + in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. + +- If underlying memory disappears, the mmaps are replaced with harmless + zero pages where access does not raise SIGBUS. Reads return zeros,
Quoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 215
Quoted 1 line
Documentation/gpu/drm-uapi.rst +1
-0+ writes are ignored.Regarding this paragraph - what about exiting mappings ? In the first patchset we would actively invalidate all the existing CPU mappings to device memory and i think we still should do it otherwise we will see random crashes in applications as was before. I guess it's because TLBs and page tables are not updated to reflect the fact the device is gone.
Hi,
I was talking about existing mappings. What I forgot to specify is how new mmap() calls after the device disappearance should work (the end result should be the same still, not failure).
I'll clarify this in the next revision.
Thanks,
pq
Quoted 26 lines
Quoted 26 lines
Documentation/gpu/drm-uapi.rst +24
-0+ +- dmabuf which point to memory that has disappeared are rewritten to + point to harmless zero pages, similar to mmaps. Imports still succeed + both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing any dmabuf. + +- Render ioctls return EIO which is then handled in userspace drivers, + e.g. Mesa, to have the device disappearance handled in the way + specified for each API (OpenGL, GL ES: GL_KHR_robustness; + Vulkan: VK_ERROR_DEVICE_LOST; etc.) + +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like Mesa produces. Signal +handlers are not composable, you can't have different handlers for GPU1 +and GPU2 from different vendors, and a third handler for mmapped regular +files. Threads cause additional pain with signal handling as well. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + .. _drm_driver_ioctl:
Quoted 1 line
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 242
Quoted 1 line
Documentation/gpu/drm-uapi.rst +0
-0IOCTL Support on Device Nodes
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 102 lines · Pekka Paalanen
Quoted 85 lines · On Tue, 26 May 2020 10:30:20 -0400 Andrey Grodzovsky…
Quoted 84 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Ce8e13dc4c85648e5fcd408d7fbdc5f2b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637254796242596783&sdata=bA%2FAy3VGvzNqmV1kGigNROSZQEws2E98JibDxvEICNs%3D&reserved=0 .Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>---
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 75 ++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)Documentation/gpu/drm-uapi.rst +49
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,79 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May 13). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some cases specific +ioctls returning EIO. + +This goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was + in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. + +- If underlying memory disappears, the mmaps are replaced with harmlessQuoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 215
Quoted 2 lines
Documentation/gpu/drm-uapi.rst +2
-0+ zero pages where access does not raise SIGBUS. Reads return zeros, + writes are ignored.Regarding this paragraph - what about exiting mappings ? In the first patchset we would actively invalidate all the existing CPU mappings to device memory and i think we still should do it otherwise we will see random crashes in applications as was before. I guess it's because TLBs and page tables are not updated to reflect the fact the device is gone.
Hi,
I was talking about existing mappings. What I forgot to specify is how new
mmap()calls after the device disappearance should work (the end result should be the same still, not failure).
Quoted 5 lines
I'll clarify this in the next revision.
Thanks,
pq
I see, that ok.
Next related question is more for Daniel/Christian - about the implementation of this paragraph, I was thinking about something like checking for device disconnect in ttm_bo_vm_fault_reserved and if so remap the entire VA range for the VMA where the fault address belongs to the global zero page (i.e. (remap_pfn_range(vma, vma->vm_start, page_to_pfn(ZERO_PAGE(vma->vm_start), vma->vm_end - vma->vm_start, vma->vm_page_prot)). Question is, when the doc says 'writes are ignored' does it mean i should use copy on write for the vma->vm_page_prot and if so how i actually do it as i was not able to find what flags to set into vm_page_prot to force copy on write behavior.
Andrey
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Documentation/gpu/drm-uapi.rst +24
-0+ +- dmabuf which point to memory that has disappeared are rewritten to + point to harmless zero pages, similar to mmaps. Imports still succeed + both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing any dmabuf. + +- Render ioctls return EIO which is then handled in userspace drivers, + e.g. Mesa, to have the device disappearance handled in the way + specified for each API (OpenGL, GL ES: GL_KHR_robustness; + Vulkan: VK_ERROR_DEVICE_LOST; etc.) + +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like Mesa produces. Signal +handlers are not composable, you can't have different handlers for GPU1 +and GPU2 from different vendors, and a third handler for mmapped regular +files. Threads cause additional pain with signal handling as well. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + .. _drm_driver_ioctl:
Quoted 1 line
Quoted 1 line
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 242
Quoted 1 line
Documentation/gpu/drm-uapi.rst +0
-0IOCTL Support on Device Nodes
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 114 lines · Andrey Grodzovsky
Quoted 102 lines · Pekka Paalanen
Quoted 85 lines · On Tue, 26 May 2020 10:30:20 -0400 Andrey Grodzovsky…
Quoted 84 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Ce8e13dc4c85648e5fcd408d7fbdc5f2b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637254796242596783&sdata=bA%2FAy3VGvzNqmV1kGigNROSZQEws2E98JibDxvEICNs%3D&reserved=0 .Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>---
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 75 ++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)Documentation/gpu/drm-uapi.rst +49
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,79 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May 13). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some cases specific +ioctls returning EIO. + +This goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was + in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. + +- If underlying memory disappears, the mmaps are replaced with harmlessQuoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 215
Quoted 2 lines
Documentation/gpu/drm-uapi.rst +2
-0+ zero pages where access does not raise SIGBUS. Reads return zeros, + writes are ignored.Regarding this paragraph - what about exiting mappings ? In the first patchset we would actively invalidate all the existing CPU mappings to device memory and i think we still should do it otherwise we will see random crashes in applications as was before. I guess it's because TLBs and page tables are not updated to reflect the fact the device is gone.
Hi,
I was talking about existing mappings. What I forgot to specify is how new
mmap()calls after the device disappearance should work (the end result should be the same still, not failure).Quoted 5 lines
I'll clarify this in the next revision.
Thanks,
pqI see, that ok.
Quoted 10 lines
Next related question is more for Daniel/Christian - about the implementation of this paragraph, I was thinking about something like checking for device disconnect in
ttm_bo_vm_fault_reservedand if so remap the entire VA range for the VMA where the fault address belongs to the global zero page (i.e. (remap_pfn_range(vma,vma->vm_start,page_to_pfn(ZERO_PAGE(vma->vm_start), vma->vm_end - vma->vm_start, vma->vm_page_prot)). Question is, when the doc says 'writes are ignored' does it mean i should use copy on write for thevma->vm_page_protand if so how i actually do it as i was not able to find what flags to set intovm_page_protto force copy on write behavior.
Already discussed this with Pekka on irc, I think simply a private page (per gpu ctx to avoid leaks) is good enough. Otherwise we need to catch write faults and throw the writes away, and that's a) a bit tricky to implement and b) slow, which we kinda don't want to. If the desktop is stuck for a few seconds because we're trapping every write of a 4k buffer that's getting uploaded, the user is going to have a bad time :-/
-Daniel
Quoted 35 lines
Andrey
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Documentation/gpu/drm-uapi.rst +24
-0+ +- dmabuf which point to memory that has disappeared are rewritten to + point to harmless zero pages, similar to mmaps. Imports still succeed + both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing any dmabuf. + +- Render ioctls return EIO which is then handled in userspace drivers, + e.g. Mesa, to have the device disappearance handled in the way + specified for each API (OpenGL, GL ES: GL_KHR_robustness; + Vulkan: VK_ERROR_DEVICE_LOST; etc.) + +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like Mesa produces. Signal +handlers are not composable, you can't have different handlers for GPU1 +and GPU2 from different vendors, and a third handler for mmapped regular +files. Threads cause additional pain with signal handling as well. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 123 lines · Daniel Vetter
Quoted 111 lines · Andrey Grodzovsky
Quoted 101 lines · Pekka Paalanen
Quoted 84 lines · On Tue, 26 May 2020 10:30:20 -0400 Andrey Grodzovsky…
Quoted 83 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7CAndrey.Grodzovsky%40amd.com%7C3c671803b2ba41b2ceac08d8024bcc9a%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637261871869742519&sdata=ZnhylRubOM0%2BjoreSSYMqVDzZuUdybEsoyBVcTKgxWE%3D&reserved=0 .Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>---
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 75 ++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)Documentation/gpu/drm-uapi.rst +48
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,79 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May 13). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some cases specific +ioctls returning EIO. + +This goal raises at least the following requirements for the kernel and +drivers: + +- The kernel must not hang, crash or oops, no matter what userspace was + in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. +Quoted 8 lines
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 215
Quoted 3 lines
Documentation/gpu/drm-uapi.rst +3
-0+- If underlying memory disappears, the mmaps are replaced with harmless + zero pages where access does not raise SIGBUS. Reads return zeros, + writes are ignored.Regarding this paragraph - what about exiting mappings ? In the first patchset we would actively invalidate all the existing CPU mappings to device memory and i think we still should do it otherwise we will see random crashes in applications as was before. I guess it's because TLBs and page tables are not updated to reflect the fact the device is gone.
Hi,
I was talking about existing mappings. What I forgot to specify is how new
mmap()calls after the device disappearance should work (the end result should be the same still, not failure).Quoted 5 lines
I'll clarify this in the next revision.
Thanks,
pqI see, that ok.
Quoted 10 lines
Next related question is more for Daniel/Christian - about the implementation of this paragraph, I was thinking about something like checking for device disconnect in
ttm_bo_vm_fault_reservedand if so remap the entire VA range for the VMA where the fault address belongs to the global zero page (i.e. (remap_pfn_range(vma,vma->vm_start,page_to_pfn(ZERO_PAGE(vma->vm_start), vma->vm_end - vma->vm_start, vma->vm_page_prot)). Question is, when the doc says 'writes are ignored' does it mean i should use copy on write for thevma->vm_page_protand if so how i actually do it as i was not able to find what flags to set intovm_page_protto force copy on write behavior.
Quoted 8 lines
Already discussed this with Pekka on irc, I think simply a private page (per gpu ctx to avoid leaks) is good enough. Otherwise we need to catch write faults and throw the writes away, and that's a) a bit tricky to implement and b) slow, which we kinda don't want to. If the desktop is stuck for a few seconds because we're trapping every write of a 4k buffer that's getting uploaded, the user is going to have a bad time :-/
-Daniel
So like allocating a page per process context in the driver (struct amdgpu_ctx in amdgpu) and mapping this page into the faulting VMAs for when device is disconnected ? I am still not clear how i make the mapping ignore writes without catching write faults and ignoring them. I cannot just make it read only obviously and i can't make it writable as then reading back will start returning non 0's. My question is what set of flags in vm_area_struct.vm_flags can (if at all) give me 'ignore writes' behavior for the mapping of that page.
Andrey
Quoted 33 lines
Quoted 33 lines
Andrey
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Documentation/gpu/drm-uapi.rst +24
-0+ +- dmabuf which point to memory that has disappeared are rewritten to + point to harmless zero pages, similar to mmaps. Imports still succeed + both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing any dmabuf. + +- Render ioctls return EIO which is then handled in userspace drivers, + e.g. Mesa, to have the device disappearance handled in the way + specified for each API (OpenGL, GL ES: GL_KHR_robustness; + Vulkan: VK_ERROR_DEVICE_LOST; etc.) + +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like Mesa produces. Signal +handlers are not composable, you can't have different handlers for GPU1 +and GPU2 from different vendors, and a third handler for mmapped regular +files. Threads cause additional pain with signal handling as well. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + .. _drm_driver_ioctl:
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 242
Quoted 1 line
Documentation/gpu/drm-uapi.rst +0
-0IOCTL Support on Device Nodes
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Am 27.05.20 um 17:23 schrieb Andrey Grodzovsky:
Quoted 155 lines · Daniel Vetter
Quoted 143 lines
Quoted 131 lines · Andrey Grodzovsky
Quoted 121 lines · Pekka Paalanen
Quoted 104 lines · On Tue, 26 May 2020 10:30:20 -0400 Andrey Grodzovsky…
Quoted 103 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7CAndrey.Grodzovsky%40amd.com%7C3c671803b2ba41b2ceac08d8024bcc9a%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637261871869742519&sdata=ZnhylRubOM0%2BjoreSSYMqVDzZuUdybEsoyBVcTKgxWE%3D&reserved=0 .Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>---
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
---
Documentation/gpu/drm-uapi.rst| 75
++++++++++++++++++++++++++++++++++1 file changed, 75 insertions(+)Documentation/gpu/drm-uapi.rst +2
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. +===================
Userland interfaces
===================Documentation/gpu/drm-uapi.rst +0
-0@@ -162,6 +164,79 @@ other hand, a driver requires shared statebetween clients which is
visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes.Documentation/gpu/drm-uapi.rst +8
-0+Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May 13). + +Graphics devices (display and/or render) may be connected viaUSB (e.g.
+display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end
+user is able to hot-unplug this kind of devices while they are being
+used, and expects that the very least the machine does not crash. Any
+damage from hot-unplugging a DRM device needs to be limited as much as
+possible and userspace must be given the chance to handle it if it wants
+to. Ideally, unplugging a DRM device still lets a desktop to continue
+running, but that is going to need explicit support throughout the wholeDocumentation/gpu/drm-uapi.rst +2
-0+graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications andlibraries.
Documentation/gpu/drm-uapi.rst +3
-0+ +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind ofa driver
Documentation/gpu/drm-uapi.rst +3
-0+from the physical device. + +In other words, from userspace perspective everything needs tokeep on
+working more or less, until userspace stops using the disappeared DRMDocumentation/gpu/drm-uapi.rst +2
-0+device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some casesspecific
Documentation/gpu/drm-uapi.rst +3
-0+ioctls returning EIO. + +This goal raises at least the following requirements for thekernel and
Documentation/gpu/drm-uapi.rst +3
-0+drivers: + +- The kernel must not hang, crash or oops, no matter whatuserspace was
+ in the middle of doing when the device disappeared.Documentation/gpu/drm-uapi.rst +11
-0+ +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success. + +- Pending non-blocking KMS operations deliver the DRM eventsuserspace
+ is expecting.
+
+- If underlying memory disappears, the mmaps are replaced with harmlessQuoted 8 lines
Quoted 3 lines
+ zero pages where access does not raise SIGBUS. Reads return zeros,
+ writes are ignored.Regarding this paragraph - what about exiting mappings ? In the first patchset we would actively invalidate all the existing CPU mappings to device memory and i think we still should do it otherwise we will see random crashes in applications as was before. I guess it's because TLBs and page tables are not updated to reflect the fact the device is gone.
Hi,
I was talking about existing mappings. What I forgot to specify is how new
mmap()calls after the device disappearance should work (the end result should be the same still, not failure).Quoted 5 lines
I'll clarify this in the next revision.
Thanks,
pqI see, that ok.
Quoted 10 lines
Next related question is more for Daniel/Christian - about the implementation of this paragraph, I was thinking about something like checking for device disconnect in
ttm_bo_vm_fault_reservedand if so remap the entire VA range for the VMA where the fault address belongs to the global zero page (i.e. (remap_pfn_range(vma,vma->vm_start,page_to_pfn(ZERO_PAGE(vma->vm_start), vma->vm_end - vma->vm_start, vma->vm_page_prot)). Question is, when the doc says 'writes are ignored' does it mean i should use copy on write for thevma->vm_page_protand if so how i actually do it as i was not able to find what flags to set intovm_page_protto force copy on write behavior.Quoted 8 lines
Already discussed this with Pekka on irc, I think simply a private page (per gpu ctx to avoid leaks) is good enough. Otherwise we need to catch write faults and throw the writes away, and that's a) a bit tricky to implement and b) slow, which we kinda don't want to. If the desktop is stuck for a few seconds because we're trapping every write of a 4k buffer that's getting uploaded, the user is going to have a bad time :-/
-Daniel
Quoted 1 line
So like allocating a page per process context in the driver (struct
amdgpu_ctxin amdgpu) and mapping this page into the faulting VMAs for when device is disconnected ? I am still not clear how i make the mapping ignore writes without catching write faults and ignoring them. I cannot just make it read only obviously and i can't make it writable as then reading back will start returning non 0's. My question is what set of flags in vm_area_struct.vm_flags can (if at all) give me 'ignore writes' behavior for the mapping of that page.
I'm not aware of a possibility like that on x86 CPUs. As far as I know we only have something like an ignore write functionality on our GPUs for PRTs.
Could we use an address which points to a non allocated MMIO space or something like this? We would might get 0xffffffff on reads instead of 0x0, but writes would be certainly ignored.
Christian.
Quoted 49 lines
Andrey
Quoted 45 lines
Quoted 43 lines
Andrey
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
+
+- dmabuf which point to memory that has disappeared are rewritten to
+ point to harmless zero pages, similar to mmaps. Imports still succeed
+ both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing any dmabuf.
+
+- Render ioctls return EIO which is then handled in userspace drivers,
+ e.g. Mesa, to have the device disappearance handled in the way + specified for each API (OpenGL, GL ES:GL_KHR_robustness; + Vulkan:VK_ERROR_DEVICE_LOST; etc.)Documentation/gpu/drm-uapi.rst +2
-0+ +Raising SIGBUS is not an option, because userspace cannotrealistically
Documentation/gpu/drm-uapi.rst +2
-0+handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like Mesa produces.Signal
+handlers are not composable, you can't have different handlers for GPU1
+and GPU2 from different vendors, and a third handler for mmapped regularDocumentation/gpu/drm-uapi.rst +3
-0+files. Threads cause additional pain with signal handling as well. + +Only after userspace has closed all relevant DRM device anddmabuf file
Documentation/gpu/drm-uapi.rst +5
-0+descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. +..
_drm_driver_ioctl:IOCTL Support on Device Nodes
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 181 lines · Christian König
Am 27.05.20 um 17:23 schrieb Andrey Grodzovsky:
Quoted 170 lines · Daniel Vetter
Quoted 158 lines
Quoted 142 lines · Andrey Grodzovsky
Quoted 132 lines · Pekka Paalanen
Quoted 115 lines · On Tue, 26 May 2020 10:30:20 -0400 Andrey Grodzovsky…
Quoted 110 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to
userspace.Written by Daniel Vetter's request and largely based on his comments in IRC and
from
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7CAndrey.Grodzovsky%40amd.com%7C3c671803b2ba41b2ceac08d8024bcc9a%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637261871869742519&sdata=ZnhylRubOM0%2BjoreSSYMqVDzZuUdybEsoyBVcTKgxWE%3D&reserved=0
.Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>---
Disclaimer: I am a userspace developer writing for other userspace developers.
I took some liberties in defining what should happen without knowing what is
actually possible or what existing drivers already implement.--- Documentation/gpu/drm-uapi.rst | 75 ++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)Documentation/gpu/drm-uapi.rst +2
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,79 @@ other hand, a driver requires shared statebetween clients which is
visible to user-space and accessible beyond open-file
boundaries, they
cannot support render nodes.Documentation/gpu/drm-uapi.rst +8
-0+Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May 13). + +Graphics devices (display and/or render) may be connected viaUSB (e.g.
+display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end
+user is able to hot-unplug this kind of devices while they are being
+used, and expects that the very least the machine does not crash. Any
+damage from hot-unplugging a DRM device needs to be limited as much as
+possible and userspace must be given the chance to handle it if it wants
+to. Ideally, unplugging a DRM device still lets a desktop to continue
+running, but that is going to need explicit support throughout the wholeDocumentation/gpu/drm-uapi.rst +2
-0+graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications andlibraries.
Documentation/gpu/drm-uapi.rst +3
-0+ +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind ofa driver
Documentation/gpu/drm-uapi.rst +3
-0+from the physical device. + +In other words, from userspace perspective everything needs tokeep on
+working more or less, until userspace stops using the disappeared DRMDocumentation/gpu/drm-uapi.rst +2
-0+device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some casesspecific
Documentation/gpu/drm-uapi.rst +3
-0+ioctls returning EIO. + +This goal raises at least the following requirements for thekernel and
Documentation/gpu/drm-uapi.rst +3
-0+drivers: + +- The kernel must not hang, crash or oops, no matter whatuserspace was
Documentation/gpu/drm-uapi.rst +12
-0+ in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success. + +- Pending non-blocking KMS operations deliver the DRM eventsuserspace
Documentation/gpu/drm-uapi.rst +3
-0+ is expecting. + +- If underlying memory disappears, the mmaps are replaced withQuoted 4 lines
harmless
+ zero pages where access does not raise SIGBUS. Reads return zeros,
+ writes are ignored.Quoted 8 lines
Regarding this paragraph - what about exiting mappings ? In the first patchset we would actively invalidate all the existing CPU mappings to
device memory and i think we still should do it otherwise we will see random crashes in applications as was before. I guess it's because TLBs
and page tables are not updated to reflect the fact the device is gone.Hi,
I was talking about existing mappings. What I forgot to specify is how new
mmap()calls after the device disappearance should work (the end result should be the same still, not failure).Quoted 5 lines
I'll clarify this in the next revision.
Thanks,
pqI see, that ok.
Quoted 14 lines
Next related question is more for Daniel/Christian - about the implementation of this paragraph, I was thinking about something like checking for device disconnect in
ttm_bo_vm_fault_reservedand if so remap the entire VA range for the VMA where the fault address belongs to
the global zero page (i.e. (remap_pfn_range(vma,vma->vm_start,page_to_pfn(ZERO_PAGE(vma->vm_start), vma->vm_end - vma->vm_start, vma->vm_page_prot)). Question is, when the doc says 'writes are ignored'
does it mean i should use copy on write for thevma->vm_page_protand if
so how i actually do it as i was not able to find what flags to set intovm_page_protto force copy on write behavior.Quoted 8 lines
Already discussed this with Pekka on irc, I think simply a private page (per gpu ctx to avoid leaks) is good enough. Otherwise we need to catch write faults and throw the writes away, and that's a) a bit tricky to implement and b) slow, which we kinda don't want to. If the desktop is stuck for a few seconds because we're trapping every write of a 4k buffer that's getting uploaded, the user is going to have a bad time :-/
-DanielQuoted 8 lines
So like allocating a page per process context in the driver (struct
amdgpu_ctxin amdgpu) and mapping this page into the faulting VMAs for when device is disconnected ? I am still not clear how i make the mapping ignore writes without catching write faults and ignoring them. I cannot just make it read only obviously and i can't make it writable as then reading back will start returning non 0's. My question is what set of flags in vm_area_struct.vm_flags can (if at all) give me 'ignore writes' behavior for the mapping of that page.
Quoted 7 lines
I'm not aware of a possibility like that on x86 CPUs. As far as I know we only have something like an ignore write functionality on our GPUs for PRTs.
Could we use an address which points to a non allocated MMIO space or something like this? We would might get
0xffffffffon reads instead of0x0, but writes would be certainly ignored.
I think just a page with garbage in, garbage out semantics is going to be ok. I think pretty much anything has a chance to upset userspace, so whether it's 0 or all 1s or anything else doesn't really matter.
Only thing that does matter a bit is that we have a page per fd, so that we don't accidentally leak something between processes where we shouldn't. I think as long as we don't crash&burn in a SIGBUS it's good enough.
-Daniel
Quoted 53 lines
Christian.
Quoted 49 lines
Andrey
Quoted 45 lines
Quoted 43 lines
Andrey
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
+
+- dmabuf which point to memory that has disappeared are rewritten to
+ point to harmless zero pages, similar to mmaps. Imports still succeedDocumentation/gpu/drm-uapi.rst +2
-0+ both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing anydmabuf.
+
+- Render ioctls return EIO which is then handled in userspace drivers,Documentation/gpu/drm-uapi.rst +5
-0+ e.g. Mesa, to have the device disappearance handled in the way + specified for each API (OpenGL, GL ES: GL_KHR_robustness; + Vulkan: VK_ERROR_DEVICE_LOST; etc.) + +Raising SIGBUS is not an option, because userspace cannotrealistically
Documentation/gpu/drm-uapi.rst +2
-0+handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like Mesa produces.Signal
+handlers are not composable, you can't have different handlers for GPU1
+and GPU2 from different vendors, and a third handler for mmapped regularDocumentation/gpu/drm-uapi.rst +3
-0+files. Threads cause additional pain with signal handling as well. + +Only after userspace has closed all relevant DRM device anddmabuf file
Documentation/gpu/drm-uapi.rst +5
-0+descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 43 lines · On Wed, 27 May 2020 22:25:00 +0200 Daniel Vetter <[email protected]> wrote:
Quoted 41 lines · Christian König
Am 27.05.20 um 17:23 schrieb Andrey Grodzovsky:
Quoted 39 lines · Daniel Vetter
Quoted 37 lines
Quoted 35 lines · Andrey Grodzovsky
Quoted 33 lines · Pekka Paalanen
Quoted 30 lines · On Tue, 26 May 2020 10:30:20 -0400 Andrey Grodzovsky…
Quoted 29 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to
userspace.Written by Daniel Vetter's request and largely based on his comments in IRC and
from
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7CAndrey.Grodzovsky%40amd.com%7C3c671803b2ba41b2ceac08d8024bcc9a%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637261871869742519&sdata=ZnhylRubOM0%2BjoreSSYMqVDzZuUdybEsoyBVcTKgxWE%3D&reserved=0
.Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>---
Disclaimer: I am a userspace developer writing for other userspace developers.
I took some liberties in defining what should happen without knowing what is
actually possible or what existing drivers already implement.--- Documentation/gpu/drm-uapi.rst | 75 ++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)
...
Quoted 41 lines
Quoted 33 lines
Quoted 24 lines
Quoted 14 lines
Quoted 6 lines
Next related question is more for Daniel/Christian - about the implementation of this paragraph, I was thinking about something like checking for device disconnect in
ttm_bo_vm_fault_reservedand if so remap the entire VA range for the VMA where the fault address belongs to
the global zero page (i.e. (remap_pfn_range(vma,vma->vm_start,Quoted 8 lines
page_to_pfn(ZERO_PAGE(vma->vm_start), vma->vm_end - vma->vm_start, vma->vm_page_prot)). Question is, when the doc says 'writes are ignored'
does it mean i should use copy on write for thevma->vm_page_protand if
so how i actually do it as i was not able to find what flags to set intovm_page_protto force copy on write behavior.Quoted 8 lines
Already discussed this with Pekka on irc, I think simply a private page (per gpu ctx to avoid leaks) is good enough. Otherwise we need to catch write faults and throw the writes away, and that's a) a bit tricky to implement and b) slow, which we kinda don't want to. If the desktop is stuck for a few seconds because we're trapping every write of a 4k buffer that's getting uploaded, the user is going to have a bad time :-/
-DanielQuoted 8 lines
So like allocating a page per process context in the driver (struct
amdgpu_ctxin amdgpu) and mapping this page into the faulting VMAs for when device is disconnected ? I am still not clear how i make the mapping ignore writes without catching write faults and ignoring them. I cannot just make it read only obviously and i can't make it writable as then reading back will start returning non 0's. My question is what set of flags in vm_area_struct.vm_flags can (if at all) give me 'ignore writes' behavior for the mapping of that page.Quoted 7 lines
I'm not aware of a possibility like that on x86 CPUs. As far as I know we only have something like an ignore write functionality on our GPUs for PRTs.
Could we use an address which points to a non allocated MMIO space or something like this? We would might get
0xffffffffon reads instead of0x0, but writes would be certainly ignored.
Quoted 8 lines
I think just a page with garbage in, garbage out semantics is going to be ok. I think pretty much anything has a chance to upset userspace, so whether it's 0 or all 1s or anything else doesn't really matter.
Only thing that does matter a bit is that we have a page per fd, so that we don't accidentally leak something between processes where we shouldn't. I think as long as we don't crash&burn in a SIGBUS it's good enough.
Hi,
the v2 I sent on Monday already changed the wording to have undefined reads/writes instead of read zero / ignore write. v3 is coming.
Thanks,
pq
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 188 lines · Daniel Vetter
Quoted 175 lines · Christian König
Am 27.05.20 um 17:23 schrieb Andrey Grodzovsky:
Quoted 166 lines · Daniel Vetter
Quoted 156 lines
Quoted 140 lines · Andrey Grodzovsky
Quoted 132 lines · Pekka Paalanen
Quoted 115 lines · On Tue, 26 May 2020 10:30:20 -0400 Andrey Grodzovsky…
Quoted 110 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to
userspace.Written by Daniel Vetter's request and largely based on his comments in IRC and
from
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7CAndrey.Grodzovsky%40amd.com%7Cd1aab2c6fe71407a287708d8027c0f3c%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637262079143242123&sdata=krqBSHMfzl%2F4TMaAgEPDq8Y%2BPYWJATZyeDPfhtWQmeg%3D&reserved=0
.Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>---
Disclaimer: I am a userspace developer writing for other userspace developers.
I took some liberties in defining what should happen without knowing what is
actually possible or what existing drivers already implement.--- Documentation/gpu/drm-uapi.rst | 75 ++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)Documentation/gpu/drm-uapi.rst +2
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,79 @@ other hand, a driver requires shared statebetween clients which is
visible to user-space and accessible beyond open-file
boundaries, they
cannot support render nodes.Documentation/gpu/drm-uapi.rst +8
-0+Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May 13). + +Graphics devices (display and/or render) may be connected viaUSB (e.g.
+display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end
+user is able to hot-unplug this kind of devices while they are being
+used, and expects that the very least the machine does not crash. Any
+damage from hot-unplugging a DRM device needs to be limited as much as
+possible and userspace must be given the chance to handle it if it wants
+to. Ideally, unplugging a DRM device still lets a desktop to continue
+running, but that is going to need explicit support throughout the wholeDocumentation/gpu/drm-uapi.rst +2
-0+graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications andlibraries.
Documentation/gpu/drm-uapi.rst +3
-0+ +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind ofa driver
Documentation/gpu/drm-uapi.rst +3
-0+from the physical device. + +In other words, from userspace perspective everything needs tokeep on
+working more or less, until userspace stops using the disappeared DRMDocumentation/gpu/drm-uapi.rst +2
-0+device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent or in some casesspecific
Documentation/gpu/drm-uapi.rst +3
-0+ioctls returning EIO. + +This goal raises at least the following requirements for thekernel and
Documentation/gpu/drm-uapi.rst +3
-0+drivers: + +- The kernel must not hang, crash or oops, no matter whatuserspace was
Documentation/gpu/drm-uapi.rst +12
-0+ in the middle of doing when the device disappeared. + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips fake success. + +- Atomic commits, both real and TEST_ONLY, fake success. + +- Pending non-blocking KMS operations deliver the DRM eventsuserspace
Documentation/gpu/drm-uapi.rst +3
-0+ is expecting. + +- If underlying memory disappears, the mmaps are replaced withQuoted 4 lines
harmless
+ zero pages where access does not raise SIGBUS. Reads return zeros,
+ writes are ignored.Quoted 8 lines
Regarding this paragraph - what about exiting mappings ? In the first patchset we would actively invalidate all the existing CPU mappings to
device memory and i think we still should do it otherwise we will see random crashes in applications as was before. I guess it's because TLBs
and page tables are not updated to reflect the fact the device is gone.Hi,
I was talking about existing mappings. What I forgot to specify is how new
mmap()calls after the device disappearance should work (the end result should be the same still, not failure).Quoted 5 lines
I'll clarify this in the next revision.
Thanks,
pqI see, that ok.
Quoted 14 lines
Next related question is more for Daniel/Christian - about the implementation of this paragraph, I was thinking about something like checking for device disconnect in
ttm_bo_vm_fault_reservedand if so remap the entire VA range for the VMA where the fault address belongs to
the global zero page (i.e. (remap_pfn_range(vma,vma->vm_start,page_to_pfn(ZERO_PAGE(vma->vm_start), vma->vm_end - vma->vm_start, vma->vm_page_prot)). Question is, when the doc says 'writes are ignored'
does it mean i should use copy on write for thevma->vm_page_protand if
so how i actually do it as i was not able to find what flags to set intovm_page_protto force copy on write behavior.Quoted 8 lines
Already discussed this with Pekka on irc, I think simply a private page (per gpu ctx to avoid leaks) is good enough. Otherwise we need to catch write faults and throw the writes away, and that's a) a bit tricky to implement and b) slow, which we kinda don't want to. If the desktop is stuck for a few seconds because we're trapping every write of a 4k buffer that's getting uploaded, the user is going to have a bad time :-/
-DanielQuoted 8 lines
So like allocating a page per process context in the driver (struct
amdgpu_ctxin amdgpu) and mapping this page into the faulting VMAs for when device is disconnected ? I am still not clear how i make the mapping ignore writes without catching write faults and ignoring them. I cannot just make it read only obviously and i can't make it writable as then reading back will start returning non 0's. My question is what set of flags in vm_area_struct.vm_flags can (if at all) give me 'ignore writes' behavior for the mapping of that page.Quoted 7 lines
I'm not aware of a possibility like that on x86 CPUs. As far as I know we only have something like an ignore write functionality on our GPUs for PRTs.
Could we use an address which points to a non allocated MMIO space or something like this? We would might get
0xffffffffon reads instead of0x0, but writes would be certainly ignored.I think just a page with garbage in, garbage out semantics is going to be ok. I think pretty much anything has a chance to upset userspace, so whether it's 0 or all 1s or anything else doesn't really matter.
Quoted 5 lines
Only thing that does matter a bit is that we have a page per fd, so that we don't accidentally leak something between processes where we shouldn't. I think as long as we don't crash&burn in a SIGBUS it's good enough.
-Daniel
To use non allocated MMIO space i would need first to know which range is currently not used (how ?) and then reserve it (and free later) to avoid other devices start using it. I think the interface for this is https://elixir.bootlin.com/linux/v5.7-rc7/source/include/linux/ioport.h#L233. But still i like the zero page approach more where we map the zero page during new page faults into the faulting process page table with setting adding ~(VM_SHARED | VM_MAYSHARE) to vma->vm_flags or at least to pgprot_t for this particular maping which i think makes the mapping copy on write and so each process (each FD) will also not leak data to other processes.
Andrey
Quoted 46 lines
Quoted 46 lines
Christian.
Quoted 43 lines
Andrey
Quoted 40 lines
Quoted 40 lines
Andrey
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
+
+- dmabuf which point to memory that has disappeared are rewritten to
+ point to harmless zero pages, similar to mmaps. Imports still succeedDocumentation/gpu/drm-uapi.rst +2
-0+ both ways: an existing device importing a dmabuf pointing to + disappeared memory, and a disappeared device importing anydmabuf.
+
+- Render ioctls return EIO which is then handled in userspace drivers,Documentation/gpu/drm-uapi.rst +5
-0+ e.g. Mesa, to have the device disappearance handled in the way + specified for each API (OpenGL, GL ES: GL_KHR_robustness; + Vulkan: VK_ERROR_DEVICE_LOST; etc.) + +Raising SIGBUS is not an option, because userspace cannotrealistically
Documentation/gpu/drm-uapi.rst +2
-0+handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like Mesa produces.Signal
+handlers are not composable, you can't have different handlers for GPU1
+and GPU2 from different vendors, and a third handler for mmapped regularDocumentation/gpu/drm-uapi.rst +3
-0+files. Threads cause additional pain with signal handling as well. + +Only after userspace has closed all relevant DRM device anddmabuf file
Documentation/gpu/drm-uapi.rst +5
-0+descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + .. _drm_driver_ioctl:
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
IOCTL Support on Device Nodes
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
PATCH v3 1/1 drm/doc: device hot-unplug for userspace
In reply to Pekka PaalanenFrom: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and from https://lists.freedesktop.org/archives/dri-devel/2020-May/265484.html .
A related Wayland protocol change proposal is at
https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/35
---
v3:
- update ENODEV doc (Daniel)
- clarify existing vs. new mmaps (Andrey)
- split into KMS and render/cross sections (Andrey, Daniel)
- open() returns ENXIO (open(2) man page)
- ioctls may return ENODEV (Andrey, Daniel)
- new wayland-protocols MR
v2:
- mmap reads/writes undefined (Daniel)
- make render ioctl behaviour driver-specific (Daniel)
- restructure the mmap paragraphs (Daniel)
- chardev minor notes (Simon)
- open behaviour (Daniel)
- DRM leasing behaviour (Daniel)
- added links
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 114 ++++++++++++++++++++++++++++++++- 1 file changed, 113 insertions(+), 1 deletion(-)
Documentation/gpu/drm-uapi.rst +113 -1
@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================
@@ -162,6 +164,116 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent, ioctls returning ENODEV +(or driver-specific ioctls returning driver-specific things), or open() +returning ENXIO. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + +Similar to PIDs, chardev minor numbers are not recycled immediately. A +new DRM device always picks the next free minor number compared to the +previous one allocated, and wraps around when minor numbers are +exhausted. + +The goal raises at least the following requirements for the kernel and +drivers. + +Requirements for KMS UAPI +------------------------- + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips, and atomic commits, both real and + TEST_ONLY, and any other ioctls either fail with ENODEV or fake + success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. This applies also to ioctls that faked success. + +- open() on a device node whose underlying device has disappeared will + fail with ENXIO. + +- Attempting to create a DRM lease on a disappeared DRM device will + fail with ENODEV. Existing DRM leases remain and work as listed + above. + +Requirements for Render and Cross-Device UAPI +--------------------------------------------- + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + The associated error code is ENODEV. + +- Some userspace APIs already define what should happen when the device + disappears (OpenGL, GL ES: `GL_KHR_robustness`_; `Vulkan`_: + VK_ERROR_DEVICE_LOST; etc.). DRM drivers are free to implement this + behaviour the way they see best, e.g. returning failures in + driver-specific ioctls and handling those in userspace drivers, or + rely on uevents, and so on. + +- dmabuf which point to memory that has disappeared will either fail to + import with ENODEV or continue to be successfully imported if it would + have succeeded before the disappearance. See also about memory maps + below for already imported dmabufs. + +- Attempting to import a dmabuf to a disappeared device will either fail + with ENODEV or succeed if it would have succeeded without the + disappearance. + +- open() on a device node whose underlying device has disappeared will + fail with ENXIO. + +.. _GL_KHR_robustness: https://www.khronos.org/registry/OpenGL/extensions/KHR/KHR_robustness.txt +.. _Vulkan: https://www.khronos.org/vulkan/ + +Requirements for Memory Maps +---------------------------- + +Memory maps have further requirements that apply to both existing maps +and maps created after the device has disappeared. If the underlying +memory disappeared, the map is created or modified such that reads and +writes will still complete successfully but the result is undefined. +This applies to both userspace mmap()'d memory and memory pointed to by +dmabuf which might be mapped to other devices (cross-device dmabuf +imports). + +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like those that Mesa produces. +Signal handlers are not composable, you can't have different handlers +for GPU1 and GPU2 from different vendors, and a third handler for +mmapped regular files. Threads cause additional pain with signal +handling as well. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes
@@ -199,7 +311,7 @@ EPERM/EACCES: difference between EACCES and EPERM. ENODEV: - The device is not (yet) present or fully initialized. + The device is not anymore present or is not yet fully initialized. EOPNOTSUPP: Feature (like PRIME, modesetting, GEM) is not supported by the driver.
Signature · 6 lines
2.20.1 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 149 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cfc4392da89ea4fc4281b08d806389835%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637266187434486889&sdata=Mbsx6qBXN9MBnDDJi4shRZobf7tjcClvNUlUCPsiVtw%3D&reserved=0 .A related Wayland protocol change proposal is at
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fwayland%2Fwayland-protocols%2F-%2Fmerge_requests%2F35&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cfc4392da89ea4fc4281b08d806389835%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637266187434486889&sdata=5j%2FNQFW0C1wvdnR%2BC0WgGU0JcNCb8fIYBPXFOmp36Ck%3D&reserved=0Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>Cc Simon Ser <[email protected]>---
v3:
- update ENODEV doc (Daniel)
- clarify existing vs. new mmaps (Andrey)
- split into KMS and render/cross sections (Andrey, Daniel)
-open()returns ENXIO (open(2) man page)
- ioctls may return ENODEV (Andrey, Daniel)
- new wayland-protocols MRv2:
- mmap reads/writes undefined (Daniel)
- make render ioctl behaviour driver-specific (Daniel)
- restructure the mmap paragraphs (Daniel)
- chardev minor notes (Simon)
- open behaviour (Daniel)
- DRM leasing behaviour (Daniel)
- added linksDisclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 114 ++++++++++++++++++++++++++++++++- 1 file changed, 113 insertions(+), 1 deletion(-)Documentation/gpu/drm-uapi.rst +93
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,116 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent, ioctls returning ENODEV +(or driver-specific ioctls returning driver-specific things), or open() +returning ENXIO. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + +Similar to PIDs, chardev minor numbers are not recycled immediately. A +new DRM device always picks the next free minor number compared to the +previous one allocated, and wraps around when minor numbers are +exhausted. + +The goal raises at least the following requirements for the kernel and +drivers. + +Requirements for KMS UAPI +------------------------- + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips, and atomic commits, both real and + TEST_ONLY, and any other ioctls either fail with ENODEV or fake + success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. This applies also to ioctls that faked success. + +- open() on a device node whose underlying device has disappeared will + fail with ENXIO. + +- Attempting to create a DRM lease on a disappeared DRM device will + fail with ENODEV. Existing DRM leases remain and work as listed + above. + +Requirements for Render and Cross-Device UAPI +--------------------------------------------- + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + The associated error code is ENODEV. + +- Some userspace APIs already define what should happen when the device + disappears (OpenGL, GL ES: `GL_KHR_robustness`_; `Vulkan`_: + VK_ERROR_DEVICE_LOST; etc.). DRM drivers are free to implement this + behaviour the way they see best, e.g. returning failures in + driver-specific ioctls and handling those in userspace drivers, or + rely on uevents, and so on. + +- dmabuf which point to memory that has disappeared will either fail to + import with ENODEV or continue to be successfully imported if it would + have succeeded before the disappearance. See also about memory maps + below for already imported dmabufs. + +- Attempting to import a dmabuf to a disappeared device will either fail + with ENODEV or succeed if it would have succeeded without the + disappearance. + +- open() on a device node whose underlying device has disappeared will + fail with ENXIO. + +.. _GL_KHR_robustness: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.khronos.org%2Fregistry%2FOpenGL%2Fextensions%2FKHR%2FKHR_robustness.txt&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cfc4392da89ea4fc4281b08d806389835%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637266187434486889&sdata=69J%2FRk9%2BX4Hew4UVZCvnVvBUFtKjU3lDz5DthyEvaR4%3D&reserved=0 +.. _Vulkan: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.khronos.org%2Fvulkan%2F&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cfc4392da89ea4fc4281b08d806389835%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637266187434486889&sdata=g%2F8ITu9GYokOr%2FgrwoDOKEWMcOV0EZGZDI4quRkjrVQ%3D&reserved=0 +
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 267
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+Requirements for Memory Maps +---------------------------- + +Memory maps have further requirements that apply to both existing maps +and maps created after the device has disappeared. If the underlying +memory disappeared, the map is created or modified such that reads and +writes will still complete successfully but the result is undefined. +This applies to both userspace mmap()'d memory and memory pointed to by +dmabuf which might be mapped to other devices (cross-device dmabuf +imports).
I haven't actually tested this yet but my guess is that for existing mappings nothing needs to be done as the result returned from reading/writing to pages backed by removed card will not fail since the page tables are in place but rather will return all 1s or just some undefined value for reads and nothing will happen for right - does this assumption makes sense ?
Andrey
Quoted 13 lines
Documentation/gpu/drm-uapi.rst +9
-0+ +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like those that Mesa produces. +Signal handlers are not composable, you can't have different handlers +for GPU1 and GPU2 from different vendors, and a third handler for +mmapped regular files. Threads cause additional pain with signal +handling as well. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes
Documentation/gpu/drm-uapi.rst EPERM/EACCES: · line 317
Quoted 8 lines
Documentation/gpu/drm-uapi.rst +1
-1@@ -199,7 +311,7 @@ EPERM/EACCES: difference between EACCES and EPERM. ENODEV: - The device is not (yet) present or fully initialized. + The device is not anymore present or is not yet fully initialized. EOPNOTSUPP: Feature (like PRIME, modesetting, GEM) is not supported by the driver.
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 162 lines · Andrey Grodzovsky
Quoted 148 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and
from https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2020-May%2F265484.html&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cfc4392da89ea4fc4281b08d806389835%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637266187434486889&sdata=Mbsx6qBXN9MBnDDJi4shRZobf7tjcClvNUlUCPsiVtw%3D&reserved=0 .A related Wayland protocol change proposal is at
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fwayland%2Fwayland-protocols%2F-%2Fmerge_requests%2F35&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cfc4392da89ea4fc4281b08d806389835%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637266187434486889&sdata=5j%2FNQFW0C1wvdnR%2BC0WgGU0JcNCb8fIYBPXFOmp36Ck%3D&reserved=0Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>Cc Simon Ser <[email protected]>---
v3:
- update ENODEV doc (Daniel)
- clarify existing vs. new mmaps (Andrey)
- split into KMS and render/cross sections (Andrey, Daniel)
-open()returns ENXIO (open(2) man page)
- ioctls may return ENODEV (Andrey, Daniel)
- new wayland-protocols MRv2:
- mmap reads/writes undefined (Daniel)
- make render ioctl behaviour driver-specific (Daniel)
- restructure the mmap paragraphs (Daniel)
- chardev minor notes (Simon)
- open behaviour (Daniel)
- DRM leasing behaviour (Daniel)
- added linksDisclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 114 ++++++++++++++++++++++++++++++++- 1 file changed, 113 insertions(+), 1 deletion(-)Documentation/gpu/drm-uapi.rst +93
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,116 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue +running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent, ioctls returning ENODEV +(or driver-specific ioctls returning driver-specific things), or open() +returning ENXIO. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + +Similar to PIDs, chardev minor numbers are not recycled immediately. A +new DRM device always picks the next free minor number compared to the +previous one allocated, and wraps around when minor numbers are +exhausted. + +The goal raises at least the following requirements for the kernel and +drivers. + +Requirements for KMS UAPI +------------------------- + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips, and atomic commits, both real and + TEST_ONLY, and any other ioctls either fail with ENODEV or fake + success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. This applies also to ioctls that faked success. + +- open() on a device node whose underlying device has disappeared will + fail with ENXIO. + +- Attempting to create a DRM lease on a disappeared DRM device will + fail with ENODEV. Existing DRM leases remain and work as listed + above. + +Requirements for Render and Cross-Device UAPI +--------------------------------------------- + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace. + The associated error code is ENODEV. + +- Some userspace APIs already define what should happen when the device + disappears (OpenGL, GL ES: `GL_KHR_robustness`_; `Vulkan`_: + VK_ERROR_DEVICE_LOST; etc.). DRM drivers are free to implement this + behaviour the way they see best, e.g. returning failures in + driver-specific ioctls and handling those in userspace drivers, or + rely on uevents, and so on. + +- dmabuf which point to memory that has disappeared will either fail to + import with ENODEV or continue to be successfully imported if it would + have succeeded before the disappearance. See also about memory maps + below for already imported dmabufs. + +- Attempting to import a dmabuf to a disappeared device will either fail + with ENODEV or succeed if it would have succeeded without the + disappearance. + +- open() on a device node whose underlying device has disappeared will + fail with ENXIO. + +.. _GL_KHR_robustness: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.khronos.org%2Fregistry%2FOpenGL%2Fextensions%2FKHR%2FKHR_robustness.txt&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cfc4392da89ea4fc4281b08d806389835%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637266187434486889&sdata=69J%2FRk9%2BX4Hew4UVZCvnVvBUFtKjU3lDz5DthyEvaR4%3D&reserved=0 +.. _Vulkan: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.khronos.org%2Fvulkan%2F&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cfc4392da89ea4fc4281b08d806389835%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637266187434486889&sdata=g%2F8ITu9GYokOr%2FgrwoDOKEWMcOV0EZGZDI4quRkjrVQ%3D&reserved=0 +Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 266
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+Requirements for Memory Maps +---------------------------- + +Memory maps have further requirements that apply to both existing maps +and maps created after the device has disappeared. If the underlying +memory disappeared, the map is created or modified such that reads and +writes will still complete successfully but the result is undefined. +This applies to both userspace mmap()'d memory and memory pointed to by +dmabuf which might be mapped to other devices (cross-device dmabuf +imports).
Quoted 6 lines
I haven't actually tested this yet but my guess is that for existing mappings nothing needs to be done as the result returned from reading/writing to pages backed by removed card will not fail since the page tables are in place but rather will return all 1s or just some undefined value for reads and nothing will happen for right - does this assumption makes sense ?
I'm not sure that works with the pci core's assumption of routing pci bar in the mmio space ... E.g. if you plug in something else in the same thunderbolt port or pcie hotplug slot, then there might be something important at those addresses now. While userspace might still be busy hammer the same addresses through the old mappings.
So for perfect implemenation I think we need to remap these too, but for first cut probably not needed. Given that status quo is "kernel oopses". -Daniel
Quoted 22 lines
Andrey
Quoted 18 lines
Documentation/gpu/drm-uapi.rst +10
-1+ +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like those that Mesa produces. +Signal handlers are not composable, you can't have different handlers +for GPU1 and GPU2 from different vendors, and a third handler for +mmapped regular files. Threads cause additional pain with signal +handling as well. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes@@ -199,7 +311,7 @@ EPERM/EACCES: difference between EACCES and EPERM. ENODEV: - The device is not (yet) present or fully initialized. + The device is not anymore present or is not yet fully initialized. EOPNOTSUPP: Feature (like PRIME, modesetting, GEM) is not supported by the driver.
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 10 lines · On Mon, 1 Jun 2020 17:32:03 +0300 Pekka Paalanen <[email protected]> wrote:
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and from https://lists.freedesktop.org/archives/dri-devel/2020-May/265484.html .
A related Wayland protocol change proposal is at
https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/35Signed-off-by Pekka Paalanen <[email protected]>
Hi,
how's this version looking? Anything to fix?
Thanks,
pq
Quoted 20 lines
---
v3:
- update ENODEV doc (Daniel)
- clarify existing vs. new mmaps (Andrey)
- split into KMS and render/cross sections (Andrey, Daniel)
-open()returns ENXIO (open(2) man page)
- ioctls may return ENODEV (Andrey, Daniel)
- new wayland-protocols MRv2:
- mmap reads/writes undefined (Daniel)
- make render ioctl behaviour driver-specific (Daniel)
- restructure the mmap paragraphs (Daniel)
- chardev minor notes (Simon)
- open behaviour (Daniel)
- DRM leasing behaviour (Daniel)
- added links
Quoted 6 lines
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 114 ++++++++++++++++++++++++++++++++- 1 file changed, 113 insertions(+), 1 deletion(-)
Sorry to trim like this, but gmail refuses to send my email otherwise.
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 62 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and from https://lists.freedesktop.org/archives/dri-devel/2020-May/265484.html .
A related Wayland protocol change proposal is at
https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/35Signed-off-by Pekka Paalanen <[email protected]>Cc Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>Cc Simon Ser <[email protected]>---
v3:
- update ENODEV doc (Daniel)
- clarify existing vs. new mmaps (Andrey)
- split into KMS and render/cross sections (Andrey, Daniel)
-open()returns ENXIO (open(2) man page)
- ioctls may return ENODEV (Andrey, Daniel)
- new wayland-protocols MRv2:
- mmap reads/writes undefined (Daniel)
- make render ioctl behaviour driver-specific (Daniel)
- restructure the mmap paragraphs (Daniel)
- chardev minor notes (Simon)
- open behaviour (Daniel)
- DRM leasing behaviour (Daniel)
- added linksDisclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 114 ++++++++++++++++++++++++++++++++- 1 file changed, 113 insertions(+), 1 deletion(-)Documentation/gpu/drm-uapi.rst +6
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,116 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note::
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 180
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+ The following is the plan. Implementation is not there yet + (2020 May). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop to continue
I think it's "continue to run" not "to continue running". But not native speaker either.
Quoted 43 lines
Documentation/gpu/drm-uapi.rst +43
-0+running, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent, ioctls returning ENODEV +(or driver-specific ioctls returning driver-specific things), or open() +returning ENXIO. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + +Similar to PIDs, chardev minor numbers are not recycled immediately. A +new DRM device always picks the next free minor number compared to the +previous one allocated, and wraps around when minor numbers are +exhausted. + +The goal raises at least the following requirements for the kernel and +drivers. + +Requirements for KMS UAPI +------------------------- + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips, and atomic commits, both real and + TEST_ONLY, and any other ioctls either fail with ENODEV or fake + success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. This applies also to ioctls that faked success. + +- open() on a device node whose underlying device has disappeared will + fail with ENXIO.
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 233
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+ +- Attempting to create a DRM lease on a disappeared DRM device will + fail with ENODEV. Existing DRM leases remain and work as listed + above. + +Requirements for Render and Cross-Device UAPI +--------------------------------------------- + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs to userspace.
I think "on userspace" not "to userspace"
Quoted 32 lines
Documentation/gpu/drm-uapi.rst +32
-0+ The associated error code is ENODEV. + +- Some userspace APIs already define what should happen when the device + disappears (OpenGL, GL ES: `GL_KHR_robustness`_; `Vulkan`_: + VK_ERROR_DEVICE_LOST; etc.). DRM drivers are free to implement this + behaviour the way they see best, e.g. returning failures in + driver-specific ioctls and handling those in userspace drivers, or + rely on uevents, and so on. + +- dmabuf which point to memory that has disappeared will either fail to + import with ENODEV or continue to be successfully imported if it would + have succeeded before the disappearance. See also about memory maps + below for already imported dmabufs. + +- Attempting to import a dmabuf to a disappeared device will either fail + with ENODEV or succeed if it would have succeeded without the + disappearance. + +- open() on a device node whose underlying device has disappeared will + fail with ENXIO. + +.. _GL_KHR_robustness: https://www.khronos.org/registry/OpenGL/extensions/KHR/KHR_robustness.txt +.. _Vulkan: https://www.khronos.org/vulkan/ + +Requirements for Memory Maps +---------------------------- + +Memory maps have further requirements that apply to both existing maps +and maps created after the device has disappeared. If the underlying +memory disappeared, the map is created or modified such that reads and +writes will still complete successfully but the result is undefined. +This applies to both userspace mmap()'d memory and memory pointed to by
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 275
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+dmabuf which might be mapped to other devices (cross-device dmabuf +imports). + +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like those that Mesa produces. +Signal handlers are not composable, you can't have different handlers +for GPU1 and GPU2 from different vendors, and a third handler for +mmapped regular files. Threads cause additional pain with signal +handling as well.
Only gap I could spot is additional other fd that float around, like dma-buf, sync_file, drm_syncobj and stuff like that. I guess we can figure these out as we go, which shouldn't be a big problem since this entire document is very much work in progres.
Thanks for pushing this forward.
-Daniel
Quoted 16 lines
Documentation/gpu/drm-uapi.rst +2
-1+ .. _drm_driver_ioctl: IOCTL Support on Device Nodes@@ -199,7 +311,7 @@ EPERM/EACCES: difference between EACCES and EPERM. ENODEV: - The device is not (yet) present or fully initialized. + The device is not anymore present or is not yet fully initialized. EOPNOTSUPP: Feature (like PRIME, modesetting, GEM) is not supported by the driver.Signature · 1 line
2.20.1
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
PATCH v4 1/1 drm/doc: device hot-unplug for userspace
In reply to Pekka PaalanenFrom: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and from https://lists.freedesktop.org/archives/dri-devel/2020-May/265484.html .
A related Wayland protocol change proposal is at
https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/35
---
Harry and Christian, could one of you ack this on behalf of AMD drivers?
Ben or Karol, could you ack on behalf of Nouveau?
Noralf, would this work for the tiny drivers etc.?
This is only about laying out plans for the future, not about what drivers do today. We'd just like to be sure the goals are reasonable and everyone is aware of the idea.
Thanks,
pq
v4:
- two typo fixes (Daniel)
v3:
- update ENODEV doc (Daniel)
- clarify existing vs. new mmaps (Andrey)
- split into KMS and render/cross sections (Andrey, Daniel)
- open() returns ENXIO (open(2) man page)
- ioctls may return ENODEV (Andrey, Daniel)
- new wayland-protocols MR
v2:
- mmap reads/writes undefined (Daniel)
- make render ioctl behaviour driver-specific (Daniel)
- restructure the mmap paragraphs (Daniel)
- chardev minor notes (Simon)
- open behaviour (Daniel)
- DRM leasing behaviour (Daniel)
- added links
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 114 ++++++++++++++++++++++++++++++++- 1 file changed, 113 insertions(+), 1 deletion(-)
Documentation/gpu/drm-uapi.rst +113 -1
@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================
@@ -162,6 +164,116 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop continue to +run, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent, ioctls returning ENODEV +(or driver-specific ioctls returning driver-specific things), or open() +returning ENXIO. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + +Similar to PIDs, chardev minor numbers are not recycled immediately. A +new DRM device always picks the next free minor number compared to the +previous one allocated, and wraps around when minor numbers are +exhausted. + +The goal raises at least the following requirements for the kernel and +drivers. + +Requirements for KMS UAPI +------------------------- + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips, and atomic commits, both real and + TEST_ONLY, and any other ioctls either fail with ENODEV or fake + success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. This applies also to ioctls that faked success. + +- open() on a device node whose underlying device has disappeared will + fail with ENXIO. + +- Attempting to create a DRM lease on a disappeared DRM device will + fail with ENODEV. Existing DRM leases remain and work as listed + above. + +Requirements for Render and Cross-Device UAPI +--------------------------------------------- + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs on userspace. + The associated error code is ENODEV. + +- Some userspace APIs already define what should happen when the device + disappears (OpenGL, GL ES: `GL_KHR_robustness`_; `Vulkan`_: + VK_ERROR_DEVICE_LOST; etc.). DRM drivers are free to implement this + behaviour the way they see best, e.g. returning failures in + driver-specific ioctls and handling those in userspace drivers, or + rely on uevents, and so on. + +- dmabuf which point to memory that has disappeared will either fail to + import with ENODEV or continue to be successfully imported if it would + have succeeded before the disappearance. See also about memory maps + below for already imported dmabufs. + +- Attempting to import a dmabuf to a disappeared device will either fail + with ENODEV or succeed if it would have succeeded without the + disappearance. + +- open() on a device node whose underlying device has disappeared will + fail with ENXIO. + +.. _GL_KHR_robustness: https://www.khronos.org/registry/OpenGL/extensions/KHR/KHR_robustness.txt +.. _Vulkan: https://www.khronos.org/vulkan/ + +Requirements for Memory Maps +---------------------------- + +Memory maps have further requirements that apply to both existing maps +and maps created after the device has disappeared. If the underlying +memory disappeared, the map is created or modified such that reads and +writes will still complete successfully but the result is undefined. +This applies to both userspace mmap()'d memory and memory pointed to by +dmabuf which might be mapped to other devices (cross-device dmabuf +imports). + +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like those that Mesa produces. +Signal handlers are not composable, you can't have different handlers +for GPU1 and GPU2 from different vendors, and a third handler for +mmapped regular files. Threads cause additional pain with signal +handling as well. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes
@@ -199,7 +311,7 @@ EPERM/EACCES: difference between EACCES and EPERM. ENODEV: - The device is not (yet) present or fully initialized. + The device is not anymore present or is not yet fully initialized. EOPNOTSUPP: Feature (like PRIME, modesetting, GEM) is not supported by the driver.
Signature · 6 lines
2.20.1 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 168 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and from https://lists.freedesktop.org/archives/dri-devel/2020-May/265484.html .
A related Wayland protocol change proposal is at
https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/35Signed-off-by Pekka Paalanen <[email protected]>Reviewed-by Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>Cc Simon Ser <[email protected]>Cc Noralf Trønnes <[email protected]>Cc Ben Skeggs <[email protected]>Cc Christian König <[email protected]>Cc Harry Wentland <[email protected]>Cc Karol Herbst <[email protected]>---
Harry and Christian, could one of you ack this on behalf of AMD drivers?
Ben or Karol, could you ack on behalf of Nouveau?
Noralf, would this work for the tiny drivers etc.?
This is only about laying out plans for the future, not about what drivers do today. We'd just like to be sure the goals are reasonable and everyone is aware of the idea.
Thanks,
pqv4:
- two typo fixes (Daniel)v3:
- update ENODEV doc (Daniel)
- clarify existing vs. new mmaps (Andrey)
- split into KMS and render/cross sections (Andrey, Daniel)
-open()returns ENXIO (open(2) man page)
- ioctls may return ENODEV (Andrey, Daniel)
- new wayland-protocols MRv2:
- mmap reads/writes undefined (Daniel)
- make render ioctl behaviour driver-specific (Daniel)
- restructure the mmap paragraphs (Daniel)
- chardev minor notes (Simon)
- open behaviour (Daniel)
- DRM leasing behaviour (Daniel)
- added linksDisclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 114 ++++++++++++++++++++++++++++++++- 1 file changed, 113 insertions(+), 1 deletion(-)Documentation/gpu/drm-uapi.rst +89
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,116 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop continue to +run, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent, ioctls returning ENODEV +(or driver-specific ioctls returning driver-specific things), or open() +returning ENXIO. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + +Similar to PIDs, chardev minor numbers are not recycled immediately. A +new DRM device always picks the next free minor number compared to the +previous one allocated, and wraps around when minor numbers are +exhausted. + +The goal raises at least the following requirements for the kernel and +drivers. + +Requirements for KMS UAPI +------------------------- + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips, and atomic commits, both real and + TEST_ONLY, and any other ioctls either fail with ENODEV or fake + success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. This applies also to ioctls that faked success. + +- open() on a device node whose underlying device has disappeared will + fail with ENXIO. + +- Attempting to create a DRM lease on a disappeared DRM device will + fail with ENODEV. Existing DRM leases remain and work as listed + above. + +Requirements for Render and Cross-Device UAPI +--------------------------------------------- + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs on userspace. + The associated error code is ENODEV. + +- Some userspace APIs already define what should happen when the device + disappears (OpenGL, GL ES: `GL_KHR_robustness`_; `Vulkan`_: + VK_ERROR_DEVICE_LOST; etc.). DRM drivers are free to implement this + behaviour the way they see best, e.g. returning failures in + driver-specific ioctls and handling those in userspace drivers, or + rely on uevents, and so on. + +- dmabuf which point to memory that has disappeared will either fail to + import with ENODEV or continue to be successfully imported if it would + have succeeded before the disappearance. See also about memory maps + below for already imported dmabufs. + +- Attempting to import a dmabuf to a disappeared device will either fail + with ENODEV or succeed if it would have succeeded without the + disappearance. + +- open() on a device node whose underlying device has disappeared will + fail with ENXIO.
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 263
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+ +.. _GL_KHR_robustness: https://www.khronos.org/registry/OpenGL/extensions/KHR/KHR_robustness.txt +.. _Vulkan: https://www.khronos.org/vulkan/ + +Requirements for Memory Maps +---------------------------- + +Memory maps have further requirements that apply to both existing maps +and maps created after the device has disappeared. If the underlying +memory disappeared, the map is created or modified such that reads and
disappeared -> disappears
Quoted 17 lines
Documentation/gpu/drm-uapi.rst +13
-0+writes will still complete successfully but the result is undefined. +This applies to both userspace mmap()'d memory and memory pointed to by +dmabuf which might be mapped to other devices (cross-device dmabuf +imports). + +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like those that Mesa produces. +Signal handlers are not composable, you can't have different handlers +for GPU1 and GPU2 from different vendors, and a third handler for +mmapped regular files. Threads cause additional pain with signal +handling as well. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes
Documentation/gpu/drm-uapi.rst EPERM/EACCES: · line 314
Quoted 5 lines
Documentation/gpu/drm-uapi.rst +1
-1@@ -199,7 +311,7 @@ EPERM/EACCES: difference between EACCES and EPERM. ENODEV: - The device is not (yet) present or fully initialized. + The device is not anymore present or is not yet fully initialized.
The ordering of this sentence should be fixed up like so:
The device is not present anymore or is not yet fully initialized.
With those fixed the patch is:
Alex
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +0
-1EOPNOTSUPP: Feature (like PRIME, modesetting, GEM) is not supported by the driver. --2.20.1
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Den 22.06.2020 16.05, skrev Pekka Paalanen:
Quoted 23 lines
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and from https://lists.freedesktop.org/archives/dri-devel/2020-May/265484.html .
A related Wayland protocol change proposal is at
https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/35Signed-off-by Pekka Paalanen <[email protected]>Reviewed-by Daniel Vetter <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>Cc Simon Ser <[email protected]>Cc Noralf Trønnes <[email protected]>Cc Ben Skeggs <[email protected]>Cc Christian König <[email protected]>Cc Harry Wentland <[email protected]>Cc Karol Herbst <[email protected]>
Quoted 9 lines
---
Harry and Christian, could one of you ack this on behalf of AMD drivers?
Ben or Karol, could you ack on behalf of Nouveau?
Noralf, would this work for the tiny drivers etc.?
Looks good to me:
Quoted 6 lines
This is only about laying out plans for the future, not about what drivers do today. We'd just like to be sure the goals are reasonable and everyone is aware of the idea.
Thanks,
pq
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
PATCH v5 1/1 drm/doc: device hot-unplug for userspace
In reply to Pekka PaalanenFrom: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and from https://lists.freedesktop.org/archives/dri-devel/2020-May/265484.html .
A related Wayland protocol change proposal is at
https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/35
---
This is only about laying out plans for the future, not about what drivers do today. We'd just like to be sure the goals are reasonable and everyone is aware of the idea.
Thanks,
pq
v5:
- two grammar fixes (Alex)
- added R-b/A-b, dropped extra Cc
v4:
- two typo fixes (Daniel)
v3:
- update ENODEV doc (Daniel)
- clarify existing vs. new mmaps (Andrey)
- split into KMS and render/cross sections (Andrey, Daniel)
- open() returns ENXIO (open(2) man page)
- ioctls may return ENODEV (Andrey, Daniel)
- new wayland-protocols MR
v2:
- mmap reads/writes undefined (Daniel)
- make render ioctl behaviour driver-specific (Daniel)
- restructure the mmap paragraphs (Daniel)
- chardev minor notes (Simon)
- open behaviour (Daniel)
- DRM leasing behaviour (Daniel)
- added links
Disclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 114 ++++++++++++++++++++++++++++++++- 1 file changed, 113 insertions(+), 1 deletion(-)
Documentation/gpu/drm-uapi.rst +113 -1
@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================
@@ -162,6 +164,116 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop continue to +run, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device +disappearance from the device removed uevent, ioctls returning ENODEV +(or driver-specific ioctls returning driver-specific things), or open() +returning ENXIO. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. + +Similar to PIDs, chardev minor numbers are not recycled immediately. A +new DRM device always picks the next free minor number compared to the +previous one allocated, and wraps around when minor numbers are +exhausted. + +The goal raises at least the following requirements for the kernel and +drivers. + +Requirements for KMS UAPI +------------------------- + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips, and atomic commits, both real and + TEST_ONLY, and any other ioctls either fail with ENODEV or fake + success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. This applies also to ioctls that faked success. + +- open() on a device node whose underlying device has disappeared will + fail with ENXIO. + +- Attempting to create a DRM lease on a disappeared DRM device will + fail with ENODEV. Existing DRM leases remain and work as listed + above. + +Requirements for Render and Cross-Device UAPI +--------------------------------------------- + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs on userspace. + The associated error code is ENODEV. + +- Some userspace APIs already define what should happen when the device + disappears (OpenGL, GL ES: `GL_KHR_robustness`_; `Vulkan`_: + VK_ERROR_DEVICE_LOST; etc.). DRM drivers are free to implement this + behaviour the way they see best, e.g. returning failures in + driver-specific ioctls and handling those in userspace drivers, or + rely on uevents, and so on. + +- dmabuf which point to memory that has disappeared will either fail to + import with ENODEV or continue to be successfully imported if it would + have succeeded before the disappearance. See also about memory maps + below for already imported dmabufs. + +- Attempting to import a dmabuf to a disappeared device will either fail + with ENODEV or succeed if it would have succeeded without the + disappearance. + +- open() on a device node whose underlying device has disappeared will + fail with ENXIO. + +.. _GL_KHR_robustness: https://www.khronos.org/registry/OpenGL/extensions/KHR/KHR_robustness.txt +.. _Vulkan: https://www.khronos.org/vulkan/ + +Requirements for Memory Maps +---------------------------- + +Memory maps have further requirements that apply to both existing maps +and maps created after the device has disappeared. If the underlying +memory disappears, the map is created or modified such that reads and +writes will still complete successfully but the result is undefined. +This applies to both userspace mmap()'d memory and memory pointed to by +dmabuf which might be mapped to other devices (cross-device dmabuf +imports). + +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like those that Mesa produces. +Signal handlers are not composable, you can't have different handlers +for GPU1 and GPU2 from different vendors, and a third handler for +mmapped regular files. Threads cause additional pain with signal +handling as well. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes
@@ -199,7 +311,7 @@ EPERM/EACCES: difference between EACCES and EPERM. ENODEV: - The device is not (yet) present or fully initialized. + The device is not present anymore or is not yet fully initialized. EOPNOTSUPP: Feature (like PRIME, modesetting, GEM) is not supported by the driver.
Signature · 6 lines
2.20.1 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 102 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and from https://lists.freedesktop.org/archives/dri-devel/2020-May/265484.html .
A related Wayland protocol change proposal is at
https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/35Signed-off-by Pekka Paalanen <[email protected]>Reviewed-by Daniel Vetter <[email protected]>Reviewed-by Alex Deucher <[email protected]>Acked-by Noralf Trønnes <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>Cc Simon Ser <[email protected]>Cc Ben Skeggs <[email protected]>Cc Karol Herbst <[email protected]>---
This is only about laying out plans for the future, not about what drivers do today. We'd just like to be sure the goals are reasonable and everyone is aware of the idea.
Thanks,
pqv5:
- two grammar fixes (Alex)
- added R-b/A-b, dropped extra Ccv4:
- two typo fixes (Daniel)v3:
- update ENODEV doc (Daniel)
- clarify existing vs. new mmaps (Andrey)
- split into KMS and render/cross sections (Andrey, Daniel)
-open()returns ENXIO (open(2) man page)
- ioctls may return ENODEV (Andrey, Daniel)
- new wayland-protocols MRv2:
- mmap reads/writes undefined (Daniel)
- make render ioctl behaviour driver-specific (Daniel)
- restructure the mmap paragraphs (Daniel)
- chardev minor notes (Simon)
- open behaviour (Daniel)
- DRM leasing behaviour (Daniel)
- added linksDisclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 114 ++++++++++++++++++++++++++++++++- 1 file changed, 113 insertions(+), 1 deletion(-)Documentation/gpu/drm-uapi.rst +27
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,116 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop continue to +run, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the device
Documentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 201
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+disappearance from the device removed uevent, ioctls returning ENODEV +(or driver-specific ioctls returning driver-specific things), or open() +returning ENXIO. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. +
I don't think we can make a general statement like this. Drivers might have kworker polling on the device or other asynchronous or repeating jobs it might want to get rid of as soon as possible. I think it should be fine for the kernel driver to tear down the instance as long as it doesn't conflict with the goal of system stability. And usually drivers have some kernel side state for clients anyway, so it could be handled in this layer instead.
Or maybe I am overlooking something and we really can't do that :/
Quoted 88 lines
Documentation/gpu/drm-uapi.rst +76
-2+Similar to PIDs, chardev minor numbers are not recycled immediately. A +new DRM device always picks the next free minor number compared to the +previous one allocated, and wraps around when minor numbers are +exhausted. + +The goal raises at least the following requirements for the kernel and +drivers. + +Requirements for KMS UAPI +------------------------- + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips, and atomic commits, both real and + TEST_ONLY, and any other ioctls either fail with ENODEV or fake + success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. This applies also to ioctls that faked success. + +- open() on a device node whose underlying device has disappeared will + fail with ENXIO. + +- Attempting to create a DRM lease on a disappeared DRM device will + fail with ENODEV. Existing DRM leases remain and work as listed + above. + +Requirements for Render and Cross-Device UAPI +--------------------------------------------- + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs on userspace. + The associated error code is ENODEV. + +- Some userspace APIs already define what should happen when the device + disappears (OpenGL, GL ES: `GL_KHR_robustness`_; `Vulkan`_: + VK_ERROR_DEVICE_LOST; etc.). DRM drivers are free to implement this + behaviour the way they see best, e.g. returning failures in + driver-specific ioctls and handling those in userspace drivers, or + rely on uevents, and so on. + +- dmabuf which point to memory that has disappeared will either fail to + import with ENODEV or continue to be successfully imported if it would + have succeeded before the disappearance. See also about memory maps + below for already imported dmabufs. + +- Attempting to import a dmabuf to a disappeared device will either fail + with ENODEV or succeed if it would have succeeded without the + disappearance. + +- open() on a device node whose underlying device has disappeared will + fail with ENXIO. + +.. _GL_KHR_robustness: https://www.khronos.org/registry/OpenGL/extensions/KHR/KHR_robustness.txt +.. _Vulkan: https://www.khronos.org/vulkan/ + +Requirements for Memory Maps +---------------------------- + +Memory maps have further requirements that apply to both existing maps +and maps created after the device has disappeared. If the underlying +memory disappears, the map is created or modified such that reads and +writes will still complete successfully but the result is undefined. +This applies to both userspace mmap()'d memory and memory pointed to by +dmabuf which might be mapped to other devices (cross-device dmabuf +imports). + +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like those that Mesa produces. +Signal handlers are not composable, you can't have different handlers +for GPU1 and GPU2 from different vendors, and a third handler for +mmapped regular files. Threads cause additional pain with signal +handling as well. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes@@ -199,7 +311,7 @@ EPERM/EACCES: difference between EACCES and EPERM. ENODEV: - The device is not (yet) present or fully initialized. + The device is not present anymore or is not yet fully initialized. EOPNOTSUPP: Feature (like PRIME, modesetting, GEM) is not supported by the driver. --
Quoted 2 lines
2.20.1
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 115 lines · Karol Herbst
Quoted 102 lines · Pekka Paalanen
From: Pekka Paalanen <[email protected]>
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and from https://lists.freedesktop.org/archives/dri-devel/2020-May/265484.html .
A related Wayland protocol change proposal is at
https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/35Signed-off-by Pekka Paalanen <[email protected]>Reviewed-by Daniel Vetter <[email protected]>Reviewed-by Alex Deucher <[email protected]>Acked-by Noralf Trønnes <[email protected]>Cc Andrey Grodzovsky <[email protected]>Cc Dave Airlie <[email protected]>Cc Sean Paul <[email protected]>Cc Simon Ser <[email protected]>Cc Ben Skeggs <[email protected]>Cc Karol Herbst <[email protected]>---
This is only about laying out plans for the future, not about what drivers do today. We'd just like to be sure the goals are reasonable and everyone is aware of the idea.
Thanks,
pqv5:
- two grammar fixes (Alex)
- added R-b/A-b, dropped extra Ccv4:
- two typo fixes (Daniel)v3:
- update ENODEV doc (Daniel)
- clarify existing vs. new mmaps (Andrey)
- split into KMS and render/cross sections (Andrey, Daniel)
-open()returns ENXIO (open(2) man page)
- ioctls may return ENODEV (Andrey, Daniel)
- new wayland-protocols MRv2:
- mmap reads/writes undefined (Daniel)
- make render ioctl behaviour driver-specific (Daniel)
- restructure the mmap paragraphs (Daniel)
- chardev minor notes (Simon)
- open behaviour (Daniel)
- DRM leasing behaviour (Daniel)
- added linksDisclaimer: I am a userspace developer writing for other userspace developers. I took some liberties in defining what should happen without knowing what is actually possible or what existing drivers already implement.
--- Documentation/gpu/drm-uapi.rst | 114 ++++++++++++++++++++++++++++++++- 1 file changed, 113 insertions(+), 1 deletion(-)Documentation/gpu/drm-uapi.rst +27
-0@@ -1,3 +1,5 @@ +.. Copyright 2020 DisplayLink (UK) Ltd. + =================== Userland interfaces ===================@@ -162,6 +164,116 @@ other hand, a driver requires shared state between clients which is visible to user-space and accessible beyond open-file boundaries, they cannot support render nodes. +Device Hot-Unplug +================= + +.. note:: + The following is the plan. Implementation is not there yet + (2020 May). + +Graphics devices (display and/or render) may be connected via USB (e.g. +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end +user is able to hot-unplug this kind of devices while they are being +used, and expects that the very least the machine does not crash. Any +damage from hot-unplugging a DRM device needs to be limited as much as +possible and userspace must be given the chance to handle it if it wants +to. Ideally, unplugging a DRM device still lets a desktop continue to +run, but that is going to need explicit support throughout the whole +graphics stack: from kernel and userspace drivers, through display +servers, via window system protocols, and in applications and libraries. + +Other scenarios that should lead to the same are: unrecoverable GPU +crash, PCI device disappearing off the bus, or forced unbind of a driver +from the physical device. + +In other words, from userspace perspective everything needs to keep on +working more or less, until userspace stops using the disappeared DRM +device and closes it completely. Userspace will learn of the deviceDocumentation/gpu/drm-uapi.rst other hand, a driver requires shared state between clients which is · line 201
Quoted 10 lines
Documentation/gpu/drm-uapi.rst +10
-0+disappearance from the device removed uevent, ioctls returning ENODEV +(or driver-specific ioctls returning driver-specific things), or open() +returning ENXIO. + +Only after userspace has closed all relevant DRM device and dmabuf file +descriptors and removed all mmaps, the DRM driver can tear down its +instance for the device that no longer exists. If the same physical +device somehow comes back in the mean time, it shall be a new DRM +device. +
Quoted 7 lines
I don't think we can make a general statement like this. Drivers might have kworker polling on the device or other asynchronous or repeating jobs it might want to get rid of as soon as possible. I think it should be fine for the kernel driver to tear down the instance as long as it doesn't conflict with the goal of system stability. And usually drivers have some kernel side state for clients anyway, so it could be handled in this layer instead.
There's two clear bits:
- userspace stuff needs to stay around until the last userspace is gone, including indirect references (dma_fence, dma_buf, stuff like that)
- hw stuff must be released right away at unbind time (mmio maps, interrupts, that kinds of stuff)
Everything in-between is a bit a grey area. Some workers you might want to stop at unbind and release, for others it might be easier to not care. This is kinda all up to drivers and where exactly they put down the line between the userspace visible sw stuff and the hw related bits.
Since the above is talking about the uapi (this is an addition to drm-uapi.rst) I think it's totally fine to make the sweeping general statement, since the uapi stuff really should all stick around, or stuff starts blowing up at random.
-Daniel
Quoted 94 lines
Or maybe I am overlooking something and we really can't do that :/
Quoted 90 lines
Documentation/gpu/drm-uapi.rst +76
-2+Similar to PIDs, chardev minor numbers are not recycled immediately. A +new DRM device always picks the next free minor number compared to the +previous one allocated, and wraps around when minor numbers are +exhausted. + +The goal raises at least the following requirements for the kernel and +drivers. + +Requirements for KMS UAPI +------------------------- + +- KMS connectors must change their status to disconnected. + +- Legacy modesets and pageflips, and atomic commits, both real and + TEST_ONLY, and any other ioctls either fail with ENODEV or fake + success. + +- Pending non-blocking KMS operations deliver the DRM events userspace + is expecting. This applies also to ioctls that faked success. + +- open() on a device node whose underlying device has disappeared will + fail with ENXIO. + +- Attempting to create a DRM lease on a disappeared DRM device will + fail with ENODEV. Existing DRM leases remain and work as listed + above. + +Requirements for Render and Cross-Device UAPI +--------------------------------------------- + +- All GPU jobs that can no longer run must have their fences + force-signalled to avoid inflicting hangs on userspace. + The associated error code is ENODEV. + +- Some userspace APIs already define what should happen when the device + disappears (OpenGL, GL ES: `GL_KHR_robustness`_; `Vulkan`_: + VK_ERROR_DEVICE_LOST; etc.). DRM drivers are free to implement this + behaviour the way they see best, e.g. returning failures in + driver-specific ioctls and handling those in userspace drivers, or + rely on uevents, and so on. + +- dmabuf which point to memory that has disappeared will either fail to + import with ENODEV or continue to be successfully imported if it would + have succeeded before the disappearance. See also about memory maps + below for already imported dmabufs. + +- Attempting to import a dmabuf to a disappeared device will either fail + with ENODEV or succeed if it would have succeeded without the + disappearance. + +- open() on a device node whose underlying device has disappeared will + fail with ENXIO. + +.. _GL_KHR_robustness: https://www.khronos.org/registry/OpenGL/extensions/KHR/KHR_robustness.txt +.. _Vulkan: https://www.khronos.org/vulkan/ + +Requirements for Memory Maps +---------------------------- + +Memory maps have further requirements that apply to both existing maps +and maps created after the device has disappeared. If the underlying +memory disappears, the map is created or modified such that reads and +writes will still complete successfully but the result is undefined. +This applies to both userspace mmap()'d memory and memory pointed to by +dmabuf which might be mapped to other devices (cross-device dmabuf +imports). + +Raising SIGBUS is not an option, because userspace cannot realistically +handle it. Signal handlers are global, which makes them extremely +difficult to use correctly from libraries like those that Mesa produces. +Signal handlers are not composable, you can't have different handlers +for GPU1 and GPU2 from different vendors, and a third handler for +mmapped regular files. Threads cause additional pain with signal +handling as well. + .. _drm_driver_ioctl: IOCTL Support on Device Nodes@@ -199,7 +311,7 @@ EPERM/EACCES: difference between EACCES and EPERM. ENODEV: - The device is not (yet) present or fully initialized. + The device is not present anymore or is not yet fully initialized. EOPNOTSUPP: Feature (like PRIME, modesetting, GEM) is not supported by the driver. --2.20.1
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 11 lines · Pekka Paalanen
From: Pekka Paalanen [email protected]
Set up the expectations on how hot-unplugging a DRM device should look like to userspace.
Written by Daniel Vetter's request and largely based on his comments in IRC and from https://lists.freedesktop.org/archives/dri-devel/2020-May/265484.html .
A related Wayland protocol change proposal is at
https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/35
Quoted 10 lines
Signed-off-by Pekka Paalanen [email protected]Reviewed-by Daniel Vetter [email protected]Reviewed-by Alex Deucher [email protected]Acked-by Noralf Trønnes [email protected]Cc Andrey Grodzovsky [email protected]Cc Dave Airlie [email protected]Cc Sean Paul [email protected]Cc Simon Ser [email protected]Cc Ben Skeggs [email protected]Cc Karol Herbst [email protected]
From a user-space point-of-view, this sounds fine.
Thanks!
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
drm/doc: missing SPDX license identifier (Was: Re: [PATCH v5] drm/doc: device hot-unplug for userspace)
In reply to Pekka PaalanenHi Pekka,
Thanks for the patch - merged to drm-misc-next. While applying the patch the following warning popped-up.
-:37: WARNING:SPDX_LICENSE_TAG: Missing or malformed SPDX-License-Identifier tag in line 1
#37: FILE: Documentation/gpu/drm-uapi.rst:1:
+.. Copyright 2020 DisplayLink (UK) Ltd.
│--
Upon closer look, it seems that the DRM documentation lacks a license. An SPDX one at least - I haven't done extensive reading through each individual file.
I guess we should add one - be that GPL-2.0, MIT and/or otherwise. Since I haven't contributed such that much on the documentation side, I'll defer the suggestion to the maintainers.
Daniel, Dave, care to do the honours?
Tl;DR: the DRM documentation is missing SPDX license identifier.
Thanks
Emil
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel