Safety of opening up /dev/dma_heap/* to physically present users (udev uaccess tag) ?
Hi dma-buf maintainers, et.al.,
Various people have been working on making complex/MIPI cameras work OOTB with mainline Linux kernels and an opensource userspace stack.
The generic solution adds a software ISP (for Debayering and 3A) to libcamera. Libcamera's API guarantees that buffers handed to applications using it are dma-bufs so that these can be passed to e.g. a video encoder.
In order to meet this API guarantee the libcamera software ISP allocates dma-bufs from userspace through one of the /dev/dma_heap/* heaps. For the Fedora COPR repo for the PoC of this:
https://hansdegoede.dreamwidth.org/28153.html
I have added a simple udev rule to give physically present users access to the dma_heap-s:
KERNEL=="system", SUBSYSTEM=="dma_heap", TAG+="uaccess"
(and on Rasperry Pi devices any users in the video group get access)
This was just a quick fix for the PoC. Now that we are ready to move out of the PoC phase and start actually integrating this into distributions the question becomes if this is an acceptable solution; or if we need some other way to deal with this ?
Specifically the question is if this will have any negative security implications? I can certainly see this being used to do some sort of denial of service attack on the system (1). This is especially true for the cma heap which generally speaking is a limited resource.
But devices tagged for uaccess are only opened up to users who are physcially present behind the machine and those can just hit the powerbutton, so I don't believe that any *on purpose* DOS is part of the thread model. Any accidental DOS would be a userspace stack bug.
Do you foresee any other negative security implications from allowing physically present non root users to create (u)dma-bufs ?
Regards,
Hans
1) There are some limits in drivers/dma-buf/udmabuf.c and distributions could narrow these.
Hi,
Quoted 5 lines · Hans de Goede
Hi dma-buf maintainers, et.al.,
Various people have been working on making complex/MIPI cameras work OOTB with mainline Linux kernels and an opensource userspace stack.
Quoted 8 lines
The generic solution adds a software ISP (for Debayering and 3A) to libcamera. Libcamera's API guarantees that buffers handed to applications using it are dma-bufs so that these can be passed to e.g. a video encoder.
In order to meet this API guarantee the libcamera software ISP allocates dma-bufs from userspace through one of the
/dev/dma_heap/*heaps. For the Fedora COPR repo for the PoC of this:
https://hansdegoede.dreamwidth.org/28153.html
For the record, we're also considering using them for ARM KMS devices, so it would be better if the solution wasn't only considering v4l2 devices.
Quoted 12 lines
I have added a simple udev rule to give physically present users access to the dma_heap-s:
KERNEL=="system", SUBSYSTEM=="dma_heap", TAG+="uaccess"
(and on Rasperry Pi devices any users in the video group get access)
This was just a quick fix for the PoC. Now that we are ready to move out of the PoC phase and start actually integrating this into distributions the question becomes if this is an acceptable solution; or if we need some other way to deal with this ?
Quoted 4 lines
Specifically the question is if this will have any negative security implications? I can certainly see this being used to do some sort of denial of service attack on the system (1). This is especially true for the cma heap which generally speaking is a limited resource.
There's plenty of other ways to exhaust CMA, like allocating too much KMS or v4l2 buffers. I'm not sure we should consider dma-heaps differently than those if it's part of our threat model.
Quoted 4 lines
But devices tagged for uaccess are only opened up to users who are physcially present behind the machine and those can just hit the powerbutton, so I don't believe that any *on purpose* DOS is part of the thread model.
How would that work for headless devices?
Maxime
Hi Maxime,
Quoted 33 lines · Maxime Ripard
Hi,
Quoted 5 lines · Hans de Goede
Hi dma-buf maintainers, et.al.,
Various people have been working on making complex/MIPI cameras work OOTB with mainline Linux kernels and an opensource userspace stack.
Quoted 8 lines
The generic solution adds a software ISP (for Debayering and 3A) to libcamera. Libcamera's API guarantees that buffers handed to applications using it are dma-bufs so that these can be passed to e.g. a video encoder.
In order to meet this API guarantee the libcamera software ISP allocates dma-bufs from userspace through one of the
/dev/dma_heap/*heaps. For the Fedora COPR repo for the PoC of this:
https://hansdegoede.dreamwidth.org/28153.htmlFor the record, we're also considering using them for ARM KMS devices, so it would be better if the solution wasn't only considering v4l2 devices.
Quoted 12 lines
I have added a simple udev rule to give physically present users access to the dma_heap-s:
KERNEL=="system", SUBSYSTEM=="dma_heap", TAG+="uaccess"
(and on Rasperry Pi devices any users in the video group get access)
This was just a quick fix for the PoC. Now that we are ready to move out of the PoC phase and start actually integrating this into distributions the question becomes if this is an acceptable solution; or if we need some other way to deal with this ?
Quoted 8 lines
Quoted 4 lines
Specifically the question is if this will have any negative security implications? I can certainly see this being used to do some sort of denial of service attack on the system (1). This is especially true for the cma heap which generally speaking is a limited resource.
There's plenty of other ways to exhaust CMA, like allocating too much KMS or v4l2 buffers. I'm not sure we should consider dma-heaps differently than those if it's part of our threat model.
Ack.
Quoted 6 lines
Quoted 4 lines
But devices tagged for uaccess are only opened up to users who are physcially present behind the machine and those can just hit the powerbutton, so I don't believe that any *on purpose* DOS is part of the thread model.
How would that work for headless devices?
The uaccess tag solution does not work for headless devices, but it also should not hurt any headless scenarios.
Headless devices could use something like the video group solution (dma_heap group?) which Raspberry Pi is using and them make sure that any services which need access run as a user in that group.
This can co-exist with uaccess tags since those use ACLs not classic Unix permissions.
Regards,
Hans
Quoted 33 lines · Maxime Ripard
Hi,
Quoted 5 lines · Hans de Goede
Hi dma-buf maintainers, et.al.,
Various people have been working on making complex/MIPI cameras work OOTB with mainline Linux kernels and an opensource userspace stack.
Quoted 8 lines
The generic solution adds a software ISP (for Debayering and 3A) to libcamera. Libcamera's API guarantees that buffers handed to applications using it are dma-bufs so that these can be passed to e.g. a video encoder.
In order to meet this API guarantee the libcamera software ISP allocates dma-bufs from userspace through one of the
/dev/dma_heap/*heaps. For the Fedora COPR repo for the PoC of this:
https://hansdegoede.dreamwidth.org/28153.htmlFor the record, we're also considering using them for ARM KMS devices, so it would be better if the solution wasn't only considering v4l2 devices.
Quoted 12 lines
I have added a simple udev rule to give physically present users access to the dma_heap-s:
KERNEL=="system", SUBSYSTEM=="dma_heap", TAG+="uaccess"
(and on Rasperry Pi devices any users in the video group get access)
This was just a quick fix for the PoC. Now that we are ready to move out of the PoC phase and start actually integrating this into distributions the question becomes if this is an acceptable solution; or if we need some other way to deal with this ?
Quoted 8 lines
Quoted 4 lines
Specifically the question is if this will have any negative security implications? I can certainly see this being used to do some sort of denial of service attack on the system (1). This is especially true for the cma heap which generally speaking is a limited resource.
There's plenty of other ways to exhaust CMA, like allocating too much KMS or v4l2 buffers. I'm not sure we should consider dma-heaps differently than those if it's part of our threat model.
So generally for an arm soc where your display needs cma, your render node doesn't. And user applications only have access to the later, while only the compositor gets a kms fd through logind. At least in drm aside from vc4 there's really no render driver that just gives you access to cma and allows you to exhaust that, you need to be a compositor with drm master access to the display.
Which means we're mostly protected against bad applications, and that's not a threat the "user physically sits in front of the machine accounts for", and which giving cma access to everyone would open up. And with flathub/snaps/... this is very much an issue.
So you need more, either:
- cgroups limits on dma-buf and dma-buf heaps. This has been bikeshedded for years and is just not really moving.
- An allocator service which checks whether you're allowed to allocate these special buffers. Android does that through binder.
Probably also some way to nuke applications that refuse to release buffers when they're no longer the right application. cgroups is a lot more convenient for that.
-Sima
Quoted 8 lines
Quoted 4 lines
But devices tagged for uaccess are only opened up to users who are physcially present behind the machine and those can just hit the powerbutton, so I don't believe that any *on purpose* DOS is part of the thread model.
How would that work for headless devices?
Maxime
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Hi Sima,
Quoted 50 lines · Daniel Vetter
Quoted 33 lines · Maxime Ripard
Hi,
Quoted 5 lines · Hans de Goede
Hi dma-buf maintainers, et.al.,
Various people have been working on making complex/MIPI cameras work OOTB with mainline Linux kernels and an opensource userspace stack.
Quoted 8 lines
The generic solution adds a software ISP (for Debayering and 3A) to libcamera. Libcamera's API guarantees that buffers handed to applications using it are dma-bufs so that these can be passed to e.g. a video encoder.
In order to meet this API guarantee the libcamera software ISP allocates dma-bufs from userspace through one of the
/dev/dma_heap/*heaps. For the Fedora COPR repo for the PoC of this:
https://hansdegoede.dreamwidth.org/28153.htmlFor the record, we're also considering using them for ARM KMS devices, so it would be better if the solution wasn't only considering v4l2 devices.
Quoted 12 lines
I have added a simple udev rule to give physically present users access to the dma_heap-s:
KERNEL=="system", SUBSYSTEM=="dma_heap", TAG+="uaccess"
(and on Rasperry Pi devices any users in the video group get access)
This was just a quick fix for the PoC. Now that we are ready to move out of the PoC phase and start actually integrating this into distributions the question becomes if this is an acceptable solution; or if we need some other way to deal with this ?
Quoted 8 lines
Quoted 4 lines
Specifically the question is if this will have any negative security implications? I can certainly see this being used to do some sort of denial of service attack on the system (1). This is especially true for the cma heap which generally speaking is a limited resource.
There's plenty of other ways to exhaust CMA, like allocating too much KMS or v4l2 buffers. I'm not sure we should consider dma-heaps differently than those if it's part of our threat model.
So generally for an arm soc where your display needs cma, your render node doesn't. And user applications only have access to the later, while only the compositor gets a kms fd through logind. At least in drm aside from vc4 there's really no render driver that just gives you access to cma and allows you to exhaust that, you need to be a compositor with drm master access to the display.
Quoted 4 lines
Which means we're mostly protected against bad applications, and that's not a threat the "user physically sits in front of the machine accounts for", and which giving cma access to everyone would open up. And with flathub/snaps/... this is very much an issue.
I agree that bad applications are an issue, but not for the flathub / snaps case. Flatpacks / snaps run sandboxed and don't have access to a full /dev so those should not be able to open /dev/dma_heap/* independent of the ACLs on /dev/dma_heap/*. The plan is for cameras using the libcamera software ISP to always be accessed through pipewire and the camera portal, so in this case pipewere is taking the place of the compositor in your kms vs render node example.
So this reduces the problem to bad apps packaged by regular distributions and if any of those misbehave the distros should fix that.
So I think that for the denial of service side allowing physical present users (but not sandboxed apps running as those users) to access /dev/dma_heap/* should be ok.
My bigger worry is if dma_heap (u)dma-bufs can be abused in other ways then causing a denial of service.
I guess that the answer there is that causing other security issues should not be possible ?
Regards,
Hans
Quoted 58 lines · Hans de Goede
Hi Sima,
Quoted 50 lines · Daniel Vetter
Quoted 33 lines · Maxime Ripard
Hi,
Quoted 5 lines · Hans de Goede
Hi dma-buf maintainers, et.al.,
Various people have been working on making complex/MIPI cameras work OOTB with mainline Linux kernels and an opensource userspace stack.
Quoted 8 lines
The generic solution adds a software ISP (for Debayering and 3A) to libcamera. Libcamera's API guarantees that buffers handed to applications using it are dma-bufs so that these can be passed to e.g. a video encoder.
In order to meet this API guarantee the libcamera software ISP allocates dma-bufs from userspace through one of the
/dev/dma_heap/*heaps. For the Fedora COPR repo for the PoC of this:
https://hansdegoede.dreamwidth.org/28153.htmlFor the record, we're also considering using them for ARM KMS devices, so it would be better if the solution wasn't only considering v4l2 devices.
Quoted 12 lines
I have added a simple udev rule to give physically present users access to the dma_heap-s:
KERNEL=="system", SUBSYSTEM=="dma_heap", TAG+="uaccess"
(and on Rasperry Pi devices any users in the video group get access)
This was just a quick fix for the PoC. Now that we are ready to move out of the PoC phase and start actually integrating this into distributions the question becomes if this is an acceptable solution; or if we need some other way to deal with this ?
Quoted 8 lines
Quoted 4 lines
Specifically the question is if this will have any negative security implications? I can certainly see this being used to do some sort of denial of service attack on the system (1). This is especially true for the cma heap which generally speaking is a limited resource.
There's plenty of other ways to exhaust CMA, like allocating too much KMS or v4l2 buffers. I'm not sure we should consider dma-heaps differently than those if it's part of our threat model.
So generally for an arm soc where your display needs cma, your render node doesn't. And user applications only have access to the later, while only the compositor gets a kms fd through logind. At least in drm aside from vc4 there's really no render driver that just gives you access to cma and allows you to exhaust that, you need to be a compositor with drm master access to the display.
Quoted 4 lines
Which means we're mostly protected against bad applications, and that's not a threat the "user physically sits in front of the machine accounts for", and which giving cma access to everyone would open up. And with flathub/snaps/... this is very much an issue.
Quoted 7 lines
I agree that bad applications are an issue, but not for the flathub / snaps case. Flatpacks / snaps run sandboxed and don't have access to a full /dev so those should not be able to open
/dev/dma_heap/*independent of the ACLs on/dev/dma_heap/*. The plan is for cameras using the libcamera software ISP to always be accessed through pipewire and the camera portal, so in this case pipewere is taking the place of the compositor in your kms vs render node example.
Yeah essentially if you clarify to "set the permissions such that pipewire can do allocations", then I think that makes sense. And is at the same level as e.g. drm kms giving compsitors (but _only_ compositors) special access rights.
Quoted 7 lines
So this reduces the problem to bad apps packaged by regular distributions and if any of those misbehave the distros should fix that.
So I think that for the denial of service side allowing physical present users (but not sandboxed apps running as those users) to access
/dev/dma_heap/*should be ok.
Quoted 5 lines
My bigger worry is if
dma_heap(u)dma-bufs can be abused in other ways then causing a denial of service.I guess that the answer there is that causing other security issues should not be possible ?
Well pinned memory exhaustion is a very useful tool to make all kinds of other kernel issues exploitable. Like if you have that you can weaponize all kinds of kmalloc error paths (and since it's untracked memory the oom killer will not get you of these issuees).
I think for the pipewire based desktop it'd be best if you only allow pipewire to get at an fd for allocating from dma-heaps, kinda like logind furnishes the kms master fd ... Still has the issue that you can't nuke these buffers, but that's for another day. But at least from a "limit attack surface" design pov I think this would be better than just handing out access to the current user outright. But that's also not the worst option I guess, as long as snaps/flatpacks only go through the pipewire service.
-Sima
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Hi,
Quoted 10 lines · Daniel Vetter
Quoted 8 lines · Hans de Goede
On 5/6/24 3:38 PM, Daniel Vetter wrote:
I agree that bad applications are an issue, but not for the flathub / snaps case. Flatpacks / snaps run sandboxed and don't have access to a full /dev so those should not be able to open/dev/dma_heap/*independent of the ACLs on/dev/dma_heap/*. The plan is for cameras using the libcamera software ISP to always be accessed through pipewire and the camera portal, so in this case pipewere is taking the place of the compositor in your kms vs render node example.
Quoted 4 lines
Yeah essentially if you clarify to "set the permissions such that pipewire can do allocations", then I think that makes sense. And is at the same level as e.g. drm kms giving compsitors (but _only_ compositors) special access rights.
That would have the unfortunate side effect of making sandboxed apps less efficient on some platforms, since they wouldn't be able to do direct scanout anymore ...
Cheers,
Daniel
Quoted 13 lines · Daniel Stone
Hi,
Quoted 10 lines · Daniel Vetter
Quoted 8 lines · Hans de Goede
On 5/6/24 3:38 PM, Daniel Vetter wrote:
I agree that bad applications are an issue, but not for the flathub / snaps case. Flatpacks / snaps run sandboxed and don't have access to a full /dev so those should not be able to open/dev/dma_heap/*independent of the ACLs on/dev/dma_heap/*. The plan is for cameras using the libcamera software ISP to always be accessed through pipewire and the camera portal, so in this case pipewere is taking the place of the compositor in your kms vs render node example.
Quoted 8 lines
Quoted 4 lines
Yeah essentially if you clarify to "set the permissions such that pipewire can do allocations", then I think that makes sense. And is at the same level as e.g. drm kms giving compsitors (but _only_ compositors) special access rights.
That would have the unfortunate side effect of making sandboxed apps less efficient on some platforms, since they wouldn't be able to do direct scanout anymore ...
I was assuming that everyone goes through pipewire, and ideally that is the only one that can even get at these special chardev.
If pipewire is only for sandboxed apps then yeah this aint great :-/ -Sima
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Quoted 9 lines · Daniel Vetter
Quoted 3 lines · Daniel Stone
That would have the unfortunate side effect of making sandboxed apps less efficient on some platforms, since they wouldn't be able to do direct scanout anymore ...
I was assuming that everyone goes through pipewire, and ideally that is the only one that can even get at these special chardev.
If pipewire is only for sandboxed apps then yeah this aint great :-/
No, PipeWire is fine, I mean graphical apps.
Right now, if your platform requires CMA for display, then the app needs access to the GPU render node and the display node too, in order to allocate buffers which the compositor can scan out directly. If it only has access to the render nodes and not the display node, it won't be able to allocate correctly, so its content will need a composition pass, i.e. performance penalty for sandboxing. But if it can allocate correctly, then hey, it can exhaust CMA just like heaps can.
Personally I think we'd be better off just allowing access and figuring out cgroups later. It's not like the OOM story is great generally, and hey, you can get there with just render nodes ...
Cheers,
Daniel
Quoted 21 lines · Daniel Stone
Quoted 9 lines · Daniel Vetter
Quoted 3 lines · Daniel Stone
That would have the unfortunate side effect of making sandboxed apps less efficient on some platforms, since they wouldn't be able to do direct scanout anymore ...
I was assuming that everyone goes through pipewire, and ideally that is the only one that can even get at these special chardev.
If pipewire is only for sandboxed apps then yeah this aint great :-/
No, PipeWire is fine, I mean graphical apps.
Right now, if your platform requires CMA for display, then the app needs access to the GPU render node and the display node too, in order to allocate buffers which the compositor can scan out directly. If it only has access to the render nodes and not the display node, it won't be able to allocate correctly, so its content will need a composition pass, i.e. performance penalty for sandboxing. But if it can allocate correctly, then hey, it can exhaust CMA just like heaps can.
Quoted 3 lines
Personally I think we'd be better off just allowing access and figuring out cgroups later. It's not like the OOM story is great generally, and hey, you can get there with just render nodes ...
Imo the right fix is to ask the compositor to allocate the buffers in this case, and then maybe have some kind of revoke/purge behaviour on these buffers. Compositor has an actual idea of who's a candidate for direct scanout after all, not the app. Or well at least force migrate the memory from cma to shmem.
If you only whack cgroups on this issue you're still stuck in the world where either all apps together can ddos the display or no one can realistically direct scanout.
So yeah on the display side the problem isn't solved either, but we knew that already.
-Sima
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Hi,
Quoted 19 lines · Daniel Vetter
Quoted 11 lines · Daniel Stone
Right now, if your platform requires CMA for display, then the app needs access to the GPU render node and the display node too, in order to allocate buffers which the compositor can scan out directly. If it only has access to the render nodes and not the display node, it won't be able to allocate correctly, so its content will need a composition pass, i.e. performance penalty for sandboxing. But if it can allocate correctly, then hey, it can exhaust CMA just like heaps can.
Personally I think we'd be better off just allowing access and figuring out cgroups later. It's not like the OOM story is great generally, and hey, you can get there with just render nodes ...
Imo the right fix is to ask the compositor to allocate the buffers in this case, and then maybe have some kind of revoke/purge behaviour on these buffers. Compositor has an actual idea of who's a candidate for direct scanout after all, not the app. Or well at least force migrate the memory from cma to shmem.
Quoted 3 lines
If you only whack cgroups on this issue you're still stuck in the world where either all apps together can ddos the display or no one can realistically direct scanout.
Mmm, back to DRI2. I can't say I'm wildly enthused about that, not least because a client using GPU/codec/etc for those buffers would have to communicate its requirements (alignment etc) forward to the compositor in order for the compositor to allocate for it. Obviously passing the constraints etc around isn't a solved problem yet, but it is at least contained down in clients rather than making it back and forth between client and compositor.
I'm extremely not-wild about the compositor migrating memory from CMA to shmem behind the client's back, and tbh I'm not sure how that would even work if the client has it pinned through whatever API it's imported into.
Anyway, like Laurent says, if we're deciding that heaps can't be used by generic apps (unlike DRM/V4L2/etc), then we need gralloc.
Cheers,
Daniel
Quoted 26 lines · Daniel Stone
Hi,
Quoted 19 lines · Daniel Vetter
Quoted 11 lines · Daniel Stone
Right now, if your platform requires CMA for display, then the app needs access to the GPU render node and the display node too, in order to allocate buffers which the compositor can scan out directly. If it only has access to the render nodes and not the display node, it won't be able to allocate correctly, so its content will need a composition pass, i.e. performance penalty for sandboxing. But if it can allocate correctly, then hey, it can exhaust CMA just like heaps can.
Personally I think we'd be better off just allowing access and figuring out cgroups later. It's not like the OOM story is great generally, and hey, you can get there with just render nodes ...
Imo the right fix is to ask the compositor to allocate the buffers in this case, and then maybe have some kind of revoke/purge behaviour on these buffers. Compositor has an actual idea of who's a candidate for direct scanout after all, not the app. Or well at least force migrate the memory from cma to shmem.
Quoted 3 lines
If you only whack cgroups on this issue you're still stuck in the world where either all apps together can ddos the display or no one can realistically direct scanout.
Quoted 7 lines
Mmm, back to DRI2. I can't say I'm wildly enthused about that, not least because a client using GPU/codec/etc for those buffers would have to communicate its requirements (alignment etc) forward to the compositor in order for the compositor to allocate for it. Obviously passing the constraints etc around isn't a solved problem yet, but it is at least contained down in clients rather than making it back and forth between client and compositor.
I don't think you need the compositor to allocate the buffer from the requirements, you only need a protocol that a) allocates a buffer of a given size from a given heap and b) has some kinda of revoke provisions so that the compositor can claw back the memory again when it needs it.
Quoted 4 lines
I'm extremely not-wild about the compositor migrating memory from CMA to shmem behind the client's back, and tbh I'm not sure how that would even work if the client has it pinned through whatever API it's imported into.
Other option is revoke on cma buffers that are allocated by clients, for the case the compositor needs it.
Quoted 2 lines
Anyway, like Laurent says, if we're deciding that heaps can't be used by generic apps (unlike DRM/V4L2/etc), then we need gralloc.
gralloc doesn't really fix this, it's just abstraction around how/where you allocate?
Anyway the current plan is that we all pretend this issue of CMA allocated buffers don't exist and we let clients allocate without limits. Given that we don't even have cgroups to sort out the mess for anything else I wouldn't worry too much ...
-Sima
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Quoted 35 lines · Daniel Vetter
Quoted 23 lines · Daniel Stone
Quoted 10 lines · On Wed, 8 May 2024 at 09:33, Daniel Vetter [email protected] wrote:
Quoted 3 lines · Daniel Stone
That would have the unfortunate side effect of making sandboxed apps less efficient on some platforms, since they wouldn't be able to do direct scanout anymore ...
I was assuming that everyone goes through pipewire, and ideally that is the only one that can even get at these special chardev.
If pipewire is only for sandboxed apps then yeah this aint great :-/
No, PipeWire is fine, I mean graphical apps.
Right now, if your platform requires CMA for display, then the app needs access to the GPU render node and the display node too, in order to allocate buffers which the compositor can scan out directly. If it only has access to the render nodes and not the display node, it won't be able to allocate correctly, so its content will need a composition pass, i.e. performance penalty for sandboxing. But if it can allocate correctly, then hey, it can exhaust CMA just like heaps can.
Quoted 3 lines
Personally I think we'd be better off just allowing access and figuring out cgroups later. It's not like the OOM story is great generally, and hey, you can get there with just render nodes ...
Imo the right fix is to ask the compositor to allocate the buffers in this case, and then maybe have some kind of revoke/purge behaviour on these buffers. Compositor has an actual idea of who's a candidate for direct scanout after all, not the app. Or well at least force migrate the memory from cma to shmem.
Quoted 6 lines
If you only whack cgroups on this issue you're still stuck in the world where either all apps together can ddos the display or no one can realistically direct scanout.
So yeah on the display side the problem isn't solved either, but we knew that already.
What makes scanout memory so special?
The way I see it, any kind of memory will always be a limited resource:
regular programs can exhaust system memory, as well as GPU VRAM, as well as scanout memory. I think we need to have ways to limit/control/arbiter the allocations regardless, and I don't think scanout memory should be a special case here.
Quoted 44 lines · Simon Ser
Quoted 35 lines · Daniel Vetter
Quoted 23 lines · Daniel Stone
Quoted 10 lines · On Wed, 8 May 2024 at 09:33, Daniel Vetter [email protected] wrote:
Quoted 3 lines · Daniel Stone
That would have the unfortunate side effect of making sandboxed apps less efficient on some platforms, since they wouldn't be able to do direct scanout anymore ...
I was assuming that everyone goes through pipewire, and ideally that is the only one that can even get at these special chardev.
If pipewire is only for sandboxed apps then yeah this aint great :-/
No, PipeWire is fine, I mean graphical apps.
Right now, if your platform requires CMA for display, then the app needs access to the GPU render node and the display node too, in order to allocate buffers which the compositor can scan out directly. If it only has access to the render nodes and not the display node, it won't be able to allocate correctly, so its content will need a composition pass, i.e. performance penalty for sandboxing. But if it can allocate correctly, then hey, it can exhaust CMA just like heaps can.
Quoted 3 lines
Personally I think we'd be better off just allowing access and figuring out cgroups later. It's not like the OOM story is great generally, and hey, you can get there with just render nodes ...
Imo the right fix is to ask the compositor to allocate the buffers in this case, and then maybe have some kind of revoke/purge behaviour on these buffers. Compositor has an actual idea of who's a candidate for direct scanout after all, not the app. Or well at least force migrate the memory from cma to shmem.
Quoted 6 lines
If you only whack cgroups on this issue you're still stuck in the world where either all apps together can ddos the display or no one can realistically direct scanout.
So yeah on the display side the problem isn't solved either, but we knew that already.
Quoted 7 lines
What makes scanout memory so special?
The way I see it, any kind of memory will always be a limited resource:
regular programs can exhaust system memory, as well as GPU VRAM, as well as scanout memory. I think we need to have ways to limit/control/arbiter the allocations regardless, and I don't think scanout memory should be a special case here.
(Long w/en and I caught a cold)
It's not scanout that's special, it's cma memory that's special. Because once you've allocated it, it's gone since it cannot be swapped out, and there's not a lot of it to go around. Which means even if we'd have cgroups for all the various gpu allocation heaps, you can't use cgroups to manage cma in a meaningful way:
- You set the cgroup limits so low for apps that it's guaranteed that the compositor will always be able to allocate enough scanout memory for it's need. That will be low enough that apps can never allocate scanout buffers themselves.
- Or you set the limit high enough so that apps can allocate enough, which means (as soon as you have more than just one app and not a totally bonkers amount of cma) that the compositor might not be able to allocate anymore.
It's kinda shit situation, which is also why you need the compositor to be able to revoke cma allocations it has handed to clients (like with drm leases).
Or we just keep the current yolo situation.
For any other memory type than CMA most of the popular drivers at least implement swapping, which gives you a ton more flexibility in setting up limits in a way that actually work. But even there we'd need cgroups first to make sure things don't go wrong too badly in the face of evil apps ... -Sima
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Am 16.05.24 um 12:13 schrieb Daniel Vetter:
Quoted 1 line
(Long w/en and I caught a cold)
Handing over a coup of tea.
I'm fighting with a cold since last week and I think it's one of the worst I've ever had.
(On the other hand every cold feels like the worst you ever had).
Christian.
Quoted 66 lines · Hans de Goede
Hi Sima,
Quoted 50 lines · Daniel Vetter
Quoted 33 lines · Maxime Ripard
Hi,
Quoted 5 lines · Hans de Goede
Hi dma-buf maintainers, et.al.,
Various people have been working on making complex/MIPI cameras work OOTB with mainline Linux kernels and an opensource userspace stack.
Quoted 8 lines
The generic solution adds a software ISP (for Debayering and 3A) to libcamera. Libcamera's API guarantees that buffers handed to applications using it are dma-bufs so that these can be passed to e.g. a video encoder.
In order to meet this API guarantee the libcamera software ISP allocates dma-bufs from userspace through one of the
/dev/dma_heap/*heaps. For the Fedora COPR repo for the PoC of this:
https://hansdegoede.dreamwidth.org/28153.htmlFor the record, we're also considering using them for ARM KMS devices, so it would be better if the solution wasn't only considering v4l2 devices.
Quoted 12 lines
I have added a simple udev rule to give physically present users access to the dma_heap-s:
KERNEL=="system", SUBSYSTEM=="dma_heap", TAG+="uaccess"
(and on Rasperry Pi devices any users in the video group get access)
This was just a quick fix for the PoC. Now that we are ready to move out of the PoC phase and start actually integrating this into distributions the question becomes if this is an acceptable solution; or if we need some other way to deal with this ?
Quoted 8 lines
Quoted 4 lines
Specifically the question is if this will have any negative security implications? I can certainly see this being used to do some sort of denial of service attack on the system (1). This is especially true for the cma heap which generally speaking is a limited resource.
There's plenty of other ways to exhaust CMA, like allocating too much KMS or v4l2 buffers. I'm not sure we should consider dma-heaps differently than those if it's part of our threat model.
So generally for an arm soc where your display needs cma, your render node doesn't. And user applications only have access to the later, while only the compositor gets a kms fd through logind. At least in drm aside from vc4 there's really no render driver that just gives you access to cma and allows you to exhaust that, you need to be a compositor with drm master access to the display.
Quoted 4 lines
Which means we're mostly protected against bad applications, and that's not a threat the "user physically sits in front of the machine accounts for", and which giving cma access to everyone would open up. And with flathub/snaps/... this is very much an issue.
I agree that bad applications are an issue, but not for the flathub / snaps case. Flatpacks / snaps run sandboxed and don't have access to a full /dev so those should not be able to open
/dev/dma_heap/*independent of the ACLs on/dev/dma_heap/*. The plan is for cameras using the libcamera software ISP to always be accessed through pipewire and the camera portal, so in this case pipewere is taking the place of the compositor in your kms vs render node example.
Quoted 6 lines
So this reduces the problem to bad apps packaged by regular distributions and if any of those misbehave the distros should fix that.
So I think that for the denial of service side allowing physical present users (but not sandboxed apps running as those users) to access
/dev/dma_heap/*should be ok.
What about an app built by the user? The machines can still be multi-seat or have multiple users.
Quoted 10 lines
My bigger worry is if
dma_heap(u)dma-bufs can be abused in other ways then causing a denial of service.I guess that the answer there is that causing other security issues should not be possible ?
Regards,
Hans
Signature · 2 lines
With best wishes Dmitry
Quoted 53 lines · Daniel Vetter
Quoted 31 lines · Maxime Ripard
Quoted 5 lines · Hans de Goede
Hi dma-buf maintainers, et.al.,
Various people have been working on making complex/MIPI cameras work OOTB with mainline Linux kernels and an opensource userspace stack.
Quoted 8 lines
The generic solution adds a software ISP (for Debayering and 3A) to libcamera. Libcamera's API guarantees that buffers handed to applications using it are dma-bufs so that these can be passed to e.g. a video encoder.
In order to meet this API guarantee the libcamera software ISP allocates dma-bufs from userspace through one of the
/dev/dma_heap/*heaps. For the Fedora COPR repo for the PoC of this:
https://hansdegoede.dreamwidth.org/28153.htmlFor the record, we're also considering using them for ARM KMS devices, so it would be better if the solution wasn't only considering v4l2 devices.
Quoted 12 lines
I have added a simple udev rule to give physically present users access to the dma_heap-s:
KERNEL=="system", SUBSYSTEM=="dma_heap", TAG+="uaccess"
(and on Rasperry Pi devices any users in the video group get access)
This was just a quick fix for the PoC. Now that we are ready to move out of the PoC phase and start actually integrating this into distributions the question becomes if this is an acceptable solution; or if we need some other way to deal with this ?
Quoted 8 lines
Quoted 4 lines
Specifically the question is if this will have any negative security implications? I can certainly see this being used to do some sort of denial of service attack on the system (1). This is especially true for the cma heap which generally speaking is a limited resource.
There's plenty of other ways to exhaust CMA, like allocating too much KMS or v4l2 buffers. I'm not sure we should consider dma-heaps differently than those if it's part of our threat model.
So generally for an arm soc where your display needs cma, your render node doesn't. And user applications only have access to the later, while only the compositor gets a kms fd through logind. At least in drm aside from vc4 there's really no render driver that just gives you access to cma and allows you to exhaust that, you need to be a compositor with drm master access to the display.
Which means we're mostly protected against bad applications, and that's not a threat the "user physically sits in front of the machine accounts for", and which giving cma access to everyone would open up. And with flathub/snaps/... this is very much an issue.
Quoted 7 lines
So you need more, either:
- cgroups limits on dma-buf and dma-buf heaps. This has been bikeshedded for years and is just not really moving.
- An allocator service which checks whether you're allowed to allocate these special buffers. Android does that through binder.
I would split the latter into a centralized frame buffer allocator library for userspace (James Jones' Unix device memory allocator comes to mind), providing dma-buf instances using whatever backend is available and suitable (DMA heaps would likely be one of them), and a safe way to make this allocator available to applications. Exposing it through some system services could be useful, but with proper tracking and accounting of memory allocated through DMA heaps (and DRM, and V4L2) we could possibly do with just a library.
What I really want is to move memory allocation for devices to a separate component, and make everything else a consumer of those buffers.
Quoted 10 lines
Probably also some way to nuke applications that refuse to release buffers when they're no longer the right application. cgroups is a lot more convenient for that.
Quoted 6 lines
Quoted 4 lines
But devices tagged for uaccess are only opened up to users who are physcially present behind the machine and those can just hit the powerbutton, so I don't believe that any *on purpose* DOS is part of the thread model.
How would that work for headless devices?
Signature · 3 lines
Regards, Laurent Pinchart
Hi,
Quoted 55 lines · Daniel Vetter
Quoted 33 lines · Maxime Ripard
Hi,
Quoted 5 lines · Hans de Goede
Hi dma-buf maintainers, et.al.,
Various people have been working on making complex/MIPI cameras work OOTB with mainline Linux kernels and an opensource userspace stack.
Quoted 8 lines
The generic solution adds a software ISP (for Debayering and 3A) to libcamera. Libcamera's API guarantees that buffers handed to applications using it are dma-bufs so that these can be passed to e.g. a video encoder.
In order to meet this API guarantee the libcamera software ISP allocates dma-bufs from userspace through one of the
/dev/dma_heap/*heaps. For the Fedora COPR repo for the PoC of this:
https://hansdegoede.dreamwidth.org/28153.htmlFor the record, we're also considering using them for ARM KMS devices, so it would be better if the solution wasn't only considering v4l2 devices.
Quoted 12 lines
I have added a simple udev rule to give physically present users access to the dma_heap-s:
KERNEL=="system", SUBSYSTEM=="dma_heap", TAG+="uaccess"
(and on Rasperry Pi devices any users in the video group get access)
This was just a quick fix for the PoC. Now that we are ready to move out of the PoC phase and start actually integrating this into distributions the question becomes if this is an acceptable solution; or if we need some other way to deal with this ?
Quoted 8 lines
Quoted 4 lines
Specifically the question is if this will have any negative security implications? I can certainly see this being used to do some sort of denial of service attack on the system (1). This is especially true for the cma heap which generally speaking is a limited resource.
There's plenty of other ways to exhaust CMA, like allocating too much KMS or v4l2 buffers. I'm not sure we should consider dma-heaps differently than those if it's part of our threat model.
So generally for an arm soc where your display needs cma, your render node doesn't. And user applications only have access to the later, while only the compositor gets a kms fd through logind. At least in drm aside from vc4 there's really no render driver that just gives you access to cma and allows you to exhaust that, you need to be a compositor with drm master access to the display.
Which means we're mostly protected against bad applications, and that's not a threat the "user physically sits in front of the machine accounts for", and which giving cma access to everyone would open up. And with flathub/snaps/... this is very much an issue.
Quoted 4 lines
So you need more, either:
- cgroups limits on dma-buf and dma-buf heaps. This has been bikeshedded for years and is just not really moving.
For reference, are you talking about:
https://lore.kernel.org/r/[email protected]
Or has there been a new version of that recently?
Maxime
Quoted 60 lines · Maxime Ripard
Hi,
Quoted 57 lines · Daniel Vetter
Quoted 33 lines · Maxime Ripard
Hi,
Quoted 5 lines · Hans de Goede
Hi dma-buf maintainers, et.al.,
Various people have been working on making complex/MIPI cameras work OOTB with mainline Linux kernels and an opensource userspace stack.
Quoted 8 lines
The generic solution adds a software ISP (for Debayering and 3A) to libcamera. Libcamera's API guarantees that buffers handed to applications using it are dma-bufs so that these can be passed to e.g. a video encoder.
In order to meet this API guarantee the libcamera software ISP allocates dma-bufs from userspace through one of the
/dev/dma_heap/*heaps. For the Fedora COPR repo for the PoC of this:
https://hansdegoede.dreamwidth.org/28153.htmlFor the record, we're also considering using them for ARM KMS devices, so it would be better if the solution wasn't only considering v4l2 devices.
Quoted 12 lines
I have added a simple udev rule to give physically present users access to the dma_heap-s:
KERNEL=="system", SUBSYSTEM=="dma_heap", TAG+="uaccess"
(and on Rasperry Pi devices any users in the video group get access)
This was just a quick fix for the PoC. Now that we are ready to move out of the PoC phase and start actually integrating this into distributions the question becomes if this is an acceptable solution; or if we need some other way to deal with this ?
Quoted 8 lines
Quoted 4 lines
Specifically the question is if this will have any negative security implications? I can certainly see this being used to do some sort of denial of service attack on the system (1). This is especially true for the cma heap which generally speaking is a limited resource.
There's plenty of other ways to exhaust CMA, like allocating too much KMS or v4l2 buffers. I'm not sure we should consider dma-heaps differently than those if it's part of our threat model.
So generally for an arm soc where your display needs cma, your render node doesn't. And user applications only have access to the later, while only the compositor gets a kms fd through logind. At least in drm aside from vc4 there's really no render driver that just gives you access to cma and allows you to exhaust that, you need to be a compositor with drm master access to the display.
Which means we're mostly protected against bad applications, and that's not a threat the "user physically sits in front of the machine accounts for", and which giving cma access to everyone would open up. And with flathub/snaps/... this is very much an issue.
So you need more, either:
Quoted 8 lines
Quoted 2 lines
- cgroups limits on dma-buf and dma-buf heaps. This has been bikeshedded for years and is just not really moving.
For reference, are you talking about:
https://lore.kernel.org/r/[email protected]
Or has there been a new version of that recently?
I think the design feedback from Tejun has changed to that system memory should be tracked with memcg instead (but that kinda leaves the open of what to do with cma), and only device memory be tracked with a separate cgroups controller.
But I'm also not sure whether that would actually solve all the tracking/isolation requirements people tossed around or just gives us something that wont get the job done.
Either way, yes I think that was the most recent code. -Sima
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Quoted 5 lines · Hans de Goede
Hi dma-buf maintainers, et.al.,
Various people have been working on making complex/MIPI cameras work OOTB with mainline Linux kernels and an opensource userspace stack.
Quoted 8 lines
The generic solution adds a software ISP (for Debayering and 3A) to libcamera. Libcamera's API guarantees that buffers handed to applications using it are dma-bufs so that these can be passed to e.g. a video encoder.
In order to meet this API guarantee the libcamera software ISP allocates dma-bufs from userspace through one of the
/dev/dma_heap/*heaps. For the Fedora COPR repo for the PoC of this:
https://hansdegoede.dreamwidth.org/28153.html
Is there any reason for allocating DMA buffers for libcamera through /dev/dma_heap/ rather than allocating them via corresponding media device and then giving them away to DRM / VPU / etc?
At least this should solve the permission usecase: if the app can open camera device, it can allocate a buffer.
Quoted 34 lines
I have added a simple udev rule to give physically present users access to the dma_heap-s:
KERNEL=="system", SUBSYSTEM=="dma_heap", TAG+="uaccess"
(and on Rasperry Pi devices any users in the video group get access)
This was just a quick fix for the PoC. Now that we are ready to move out of the PoC phase and start actually integrating this into distributions the question becomes if this is an acceptable solution; or if we need some other way to deal with this ?
Specifically the question is if this will have any negative security implications? I can certainly see this being used to do some sort of denial of service attack on the system (1). This is especially true for the cma heap which generally speaking is a limited resource.
But devices tagged for uaccess are only opened up to users who are physcially present behind the machine and those can just hit the powerbutton, so I don't believe that any *on purpose* DOS is part of the thread model. Any accidental DOS would be a userspace stack bug.
Do you foresee any other negative security implications from allowing physically present non root users to create (u)dma-bufs ?
Regards,
Hans
1) There are some limits in
drivers/dma-buf/udmabuf.cand distributions could narrow these.
Signature · 2 lines
With best wishes Dmitry
Hi Dmitry,
Quoted 15 lines · Dmitry Baryshkov
Quoted 5 lines · Hans de Goede
Hi dma-buf maintainers, et.al.,
Various people have been working on making complex/MIPI cameras work OOTB with mainline Linux kernels and an opensource userspace stack.
Quoted 8 lines
The generic solution adds a software ISP (for Debayering and 3A) to libcamera. Libcamera's API guarantees that buffers handed to applications using it are dma-bufs so that these can be passed to e.g. a video encoder.
In order to meet this API guarantee the libcamera software ISP allocates dma-bufs from userspace through one of the
/dev/dma_heap/*heaps. For the Fedora COPR repo for the PoC of this:
https://hansdegoede.dreamwidth.org/28153.html
Quoted 6 lines
Is there any reason for allocating DMA buffers for libcamera through
/dev/dma_heap/rather than allocating them via corresponding media device and then giving them away to DRM / VPU / etc?At least this should solve the permission usecase: if the app can open camera device, it can allocate a buffer.
This is with a software ISP, the input buffers with raw bayer data come from a v4l2 device, but the output buffers with the processed data are purely userspace managed in this case.
Regards,
Hans
Quoted 22 lines · Hans de Goede
Hi Dmitry,
Quoted 19 lines · Dmitry Baryshkov
Quoted 5 lines · Hans de Goede
Hi dma-buf maintainers, et.al.,
Various people have been working on making complex/MIPI cameras work OOTB with mainline Linux kernels and an opensource userspace stack.
Quoted 8 lines
The generic solution adds a software ISP (for Debayering and 3A) to libcamera. Libcamera's API guarantees that buffers handed to applications using it are dma-bufs so that these can be passed to e.g. a video encoder.
In order to meet this API guarantee the libcamera software ISP allocates dma-bufs from userspace through one of the
/dev/dma_heap/*heaps. For the Fedora COPR repo for the PoC of this:
https://hansdegoede.dreamwidth.org/28153.htmlIs there any reason for allocating DMA buffers for libcamera through
/dev/dma_heap/rather than allocating them via corresponding media device and then giving them away to DRM / VPU / etc?
Quoted 6 lines
Quoted 2 lines
At least this should solve the permission usecase: if the app can open camera device, it can allocate a buffer.
This is with a software ISP, the input buffers with raw bayer data come from a v4l2 device, but the output buffers with the processed data are purely userspace managed in this case.
Ah, I see. Then why do you require the DMA-ble buffer at all? If you are providing data to VPU or DRM, then you should be able to get the buffer from the data-consuming device. If the data is further processed by a userspace app, then it should not require DMA memory at all.
My main concern is that dma-heaps is both too generic and too limiting for the generic library implementation.
Signature · 2 lines
With best wishes Dmitry
Quoted 3 lines · Dmitry Baryshkov
Ah, I see. Then why do you require the DMA-ble buffer at all? If you are providing data to VPU or DRM, then you should be able to get the buffer from the data-consuming device.
Because we don't necessarily know what the consuming device is, if any.
Could be VPU, could be Zoom/Hangouts via pipewire, could for argument sake be GPU or DSP.
Also if we introduce a dependency on another device to allocate the output buffers - say always taking the output buffer from the GPU, then we've added another dependency which is more difficult to guarantee across different arches.
---
bod
Quoted 8 lines · Bryan O'Donoghue
Quoted 3 lines · Dmitry Baryshkov
Ah, I see. Then why do you require the DMA-ble buffer at all? If you are providing data to VPU or DRM, then you should be able to get the buffer from the data-consuming device.
Because we don't necessarily know what the consuming device is, if any.
Quoted 7 lines
Could be VPU, could be Zoom/Hangouts via pipewire, could for argument sake be GPU or DSP.
Also if we introduce a dependency on another device to allocate the output buffers - say always taking the output buffer from the GPU, then we've added another dependency which is more difficult to guarantee across different arches.
Yes. And it should be expected. It's a consumer who knows the restrictions on the buffer. As I wrote, Zoom/Hangouts should not require a DMA buffer at all. Applications should be able to allocate the buffer out of the generic memory. GPUs might also have different requirements. Consider GPUs with VRAM. It might be beneficial to allocate a buffer out of VRAM rather than generic DMA mem.
Signature · 2 lines
With best wishes Dmitry
Quoted 11 lines · Dmitry Baryshkov
Quoted 10 lines · Bryan O'Donoghue
Quoted 3 lines · Dmitry Baryshkov
Ah, I see. Then why do you require the DMA-ble buffer at all? If you are providing data to VPU or DRM, then you should be able to get the buffer from the data-consuming device.
Because we don't necessarily know what the consuming device is, if any.
Could be VPU, could be Zoom/Hangouts via pipewire, could for argument sake be GPU or DSP.
Quoted 8 lines
Quoted 4 lines
Also if we introduce a dependency on another device to allocate the output buffers - say always taking the output buffer from the GPU, then we've added another dependency which is more difficult to guarantee across different arches.
Yes. And it should be expected. It's a consumer who knows the restrictions on the buffer. As I wrote, Zoom/Hangouts should not require a DMA buffer at all.
Why not ? If you want to capture to a buffer that you then compose on the screen without copying data, dma-buf is the way to go. That's the Linux solution for buffer sharing.
Quoted 2 lines
Applications should be able to allocate
the buffer out of the generic memory.
If applications really want to copy data and degrade performance, they are free to shoot themselves in the foot of course. Applications (or compositors) need to support copying as a fallback in the worst case, but all components should at least aim for the zero-copy case.
Quoted 3 lines
GPUs might also have different
requirements. Consider GPUs with VRAM. It might be beneficial to allocate a buffer out of VRAM rather than generic DMA mem.
Absolutely. For that we need a centralized device memory allocator in userspace. An effort was started by James Jones in 2016, see [1]. It has unfortunately stalled. If I didn't have a camera framework to develop, I would try to tackle that issue :-)
[1] https://www.x.org/wiki/Events/XDC2016/Program/Unix_Device_Memory_Allocation.pdf
Signature · 3 lines
Regards, Laurent Pinchart
Quoted 18 lines · Laurent Pinchart
Quoted 16 lines · Dmitry Baryshkov
Quoted 14 lines · Bryan O'Donoghue
Quoted 3 lines · Dmitry Baryshkov
Ah, I see. Then why do you require the DMA-ble buffer at all? If you are providing data to VPU or DRM, then you should be able to get the buffer from the data-consuming device.
Because we don't necessarily know what the consuming device is, if any.
Could be VPU, could be Zoom/Hangouts via pipewire, could for argument sake be GPU or DSP.
Also if we introduce a dependency on another device to allocate the output buffers - say always taking the output buffer from the GPU, then we've added another dependency which is more difficult to guarantee across different arches.
Quoted 7 lines
Quoted 3 lines
Yes. And it should be expected. It's a consumer who knows the restrictions on the buffer. As I wrote, Zoom/Hangouts should not require a DMA buffer at all.
Why not ? If you want to capture to a buffer that you then compose on the screen without copying data, dma-buf is the way to go. That's the Linux solution for buffer sharing.
Yes. But it should be allocated by the DRM driver. As Sima wrote, there is no guarantee that the buffer allocated from dma-heaps is accessible to the GPU.
Quoted 8 lines
Quoted 3 lines
Applications should be able to allocate
the buffer out of the generic memory.If applications really want to copy data and degrade performance, they are free to shoot themselves in the foot of course. Applications (or compositors) need to support copying as a fallback in the worst case, but all components should at least aim for the zero-copy case.
I'd say that they should aim for the optimal case. It might include both zero-copying access from another DMA master or simple software processing of some kind.
Quoted 8 lines
Quoted 3 lines
GPUs might also have different
requirements. Consider GPUs with VRAM. It might be beneficial to allocate a buffer out of VRAM rather than generic DMA mem.Absolutely. For that we need a centralized device memory allocator in userspace. An effort was started by James Jones in 2016, see [1]. It has unfortunately stalled. If I didn't have a camera framework to develop, I would try to tackle that issue :-)
I'll review the talk. However the fact that the effort has stalled most likely means that 'one fits them all' approach didn't really fly well. We have too many usecases.
Signature · 2 lines
With best wishes Dmitry
Quoted 22 lines · Dmitry Baryshkov
Quoted 21 lines · Laurent Pinchart
Quoted 19 lines · Dmitry Baryshkov
Quoted 7 lines · Bryan O'Donoghue
Quoted 3 lines · Dmitry Baryshkov
Ah, I see. Then why do you require the DMA-ble buffer at all? If you are providing data to VPU or DRM, then you should be able to get the buffer from the data-consuming device.
Because we don't necessarily know what the consuming device is, if any.
Quoted 7 lines
Could be VPU, could be Zoom/Hangouts via pipewire, could for argument sake be GPU or DSP.
Also if we introduce a dependency on another device to allocate the output buffers - say always taking the output buffer from the GPU, then we've added another dependency which is more difficult to guarantee across different arches.
Yes. And it should be expected. It's a consumer who knows the restrictions on the buffer. As I wrote, Zoom/Hangouts should not require a DMA buffer at all.
Quoted 7 lines
Quoted 3 lines
Why not ? If you want to capture to a buffer that you then compose on the screen without copying data, dma-buf is the way to go. That's the Linux solution for buffer sharing.
Yes. But it should be allocated by the DRM driver. As Sima wrote, there is no guarantee that the buffer allocated from dma-heaps is accessible to the GPU.
No disagreement there. From a libcamera point of view, we want to import buffers allocated externally. It's for use cases where applications can't provide dma buf instances easily that we need to allocate them through the libcamera buffer allocator helper. That helper has to a) provide dma buf fds, and b) make a best effort to allocate buffers that will have a decent chance of being usable by other devices. We're open to exploring other solutions than dma heaps, although I wonder what the dma heaps are for if nobody enables them :-)
Quoted 26 lines
Quoted 7 lines
Quoted 2 lines
Applications should be able to allocate
the buffer out of the generic memory.If applications really want to copy data and degrade performance, they are free to shoot themselves in the foot of course. Applications (or compositors) need to support copying as a fallback in the worst case, but all components should at least aim for the zero-copy case.
I'd say that they should aim for the optimal case. It might include both zero-copying access from another DMA master or simple software processing of some kind.
Quoted 8 lines
Quoted 3 lines
GPUs might also have different
requirements. Consider GPUs with VRAM. It might be beneficial to allocate a buffer out of VRAM rather than generic DMA mem.Absolutely. For that we need a centralized device memory allocator in userspace. An effort was started by James Jones in 2016, see [1]. It has unfortunately stalled. If I didn't have a camera framework to develop, I would try to tackle that issue :-)
I'll review the talk. However the fact that the effort has stalled most likely means that 'one fits them all' approach didn't really fly well. We have too many usecases.
Signature · 3 lines
Regards, Laurent Pinchart
Quoted 49 lines · Dmitry Baryshkov
Quoted 18 lines · Laurent Pinchart
Quoted 16 lines · Dmitry Baryshkov
Quoted 14 lines · Bryan O'Donoghue
Quoted 3 lines · Dmitry Baryshkov
Ah, I see. Then why do you require the DMA-ble buffer at all? If you are providing data to VPU or DRM, then you should be able to get the buffer from the data-consuming device.
Because we don't necessarily know what the consuming device is, if any.
Could be VPU, could be Zoom/Hangouts via pipewire, could for argument sake be GPU or DSP.
Also if we introduce a dependency on another device to allocate the output buffers - say always taking the output buffer from the GPU, then we've added another dependency which is more difficult to guarantee across different arches.
Quoted 7 lines
Quoted 3 lines
Yes. And it should be expected. It's a consumer who knows the restrictions on the buffer. As I wrote, Zoom/Hangouts should not require a DMA buffer at all.
Why not ? If you want to capture to a buffer that you then compose on the screen without copying data, dma-buf is the way to go. That's the Linux solution for buffer sharing.
Yes. But it should be allocated by the DRM driver. As Sima wrote, there is no guarantee that the buffer allocated from dma-heaps is accessible to the GPU.
Quoted 8 lines
Quoted 3 lines
Applications should be able to allocate
the buffer out of the generic memory.If applications really want to copy data and degrade performance, they are free to shoot themselves in the foot of course. Applications (or compositors) need to support copying as a fallback in the worst case, but all components should at least aim for the zero-copy case.
I'd say that they should aim for the optimal case. It might include both zero-copying access from another DMA master or simple software processing of some kind.
Quoted 4 lines
Quoted 3 lines
GPUs might also have different
requirements. Consider GPUs with VRAM. It might be beneficial to allocate a buffer out of VRAM rather than generic DMA mem.
Quoted 8 lines
Quoted 4 lines
Absolutely. For that we need a centralized device memory allocator in userspace. An effort was started by James Jones in 2016, see [1]. It has unfortunately stalled. If I didn't have a camera framework to develop, I would try to tackle that issue :-)
I'll review the talk. However the fact that the effort has stalled most likely means that 'one fits them all' approach didn't really fly well. We have too many usecases.
I think there's two reasons:
- It's a really hard problem with many aspects. Where you need to allocate the buffer is just one of the myriad of issues a common allocator needs to solve.
- Every linux-based os has their own solution for these, and the one that suffers most has an entirely different one from everyone else: Android uses binder services to allow apps to make these allocations, keep track of them and make sure there's no abuse. And if there is, it can just nuke the app.
Cheers, Sima
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Quoted 57 lines · Daniel Vetter
Quoted 47 lines · Dmitry Baryshkov
Quoted 17 lines · Laurent Pinchart
Quoted 16 lines · Dmitry Baryshkov
Quoted 14 lines · Bryan O'Donoghue
Quoted 3 lines · Dmitry Baryshkov
Ah, I see. Then why do you require the DMA-ble buffer at all? If you are providing data to VPU or DRM, then you should be able to get the buffer from the data-consuming device.
Because we don't necessarily know what the consuming device is, if any.
Could be VPU, could be Zoom/Hangouts via pipewire, could for argument sake be GPU or DSP.
Also if we introduce a dependency on another device to allocate the output buffers - say always taking the output buffer from the GPU, then we've added another dependency which is more difficult to guarantee across different arches.
Quoted 7 lines
Quoted 3 lines
Yes. And it should be expected. It's a consumer who knows the restrictions on the buffer. As I wrote, Zoom/Hangouts should not require a DMA buffer at all.
Why not ? If you want to capture to a buffer that you then compose on the screen without copying data, dma-buf is the way to go. That's the Linux solution for buffer sharing.
Yes. But it should be allocated by the DRM driver. As Sima wrote, there is no guarantee that the buffer allocated from dma-heaps is accessible to the GPU.
Quoted 8 lines
Quoted 3 lines
Applications should be able to allocate
the buffer out of the generic memory.If applications really want to copy data and degrade performance, they are free to shoot themselves in the foot of course. Applications (or compositors) need to support copying as a fallback in the worst case, but all components should at least aim for the zero-copy case.
I'd say that they should aim for the optimal case. It might include both zero-copying access from another DMA master or simple software processing of some kind.
Quoted 4 lines
Quoted 3 lines
GPUs might also have different
requirements. Consider GPUs with VRAM. It might be beneficial to allocate a buffer out of VRAM rather than generic DMA mem.Quoted 8 lines
Quoted 4 lines
Absolutely. For that we need a centralized device memory allocator in userspace. An effort was started by James Jones in 2016, see [1]. It has unfortunately stalled. If I didn't have a camera framework to develop, I would try to tackle that issue :-)
I'll review the talk. However the fact that the effort has stalled most likely means that 'one fits them all' approach didn't really fly well. We have too many usecases.
Quoted 5 lines
I think there's two reasons:
- It's a really hard problem with many aspects. Where you need to allocate the buffer is just one of the myriad of issues a common allocator needs to solve.
The other large problem is picking up an optimal pixel format. I wonder if that could be decoupled from the allocation. That could help moving forward.
Quoted 5 lines
- Every linux-based os has their own solution for these, and the one that suffers most has an entirely different one from everyone else: Android uses binder services to allow apps to make these allocations, keep track of them and make sure there's no abuse. And if there is, it can just nuke the app.
Signature · 3 lines
Regards, Laurent Pinchart
Quoted 24 lines · Dmitry Baryshkov
Quoted 22 lines · Laurent Pinchart
Quoted 19 lines · Dmitry Baryshkov
Quoted 7 lines · Bryan O'Donoghue
Quoted 3 lines · Dmitry Baryshkov
Ah, I see. Then why do you require the DMA-ble buffer at all? If you are providing data to VPU or DRM, then you should be able to get the buffer from the data-consuming device.
Because we don't necessarily know what the consuming device is, if any.
Quoted 7 lines
Could be VPU, could be Zoom/Hangouts via pipewire, could for argument sake be GPU or DSP.
Also if we introduce a dependency on another device to allocate the output buffers - say always taking the output buffer from the GPU, then we've added another dependency which is more difficult to guarantee across different arches.
Yes. And it should be expected. It's a consumer who knows the restrictions on the buffer. As I wrote, Zoom/Hangouts should not require a DMA buffer at all.
Quoted 7 lines
Quoted 3 lines
Why not ? If you want to capture to a buffer that you then compose on the screen without copying data, dma-buf is the way to go. That's the Linux solution for buffer sharing.
Yes. But it should be allocated by the DRM driver. As Sima wrote, there is no guarantee that the buffer allocated from dma-heaps is accessible to the GPU.
And there is no guarantee that the buffer allocated from the GPU is accessible to the display engine. In practice, I've yet to see an issue with that assumption.
And there's the other elephant in the room that hasn't been addressed. Buffers typically allocated by the data-consuming frameworks are coherent buffers, which on arm/arm64 usually mean non-cacheable.
Performances are *terrible*. Meanwhile, dma-heaps and dma-buf provide cacheable buffers with a cache synchronization API, which allow to have it run much faster.
Maxime
Quoted 6 lines · Bryan O'Donoghue
Quoted 3 lines · Dmitry Baryshkov
Ah, I see. Then why do you require the DMA-ble buffer at all? If you are providing data to VPU or DRM, then you should be able to get the buffer from the data-consuming device.
Because we don't necessarily know what the consuming device is, if any.
Well ... that's an entirely different issue. And it's unsolved.
Currently the approach is to allocate where the constraints are usually most severe (like display, if you need that, or the camera module for input) and then just pray the stack works out without too much copying. All userspace (whether the generic glue or the userspace driver depends a bit upon the exact api) does need to have a copy fallback for these sharing cases, ideally with the copying accelerated by hw.
If you try to solve this by just preemptive allocating everything as cma buffers, then you'll make the situation substantially worse (because now you're wasting tons of cma memory where you might not even need it). And without really solving the problem, since for some gpus that memory might be unusable (because you cannot scan that out on any discrete gpu, and sometimes not even on an integrated one).
-Sima
Quoted 10 lines
Could be VPU, could be Zoom/Hangouts via pipewire, could for argument sake be GPU or DSP.
Also if we introduce a dependency on another device to allocate the output buffers - say always taking the output buffer from the GPU, then we've added another dependency which is more difficult to guarantee across different arches.
---
bod
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Quoted 17 lines · Daniel Vetter
Quoted 6 lines · Bryan O'Donoghue
Quoted 3 lines · Dmitry Baryshkov
Ah, I see. Then why do you require the DMA-ble buffer at all? If you are providing data to VPU or DRM, then you should be able to get the buffer from the data-consuming device.
Because we don't necessarily know what the consuming device is, if any.
Well ... that's an entirely different issue. And it's unsolved.
Currently the approach is to allocate where the constraints are usually most severe (like display, if you need that, or the camera module for input) and then just pray the stack works out without too much copying. All userspace (whether the generic glue or the userspace driver depends a bit upon the exact api) does need to have a copy fallback for these sharing cases, ideally with the copying accelerated by hw.
Quoted 6 lines
If you try to solve this by just preemptive allocating everything as cma buffers, then you'll make the situation substantially worse (because now you're wasting tons of cma memory where you might not even need it). And without really solving the problem, since for some gpus that memory might be unusable (because you cannot scan that out on any discrete gpu, and sometimes not even on an integrated one).
I think we have a general agreement that the proposed solution is a stop-gap measure for an unsolved issue.
Note that libcamera is already designed that way. The API is designed to import buffers, using dma-buf file handles. If an application has a way to allocate dma-buf instances through another means (from the display or from a video encoder for instance), it should do so, and use those buffers with libcamera.
For applications that don't have an easy way to get hold of dma-buf instances, we have a buffer allocator helper as a side component. That allocator uses the underlying camera capture device, and allocates buffers from the V4L2 video device. It's only on platforms where we have no hardware camera processing (or, rather, platforms where the hardware vendors doesn't give us access to the camera hardware, such as recent Intel SoCs, or Qualcomm SoCs used in ARM laptops) that we need to allocate memory elsewhere.
In the long run, I want a centralized memory allocator accessible by userspace applications (something similar in purpose to gralloc on Android), and I want to get rid of buffer allocation in libcamera (and even in V4L2, in the even longer term). That's the long run.
Shorter term, we have a problem to solve, and the best option we have found so far is to rely on dma-buf heaps as a backend for the frame buffer allocatro helper in libcamera for the use case described above. This won't work in 100% of the cases, clearly. It's a stop-gap measure until we can do better.
Quoted 7 lines
Quoted 7 lines
Could be VPU, could be Zoom/Hangouts via pipewire, could for argument sake be GPU or DSP.
Also if we introduce a dependency on another device to allocate the output buffers - say always taking the output buffer from the GPU, then we've added another dependency which is more difficult to guarantee across different arches.
Signature · 3 lines
Regards, Laurent Pinchart
Hi,
Le mardi 07 mai 2024 à 21:36 +0300, Laurent Pinchart a écrit :
Quoted 5 lines
Shorter term, we have a problem to solve, and the best option we have found so far is to rely on dma-buf heaps as a backend for the frame buffer allocatro helper in libcamera for the use case described above. This won't work in 100% of the cases, clearly. It's a stop-gap measure until we can do better.
Considering the security concerned raised on this thread with dmabuf heap allocation not be restricted by quotas, you'd get what you want quickly with memfd + udmabuf instead (which is accounted already).
It was raised that distro don't enable udmabuf, but as stated there by Hans, in any cases distro needs to take action to make the softISP works. This alternative is easy and does not interfere in anyway with your future plan or the libcamera API. You could even have both dmabuf heap (for Raspbian) and the safer memfd+udmabuf for the distro with security concerns.
And for the long term plan, we can certainly get closer by fixing that issue with accounting. This issue also applied to v4l2 io-ops, so it would be nice to find common set of helpers to fix these exporters.
regards,
Nicolas
Quoted 19 lines · Nicolas Dufresne
Hi,
Le mardi 07 mai 2024 à 21:36 +0300, Laurent Pinchart a écrit :
Quoted 5 lines
Shorter term, we have a problem to solve, and the best option we have found so far is to rely on dma-buf heaps as a backend for the frame buffer allocatro helper in libcamera for the use case described above. This won't work in 100% of the cases, clearly. It's a stop-gap measure until we can do better.
Considering the security concerned raised on this thread with dmabuf heap allocation not be restricted by quotas, you'd get what you want quickly with memfd + udmabuf instead (which is accounted already).
It was raised that distro don't enable udmabuf, but as stated there by Hans, in any cases distro needs to take action to make the softISP works. This alternative is easy and does not interfere in anyway with your future plan or the libcamera API. You could even have both dmabuf heap (for Raspbian) and the safer memfd+udmabuf for the distro with security concerns.
Quoted 3 lines
And for the long term plan, we can certainly get closer by fixing that issue with accounting. This issue also applied to v4l2 io-ops, so it would be nice to find common set of helpers to fix these exporters.
Yeah if this is just for softisp, then memfd + udmabuf is also what I was about to suggest. Not just as a stopgap, but as the real official thing.
udmabuf does kinda allow you to pin memory, but we can easily fix that by adding the right accounting and then either let mlock rlimits or cgroups kernel memory limits enforce good behavior.
-Sima
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Quoted 18 lines · Daniel Vetter
Quoted 17 lines · Nicolas Dufresne
Le mardi 07 mai 2024 à 21:36 +0300, Laurent Pinchart a écrit :
Quoted 5 lines
Shorter term, we have a problem to solve, and the best option we have found so far is to rely on dma-buf heaps as a backend for the frame buffer allocatro helper in libcamera for the use case described above. This won't work in 100% of the cases, clearly. It's a stop-gap measure until we can do better.
Considering the security concerned raised on this thread with dmabuf heap allocation not be restricted by quotas, you'd get what you want quickly with memfd + udmabuf instead (which is accounted already).
It was raised that distro don't enable udmabuf, but as stated there by Hans, in any cases distro needs to take action to make the softISP works. This alternative is easy and does not interfere in anyway with your future plan or the libcamera API. You could even have both dmabuf heap (for Raspbian) and the safer memfd+udmabuf for the distro with security concerns.
Quoted 6 lines
Quoted 3 lines
And for the long term plan, we can certainly get closer by fixing that issue with accounting. This issue also applied to v4l2 io-ops, so it would be nice to find common set of helpers to fix these exporters.
Yeah if this is just for softisp, then memfd + udmabuf is also what I was about to suggest. Not just as a stopgap, but as the real official thing.
Long term I still want a centralized memory allocator, at which point libcamera should stop allocating buffers at all.
Quoted 3 lines
udmabuf does kinda allow you to pin memory, but we can easily fix that by adding the right accounting and then either let mlock rlimits or cgroups kernel memory limits enforce good behavior.
Signature · 3 lines
Regards, Laurent Pinchart
Quoted 23 lines · Laurent Pinchart
Quoted 22 lines · Daniel Vetter
Quoted 20 lines · Nicolas Dufresne
Le mardi 07 mai 2024 à 21:36 +0300, Laurent Pinchart a écrit :
Quoted 5 lines
Shorter term, we have a problem to solve, and the best option we have found so far is to rely on dma-buf heaps as a backend for the frame buffer allocatro helper in libcamera for the use case described above. This won't work in 100% of the cases, clearly. It's a stop-gap measure until we can do better.
Considering the security concerned raised on this thread with dmabuf heap allocation not be restricted by quotas, you'd get what you want quickly with memfd + udmabuf instead (which is accounted already).
It was raised that distro don't enable udmabuf, but as stated there by Hans, in any cases distro needs to take action to make the softISP works. This alternative is easy and does not interfere in anyway with your future plan or the libcamera API. You could even have both dmabuf heap (for Raspbian) and the safer memfd+udmabuf for the distro with security concerns.
And for the long term plan, we can certainly get closer by fixing that issue with accounting. This issue also applied to v4l2 io-ops, so it would be nice to find common set of helpers to fix these exporters.
Quoted 5 lines
Quoted 2 lines
Yeah if this is just for softisp, then memfd + udmabuf is also what I was about to suggest. Not just as a stopgap, but as the real official thing.
Long term I still want a centralized memory allocator, at which point libcamera should stop allocating buffers at all.
And to be clear, udmabuf could be fine for the time being. At least as long as we don't find any shortcoming while testing it :-)
Quoted 3 lines
Quoted 3 lines
udmabuf does kinda allow you to pin memory, but we can easily fix that by adding the right accounting and then either let mlock rlimits or cgroups kernel memory limits enforce good behavior.
Signature · 3 lines
Regards, Laurent Pinchart
Quoted 24 lines · Daniel Vetter
Quoted 19 lines · Nicolas Dufresne
Hi,
Le mardi 07 mai 2024 à 21:36 +0300, Laurent Pinchart a écrit :
Quoted 5 lines
Shorter term, we have a problem to solve, and the best option we have found so far is to rely on dma-buf heaps as a backend for the frame buffer allocatro helper in libcamera for the use case described above. This won't work in 100% of the cases, clearly. It's a stop-gap measure until we can do better.
Considering the security concerned raised on this thread with dmabuf heap allocation not be restricted by quotas, you'd get what you want quickly with memfd + udmabuf instead (which is accounted already).
It was raised that distro don't enable udmabuf, but as stated there by Hans, in any cases distro needs to take action to make the softISP works. This alternative is easy and does not interfere in anyway with your future plan or the libcamera API. You could even have both dmabuf heap (for Raspbian) and the safer memfd+udmabuf for the distro with security concerns.
Quoted 3 lines
And for the long term plan, we can certainly get closer by fixing that issue with accounting. This issue also applied to v4l2 io-ops, so it would be nice to find common set of helpers to fix these exporters.
Quoted 6 lines
Yeah if this is just for softisp, then memfd + udmabuf is also what I was about to suggest. Not just as a stopgap, but as the real official thing.
udmabuf does kinda allow you to pin memory, but we can easily fix that by adding the right accounting and then either let mlock rlimits or cgroups kernel memory limits enforce good behavior.
I think the main drawback with memfd is that it'll be broken for devices without an IOMMU, and while you said that it's uncommon for GPUs, it's definitely not for codecs and display engines.
Maxime
Quoted 28 lines · Maxime Ripard
Quoted 27 lines · Daniel Vetter
Quoted 19 lines · Nicolas Dufresne
Hi,
Le mardi 07 mai 2024 à 21:36 +0300, Laurent Pinchart a écrit :
Quoted 5 lines
Shorter term, we have a problem to solve, and the best option we have found so far is to rely on dma-buf heaps as a backend for the frame buffer allocatro helper in libcamera for the use case described above. This won't work in 100% of the cases, clearly. It's a stop-gap measure until we can do better.
Considering the security concerned raised on this thread with dmabuf heap allocation not be restricted by quotas, you'd get what you want quickly with memfd + udmabuf instead (which is accounted already).
It was raised that distro don't enable udmabuf, but as stated there by Hans, in any cases distro needs to take action to make the softISP works. This alternative is easy and does not interfere in anyway with your future plan or the libcamera API. You could even have both dmabuf heap (for Raspbian) and the safer memfd+udmabuf for the distro with security concerns.
Quoted 3 lines
And for the long term plan, we can certainly get closer by fixing that issue with accounting. This issue also applied to v4l2 io-ops, so it would be nice to find common set of helpers to fix these exporters.
Yeah if this is just for softisp, then memfd + udmabuf is also what I was about to suggest. Not just as a stopgap, but as the real official thing.
Quoted 7 lines
Quoted 3 lines
udmabuf does kinda allow you to pin memory, but we can easily fix that by adding the right accounting and then either let mlock rlimits or cgroups kernel memory limits enforce good behavior.
I think the main drawback with memfd is that it'll be broken for devices without an IOMMU, and while you said that it's uncommon for GPUs, it's definitely not for codecs and display engines.
If the application wants to share buffers between the camera and a display engine or codec, it should arguably not use the libcamera FrameBufferAllocator, but allocate the buffers from the display or the encoder. memfd wouldn't be used in that case.
We need to eat our own dogfood though. If we want to push the responsibility for buffer allocation in the buffer sharing case to the application, we need to modify the cam application to do so when using the KMS backend.
Signature · 3 lines
Regards, Laurent Pinchart
Le lundi 13 mai 2024 à 11:34 +0300, Laurent Pinchart a écrit :
Quoted 42 lines · Maxime Ripard
Quoted 28 lines · Daniel Vetter
Quoted 27 lines · Nicolas Dufresne
Quoted 19 lines
Hi,
Le mardi 07 mai 2024 à 21:36 +0300, Laurent Pinchart a écrit :
Quoted 5 lines
Shorter term, we have a problem to solve, and the best option we have found so far is to rely on dma-buf heaps as a backend for the frame buffer allocatro helper in libcamera for the use case described above. This won't work in 100% of the cases, clearly. It's a stop-gap measure until we can do better.
Considering the security concerned raised on this thread with dmabuf heap allocation not be restricted by quotas, you'd get what you want quickly with memfd + udmabuf instead (which is accounted already).
It was raised that distro don't enable udmabuf, but as stated there by Hans, in any cases distro needs to take action to make the softISP works. This alternative is easy and does not interfere in anyway with your future plan or the libcamera API. You could even have both dmabuf heap (for Raspbian) and the safer memfd+udmabuf for the distro with security concerns.
Quoted 3 lines
And for the long term plan, we can certainly get closer by fixing that issue with accounting. This issue also applied to v4l2 io-ops, so it would be nice to find common set of helpers to fix these exporters.
Yeah if this is just for softisp, then memfd + udmabuf is also what I was about to suggest. Not just as a stopgap, but as the real official thing.
Quoted 7 lines
Quoted 3 lines
udmabuf does kinda allow you to pin memory, but we can easily fix that by adding the right accounting and then either let mlock rlimits or cgroups kernel memory limits enforce good behavior.
I think the main drawback with memfd is that it'll be broken for devices without an IOMMU, and while you said that it's uncommon for GPUs, it's definitely not for codecs and display engines.
If the application wants to share buffers between the camera and a display engine or codec, it should arguably not use the libcamera FrameBufferAllocator, but allocate the buffers from the display or the encoder. memfd wouldn't be used in that case.
Quoted 5 lines
We need to eat our own dogfood though. If we want to push the responsibility for buffer allocation in the buffer sharing case to the application, we need to modify the cam application to do so when using the KMS backend.
Agreed, and the new dmabuf feedback on wayland can also be used on top of this.
You'll hit the same limitation as we hit in GStreamer, which is that KMS driver only offer allocation for render buffers and most of them are missing allocators for YUV buffers, even though they can import in these formats. (kms allocators, except dumb, which has other issues, are format aware).
Nicolas
Quoted 48 lines · Nicolas Dufresne
Le lundi 13 mai 2024 à 11:34 +0300, Laurent Pinchart a écrit :
Quoted 42 lines · Maxime Ripard
Quoted 28 lines · Daniel Vetter
Quoted 27 lines · Nicolas Dufresne
Quoted 19 lines
Hi,
Le mardi 07 mai 2024 à 21:36 +0300, Laurent Pinchart a écrit :
Quoted 5 lines
Shorter term, we have a problem to solve, and the best option we have found so far is to rely on dma-buf heaps as a backend for the frame buffer allocatro helper in libcamera for the use case described above. This won't work in 100% of the cases, clearly. It's a stop-gap measure until we can do better.
Considering the security concerned raised on this thread with dmabuf heap allocation not be restricted by quotas, you'd get what you want quickly with memfd + udmabuf instead (which is accounted already).
It was raised that distro don't enable udmabuf, but as stated there by Hans, in any cases distro needs to take action to make the softISP works. This alternative is easy and does not interfere in anyway with your future plan or the libcamera API. You could even have both dmabuf heap (for Raspbian) and the safer memfd+udmabuf for the distro with security concerns.
Quoted 3 lines
And for the long term plan, we can certainly get closer by fixing that issue with accounting. This issue also applied to v4l2 io-ops, so it would be nice to find common set of helpers to fix these exporters.
Yeah if this is just for softisp, then memfd + udmabuf is also what I was about to suggest. Not just as a stopgap, but as the real official thing.
Quoted 7 lines
Quoted 3 lines
udmabuf does kinda allow you to pin memory, but we can easily fix that by adding the right accounting and then either let mlock rlimits or cgroups kernel memory limits enforce good behavior.
I think the main drawback with memfd is that it'll be broken for devices without an IOMMU, and while you said that it's uncommon for GPUs, it's definitely not for codecs and display engines.
If the application wants to share buffers between the camera and a display engine or codec, it should arguably not use the libcamera FrameBufferAllocator, but allocate the buffers from the display or the encoder. memfd wouldn't be used in that case.
Quoted 4 lines
We need to eat our own dogfood though. If we want to push the responsibility for buffer allocation in the buffer sharing case to the application, we need to modify the cam application to do so when using the KMS backend.
Quoted 6 lines
Agreed, and the new dmabuf feedback on wayland can also be used on top of this.
You'll hit the same limitation as we hit in GStreamer, which is that KMS driver only offer allocation for render buffers and most of them are missing allocators for YUV buffers, even though they can import in these formats. (kms allocators, except dumb, which has other issues, are format aware).
My experience on Arm platforms is that the KMS drivers offer allocation for scanout buffers, not render buffers, and mostly using the dumb allocator API. If the KMS device can scan out YUV natively, YUV buffer allocation should be supported. Am I missing something here ?
Signature · 3 lines
Regards, Laurent Pinchart
Le mardi 14 mai 2024 à 23:42 +0300, Laurent Pinchart a écrit :
Quoted 9 lines
Quoted 4 lines
You'll hit the same limitation as we hit in GStreamer, which is that KMS driver only offer allocation for render buffers and most of them are missing allocators for YUV buffers, even though they can import in these formats. (kms allocators, except dumb, which has other issues, are format aware).
My experience on Arm platforms is that the KMS drivers offer allocation for scanout buffers, not render buffers, and mostly using the dumb allocator API. If the KMS device can scan out YUV natively, YUV buffer allocation should be supported. Am I missing something here ?
There is two APIs, Dumb is the legacy allocation API, only used by display drivers indeed, and the API does not include a pixel format or a modifier. The allocation of YUV buffer has been made through a small hack,
bpp = number of bits per component (of luma plane if multiple planes) width = width
height = height * X
Where X will vary, "3 / 2" is used for 420 subsampling, "2" for 422 and "3" for
444. It is far from idea, requires deep knowledge of each formats in the application and cannot allocate each planes seperatly.
The second is to use the driver specific allocation API. This is then abstracted by GBM. This allows allocating render buffers with notably modifiers and/or use cases. But no support for YUV formats or multi-planar formats.
Nicolas
Hi Nicolas,
Quoted 6 lines · On Wed, May 15, 2024 at 01:43:58PM -0400…
Le mardi 14 mai 2024 à 23:42 +0300, Laurent Pinchart a écrit :
Quoted 5 lines
Quoted 4 lines
You'll hit the same limitation as we hit in GStreamer, which is that KMS driver only offer allocation for render buffers and most of them are missing allocators for YUV buffers, even though they can import in these formats. (kms allocators, except dumb, which has other issues, are format aware).
Quoted 6 lines
Quoted 4 lines
My experience on Arm platforms is that the KMS drivers offer allocation for scanout buffers, not render buffers, and mostly using the dumb allocator API. If the KMS device can scan out YUV natively, YUV buffer allocation should be supported. Am I missing something here ?
There is two APIs, Dumb is the legacy allocation API, only used by display
Is it legacy only ? I understand the dumb buffers API to be officially supported, to allocate scanout buffers suitable for software rendering.
Quoted 10 lines
drivers indeed, and the API does not include a pixel format or a modifier. The allocation of YUV buffer has been made through a small hack,
bpp = number of bits per component (of luma plane if multiple planes) width = width
height = height * XWhere X will vary, "3 / 2" is used for 420 subsampling, "2" for 422 and "3" for
444. It is far from idea, requires deep knowledge of each formats in the application
I'm not sure I see that as an issue, but our experiences and uses cases may vary :-)
Quoted 1 line
and cannot allocate each planes seperatly.
For semi-planar or planar formats, unless I'm mistaken, you can either allocate a single buffer and use it with appropriate offsets when constructing your framebuffer (with DRM_IOCTL_MODE_ADDFB2), or allocate one buffer per plane.
Quoted 3 lines
The second is to use the driver specific allocation API. This is then abstracted by GBM. This allows allocating render buffers with notably modifiers and/or use cases. But no support for YUV formats or multi-planar formats.
GBM is the way to go for render buffers indeed. It has been designed with only graphics buffer management use cases in mind, so it's unfortunately not an option as a generic allocator, at least in its current form.
Signature · 3 lines
Regards, Laurent Pinchart
Le jeudi 16 mai 2024 à 14:27 +0300, Laurent Pinchart a écrit :
Quoted 26 lines
Hi Nicolas,
Quoted 6 lines · On Wed, May 15, 2024 at 01:43:58PM -0400…
Le mardi 14 mai 2024 à 23:42 +0300, Laurent Pinchart a écrit :
Quoted 5 lines
Quoted 4 lines
You'll hit the same limitation as we hit in GStreamer, which is that KMS driver only offer allocation for render buffers and most of them are missing allocators for YUV buffers, even though they can import in these formats. (kms allocators, except dumb, which has other issues, are format aware).
Quoted 6 lines
Quoted 4 lines
My experience on Arm platforms is that the KMS drivers offer allocation for scanout buffers, not render buffers, and mostly using the dumb allocator API. If the KMS device can scan out YUV natively, YUV buffer allocation should be supported. Am I missing something here ?
There is two APIs, Dumb is the legacy allocation API, only used by display
Is it legacy only ? I understand the dumb buffers API to be officially supported, to allocate scanout buffers suitable for software rendering.
Quoted 7 lines
drivers indeed, and the API does not include a pixel format or a modifier. The allocation of YUV buffer has been made through a small hack,
bpp = number of bits per component (of luma plane if multiple planes) width = width
height = height * X
Quoted 6 lines
Quoted 3 lines
Where X will vary, "3 / 2" is used for 420 subsampling, "2" for 422 and "3" for
444. It is far from idea, requires deep knowledge of each formats in the applicationI'm not sure I see that as an issue, but our experiences and uses cases may vary :-)
Its extra burden, and does not scale to all available pixel formats. My reply was for readers education as I feel like a lot of linux-media dev don't have a clue of what is going on at the rendering side. This ensure a minimum knowledge to everyone commenting.
And yes, within the GFX community, Dumb allocation is to be killed and replacement completely in the future, it simply does not have a complete replacement yet.
Quoted 7 lines
Quoted 2 lines
and cannot allocate each planes seperatly.
For semi-planar or planar formats, unless I'm mistaken, you can either allocate a single buffer and use it with appropriate offsets when constructing your framebuffer (with
DRM_IOCTL_MODE_ADDFB2), or allocate one buffer per plane.
We have use cases were single allocation is undesirable, but I don't really feel like this is important enough for me to type this explanation. Ping me if you care.
Quoted 10 lines
Quoted 4 lines
The second is to use the driver specific allocation API. This is then abstracted by GBM. This allows allocating render buffers with notably modifiers and/or use cases. But no support for YUV formats or multi-planar formats.
GBM is the way to go for render buffers indeed. It has been designed with only graphics buffer management use cases in mind, so it's unfortunately not an option as a generic allocator, at least in its current form.
What I perhaps should have highlighted that is that all these allocators in the GFX (called DRM, but meh) subsystem abstract away some deep knowledge of the HW requirements. Heaps are lower level APIs that assume that userspace have this knowledge. The Android and ChromeOS solution is to take the implementation from the kernel and move it into userspace, see minigbm from chromeos, or gralloc from Android. As these two projects are device centric, they are not usable on generic Linux. Heaps might have some future, but not without other piece of the puzzle.
To come back to you wanting heaps in libcamera, because it makes them better for rendered or display. Well today this is a lie you make to yourself, because this is just a tiny bit of the puzzle, it is pure luck if you allocate dmabuf is usable but a foreign device. At the end of the day, this is just a fallback to satisfy that application are not forced to allocate that memory in libcamera.
Thus, I strongly recommend the udmabuf in the short term. Finally, moving to heaps when the reported issue is resolved, as then it gives more options and reduce the number of layers.
Nicolas
Quoted 74 lines · On Thu, May 16, 2024 at 01:11:51PM GMT, [email protected]…
Le jeudi 16 mai 2024 à 14:27 +0300, Laurent Pinchart a écrit :
Quoted 26 lines
Hi Nicolas,
Quoted 6 lines · On Wed, May 15, 2024 at 01:43:58PM -0400…
Le mardi 14 mai 2024 à 23:42 +0300, Laurent Pinchart a écrit :
Quoted 5 lines
Quoted 4 lines
You'll hit the same limitation as we hit in GStreamer, which is that KMS driver only offer allocation for render buffers and most of them are missing allocators for YUV buffers, even though they can import in these formats. (kms allocators, except dumb, which has other issues, are format aware).
Quoted 6 lines
Quoted 4 lines
My experience on Arm platforms is that the KMS drivers offer allocation for scanout buffers, not render buffers, and mostly using the dumb allocator API. If the KMS device can scan out YUV natively, YUV buffer allocation should be supported. Am I missing something here ?
There is two APIs, Dumb is the legacy allocation API, only used by display
Is it legacy only ? I understand the dumb buffers API to be officially supported, to allocate scanout buffers suitable for software rendering.
Quoted 7 lines
drivers indeed, and the API does not include a pixel format or a modifier. The allocation of YUV buffer has been made through a small hack,
bpp = number of bits per component (of luma plane if multiple planes) width = width
height = height * XQuoted 6 lines
Quoted 3 lines
Where X will vary, "3 / 2" is used for 420 subsampling, "2" for 422 and "3" for
444. It is far from idea, requires deep knowledge of each formats in the applicationI'm not sure I see that as an issue, but our experiences and uses cases may vary :-)
Its extra burden, and does not scale to all available pixel formats. My reply was for readers education as I feel like a lot of linux-media dev don't have a clue of what is going on at the rendering side. This ensure a minimum knowledge to everyone commenting.
And yes, within the GFX community, Dumb allocation is to be killed and replacement completely in the future, it simply does not have a complete replacement yet.
Quoted 7 lines
Quoted 2 lines
and cannot allocate each planes seperatly.
For semi-planar or planar formats, unless I'm mistaken, you can either allocate a single buffer and use it with appropriate offsets when constructing your framebuffer (with
DRM_IOCTL_MODE_ADDFB2), or allocate one buffer per plane.We have use cases were single allocation is undesirable, but I don't really feel like this is important enough for me to type this explanation. Ping me if you care.
Quoted 10 lines
Quoted 4 lines
The second is to use the driver specific allocation API. This is then abstracted by GBM. This allows allocating render buffers with notably modifiers and/or use cases. But no support for YUV formats or multi-planar formats.
GBM is the way to go for render buffers indeed. It has been designed with only graphics buffer management use cases in mind, so it's unfortunately not an option as a generic allocator, at least in its current form.
What I perhaps should have highlighted that is that all these allocators in the GFX (called DRM, but meh) subsystem abstract away some deep knowledge of the HW requirements. Heaps are lower level APIs that assume that userspace have this knowledge. The Android and ChromeOS solution is to take the implementation from the kernel and move it into userspace, see minigbm from chromeos, or gralloc from Android. As these two projects are device centric, they are not usable on generic Linux. Heaps might have some future, but not without other piece of the puzzle.
Quoted 5 lines
To come back to you wanting heaps in libcamera, because it makes them better for rendered or display. Well today this is a lie you make to yourself, because this is just a tiny bit of the puzzle, it is pure luck if you allocate dmabuf is usable but a foreign device. At the end of the day, this is just a fallback to satisfy that application are not forced to allocate that memory in libcamera.
I mean, it's pure luck, but can you point to any platform supported upstream where it wouldn't work?
Quoted 3 lines
Thus, I strongly recommend the udmabuf in the short term. Finally, moving to heaps when the reported issue is resolved, as then it gives more options and reduce the number of layers.
udmabuf wouldn't work with any platform without an IOMMU. We have plenty of those.
All things considered, while I agree that it isn't the ideal solution, we really don't have a better (ie, works on a larger set of platforms) solution at the moment or in the next 5 years.
Maxime
Le mardi 21 mai 2024 à 10:43 +0200, Maxime Ripard a écrit :
Quoted 75 lines · On Thu, May 16, 2024 at 01:11:51PM GMT, [email protected]…
Quoted 74 lines
Le jeudi 16 mai 2024 à 14:27 +0300, Laurent Pinchart a écrit :
Quoted 26 lines
Hi Nicolas,
Quoted 6 lines · On Wed, May 15, 2024 at 01:43:58PM -0400…
Le mardi 14 mai 2024 à 23:42 +0300, Laurent Pinchart a écrit :
Quoted 5 lines
Quoted 4 lines
You'll hit the same limitation as we hit in GStreamer, which is that KMS driver only offer allocation for render buffers and most of them are missing allocators for YUV buffers, even though they can import in these formats. (kms allocators, except dumb, which has other issues, are format aware).
Quoted 6 lines
Quoted 4 lines
My experience on Arm platforms is that the KMS drivers offer allocation for scanout buffers, not render buffers, and mostly using the dumb allocator API. If the KMS device can scan out YUV natively, YUV buffer allocation should be supported. Am I missing something here ?
There is two APIs, Dumb is the legacy allocation API, only used by display
Is it legacy only ? I understand the dumb buffers API to be officially supported, to allocate scanout buffers suitable for software rendering.
Quoted 7 lines
drivers indeed, and the API does not include a pixel format or a modifier. The allocation of YUV buffer has been made through a small hack,
bpp = number of bits per component (of luma plane if multiple planes) width = width
height = height * XQuoted 6 lines
Quoted 3 lines
Where X will vary, "3 / 2" is used for 420 subsampling, "2" for 422 and "3" for
444. It is far from idea, requires deep knowledge of each formats in the applicationI'm not sure I see that as an issue, but our experiences and uses cases may vary :-)
Its extra burden, and does not scale to all available pixel formats. My reply was for readers education as I feel like a lot of linux-media dev don't have a clue of what is going on at the rendering side. This ensure a minimum knowledge to everyone commenting.
And yes, within the GFX community, Dumb allocation is to be killed and replacement completely in the future, it simply does not have a complete replacement yet.
Quoted 7 lines
Quoted 2 lines
and cannot allocate each planes seperatly.
For semi-planar or planar formats, unless I'm mistaken, you can either allocate a single buffer and use it with appropriate offsets when constructing your framebuffer (with
DRM_IOCTL_MODE_ADDFB2), or allocate one buffer per plane.We have use cases were single allocation is undesirable, but I don't really feel like this is important enough for me to type this explanation. Ping me if you care.
Quoted 10 lines
Quoted 4 lines
The second is to use the driver specific allocation API. This is then abstracted by GBM. This allows allocating render buffers with notably modifiers and/or use cases. But no support for YUV formats or multi-planar formats.
GBM is the way to go for render buffers indeed. It has been designed with only graphics buffer management use cases in mind, so it's unfortunately not an option as a generic allocator, at least in its current form.
What I perhaps should have highlighted that is that all these allocators in the GFX (called DRM, but meh) subsystem abstract away some deep knowledge of the HW requirements. Heaps are lower level APIs that assume that userspace have this knowledge. The Android and ChromeOS solution is to take the implementation from the kernel and move it into userspace, see minigbm from chromeos, or gralloc from Android. As these two projects are device centric, they are not usable on generic Linux. Heaps might have some future, but not without other piece of the puzzle.
Quoted 8 lines
Quoted 5 lines
To come back to you wanting heaps in libcamera, because it makes them better for rendered or display. Well today this is a lie you make to yourself, because this is just a tiny bit of the puzzle, it is pure luck if you allocate dmabuf is usable but a foreign device. At the end of the day, this is just a fallback to satisfy that application are not forced to allocate that memory in libcamera.
I mean, it's pure luck, but can you point to any platform supported upstream where it wouldn't work?
Most AMD GPUs needs 256 bytes aligned strides. So unless you have that hardcoded in libcamera its one case that often fail to import. There is no kernel API to know anyway, so hardcoding is becoming common with the popularity of the GPUs. Mali have a 64 bytes alignment required, except for some YUV formats on very recent Mesa. If you hardcode for AMD, it works for Mali too.
Intel display driver is an interesting one. Most of software dmabuf exporter will enable cpu cache (UVC driver included). That driver fails to reject these dmabuf assuming the exporter will always flush the cache. UVC driver as exported does not, its not so clear to me if dmaheap+softISP (assuming softISP do the dmabuf sync calls) will work or not. UVC to display artifacts were still visible on 6.8 (last time I tested).
Quoted 7 lines
Quoted 4 lines
Thus, I strongly recommend the udmabuf in the short term. Finally, moving to heaps when the reported issue is resolved, as then it gives more options and reduce the number of layers.
udmabuf wouldn't work with any platform without an IOMMU. We have plenty of those.
Its up to userspace to decide to allocate scattered or not, but again, generically there is no API to let the application (softISP) know. Many of our real-life tests concluded that using malloc data in software video processing to finally do a memcpy() into final "device" memory is faster then using "coherent" allocation or doing the cache handling.
Quoted 4 lines
All things considered, while I agree that it isn't the ideal solution, we really don't have a better (ie, works on a larger set of platforms) solution at the moment or in the next 5 years.
Indeed. In the short term, I like the idea that we'll first make it safe to expose the heaps at all time, so at least we have a choice. Today, on most major distributions none of the solution mentioned are available. I have no idea how much work this is.
Nicolas
Quoted 4 lines · Laurent Pinchart
My experience on Arm platforms is that the KMS drivers offer allocation for scanout buffers, not render buffers, and mostly using the dumb allocator API. If the KMS device can scan out YUV natively, YUV buffer allocation should be supported. Am I missing something here ?
Note that dumb buffers are only intended for simple software-rendering use-cases. Anything more complicated (e.g. involving GPU rendering) should use another mechanism.
Quoted 10 lines · Simon Ser
Quoted 4 lines · Laurent Pinchart
My experience on Arm platforms is that the KMS drivers offer allocation for scanout buffers, not render buffers, and mostly using the dumb allocator API. If the KMS device can scan out YUV natively, YUV buffer allocation should be supported. Am I missing something here ?
Note that dumb buffers are only intended for simple software-rendering use-cases. Anything more complicated (e.g. involving GPU rendering) should use another mechanism.
Sure. Even if dumb buffers may work for GPU rendering in some cases, there's no guarantee they will, so they shouldn't be used.
My comment was related to scanout buffers, as I was puzzled by Nicolas mentioning how "KMS drivers only offer allocation for render buffers". On Arm platforms the render buffers are allocated on the GPU's DRM device as far as I understand, while the KMS drivers allocate scanout buffers using the dumb buffers API.
Signature · 3 lines
Regards, Laurent Pinchart
Le jeudi 16 mai 2024 à 14:20 +0300, Laurent Pinchart a écrit :
Quoted 12 lines · Simon Ser
Quoted 10 lines · Laurent Pinchart
Quoted 5 lines
My experience on Arm platforms is that the KMS drivers offer allocation for scanout buffers, not render buffers, and mostly using the dumb allocator API. If the KMS device can scan out YUV natively, YUV buffer allocation should be supported. Am I missing something here ?
Note that dumb buffers are only intended for simple software-rendering use-cases. Anything more complicated (e.g. involving GPU rendering) should use another mechanism.
Quoted 9 lines
Sure. Even if dumb buffers may work for GPU rendering in some cases, there's no guarantee they will, so they shouldn't be used.
My comment was related to scanout buffers, as I was puzzled by Nicolas mentioning how "KMS drivers only offer allocation for render buffers". On Arm platforms the render buffers are allocated on the GPU's DRM device as far as I understand, while the KMS drivers allocate scanout buffers using the dumb buffers API.
The message is getting distorted. I'm saying that not all supported formats have an allocation API in DRM/KMS drivers. Most YUV formats needed for media handling (GPU or scannout) are not supported.
Nicolas
p.s. I feel like commenters thinks its evident for userspace application to know if they are doing scanout or GPU ... while in reality, they offload their allocated buffer to a compositor which will have to dynamically juggle between these two with its own heuristic.
Le lundi 13 mai 2024 à 10:29 +0200, Maxime Ripard a écrit :
Quoted 28 lines · Daniel Vetter
Quoted 27 lines · Nicolas Dufresne
Quoted 19 lines
Hi,
Le mardi 07 mai 2024 à 21:36 +0300, Laurent Pinchart a écrit :
Quoted 5 lines
Shorter term, we have a problem to solve, and the best option we have found so far is to rely on dma-buf heaps as a backend for the frame buffer allocatro helper in libcamera for the use case described above. This won't work in 100% of the cases, clearly. It's a stop-gap measure until we can do better.
Considering the security concerned raised on this thread with dmabuf heap allocation not be restricted by quotas, you'd get what you want quickly with memfd + udmabuf instead (which is accounted already).
It was raised that distro don't enable udmabuf, but as stated there by Hans, in any cases distro needs to take action to make the softISP works. This alternative is easy and does not interfere in anyway with your future plan or the libcamera API. You could even have both dmabuf heap (for Raspbian) and the safer memfd+udmabuf for the distro with security concerns.
Quoted 3 lines
And for the long term plan, we can certainly get closer by fixing that issue with accounting. This issue also applied to v4l2 io-ops, so it would be nice to find common set of helpers to fix these exporters.
Yeah if this is just for softisp, then memfd + udmabuf is also what I was about to suggest. Not just as a stopgap, but as the real official thing.
Quoted 7 lines
Quoted 3 lines
udmabuf does kinda allow you to pin memory, but we can easily fix that by adding the right accounting and then either let mlock rlimits or cgroups kernel memory limits enforce good behavior.
I think the main drawback with memfd is that it'll be broken for devices without an IOMMU, and while you said that it's uncommon for GPUs, it's definitely not for codecs and display engines.
In the context of libcamera, the allocation and the alignment done to the video frame is done completely blindly. In that context, there is a lot more then just the allocation type that can go wrong and will lead to a memory copy. The upside of memfd, is that the read cache will help speeding up the copies if they are needed.
Another important point is that this is only used if the application haven't provided frames. If your embedded application is non-generic, and you have permissions to access the right heap, the application can solve your specific issue. But in the generic Linux space, Linux kernel API are just insufficient for the "just work" scenario.
Nicolas
Quoted 37 lines · Nicolas Dufresne
Le lundi 13 mai 2024 à 10:29 +0200, Maxime Ripard a écrit :
Quoted 28 lines · Daniel Vetter
Quoted 27 lines · Nicolas Dufresne
Quoted 19 lines
Hi,
Le mardi 07 mai 2024 à 21:36 +0300, Laurent Pinchart a écrit :
Quoted 5 lines
Shorter term, we have a problem to solve, and the best option we have found so far is to rely on dma-buf heaps as a backend for the frame buffer allocatro helper in libcamera for the use case described above. This won't work in 100% of the cases, clearly. It's a stop-gap measure until we can do better.
Considering the security concerned raised on this thread with dmabuf heap allocation not be restricted by quotas, you'd get what you want quickly with memfd + udmabuf instead (which is accounted already).
It was raised that distro don't enable udmabuf, but as stated there by Hans, in any cases distro needs to take action to make the softISP works. This alternative is easy and does not interfere in anyway with your future plan or the libcamera API. You could even have both dmabuf heap (for Raspbian) and the safer memfd+udmabuf for the distro with security concerns.
Quoted 3 lines
And for the long term plan, we can certainly get closer by fixing that issue with accounting. This issue also applied to v4l2 io-ops, so it would be nice to find common set of helpers to fix these exporters.
Yeah if this is just for softisp, then memfd + udmabuf is also what I was about to suggest. Not just as a stopgap, but as the real official thing.
Quoted 7 lines
Quoted 3 lines
udmabuf does kinda allow you to pin memory, but we can easily fix that by adding the right accounting and then either let mlock rlimits or cgroups kernel memory limits enforce good behavior.
I think the main drawback with memfd is that it'll be broken for devices without an IOMMU, and while you said that it's uncommon for GPUs, it's definitely not for codecs and display engines.
Quoted 5 lines
In the context of libcamera, the allocation and the alignment done to the video frame is done completely blindly. In that context, there is a lot more then just the allocation type that can go wrong and will lead to a memory copy. The upside of memfd, is that the read cache will help speeding up the copies if they are needed.
dma-heaps provide cacheable buffers too...
Quoted 5 lines
Another important point is that this is only used if the application haven't provided frames. If your embedded application is non-generic, and you have permissions to access the right heap, the application can solve your specific issue. But in the generic Linux space, Linux kernel API are just insufficient for the "just work" scenario.
... but they also provide semantics around the memory buffers that no other allocation API do. There's at least the mediatek secure playback series and another one that I've started to work on to allocate ECC protected or unprotected buffers that are just the right use case for the heaps, and the target frameworks aren't.
Maxime
Le lundi 13 mai 2024 à 15:51 +0200, Maxime Ripard a écrit :
Quoted 38 lines · Nicolas Dufresne
Quoted 37 lines
Le lundi 13 mai 2024 à 10:29 +0200, Maxime Ripard a écrit :
Quoted 35 lines · Daniel Vetter
Quoted 24 lines · Nicolas Dufresne
Quoted 19 lines
Hi,
Le mardi 07 mai 2024 à 21:36 +0300, Laurent Pinchart a écrit :
Quoted 5 lines
Shorter term, we have a problem to solve, and the best option we have found so far is to rely on dma-buf heaps as a backend for the frame buffer allocatro helper in libcamera for the use case described above. This won't work in 100% of the cases, clearly. It's a stop-gap measure until we can do better.
Considering the security concerned raised on this thread with dmabuf heap allocation not be restricted by quotas, you'd get what you want quickly with memfd + udmabuf instead (which is accounted already).
It was raised that distro don't enable udmabuf, but as stated there by Hans, in any cases distro needs to take action to make the softISP works. This alternative is easy and does not interfere in anyway with your future plan or the libcamera API. You could even have both dmabuf heap (for Raspbian) and the safer memfd+udmabuf for the distro with security concerns.
Quoted 3 lines
And for the long term plan, we can certainly get closer by fixing that issue with accounting. This issue also applied to v4l2 io-ops, so it would be nice to find common set of helpers to fix these exporters.
Quoted 6 lines
Yeah if this is just for softisp, then memfd + udmabuf is also what I was about to suggest. Not just as a stopgap, but as the real official thing.
udmabuf does kinda allow you to pin memory, but we can easily fix that by adding the right accounting and then either let mlock rlimits or cgroups kernel memory limits enforce good behavior.
I think the main drawback with memfd is that it'll be broken for devices without an IOMMU, and while you said that it's uncommon for GPUs, it's definitely not for codecs and display engines.
Quoted 7 lines
Quoted 5 lines
In the context of libcamera, the allocation and the alignment done to the video frame is done completely blindly. In that context, there is a lot more then just the allocation type that can go wrong and will lead to a memory copy. The upside of memfd, is that the read cache will help speeding up the copies if they are needed.
dma-heaps provide cacheable buffers too...
Yes, and why we have cache hints in V4L2 now. There is no clue that softISP code can read to make the right call. The required cache management in undefined until all the importer are known. I also don't think heaps currently care to adapt the dmabuf sync behaviour based on the different importers, or the addition of a new importer. On top of which, there is insufficient information on the device to really deduce what is needed.
Quoted 7 lines
Quoted 6 lines
Another important point is that this is only used if the application haven't provided frames. If your embedded application is non-generic, and you have permissions to access the right heap, the application can solve your specific issue. But in the generic Linux space, Linux kernel API are just insufficient for the "just work" scenario.
Quoted 5 lines
... but they also provide semantics around the memory buffers that no other allocation API do. There's at least the mediatek secure playback series and another one that I've started to work on to allocate ECC protected or unprotected buffers that are just the right use case for the heaps, and the target frameworks aren't.
Let's agree we are both off topic now. The libcamera softISP is currently purely software, and cannot write to any form of protected memory. As for ECC, I would hope this usage will be coded in the application and that this application has been authorized to access the appropriate heaps.
And finally, none of this fixes the issue that the heap allocation are not being accounted properly and allow of an easy memory DoS. So uaccess should be granted with care, meaning that defaulting a "desktop" library to that, means it will most of the time not work at all.
Nicolas
Quoted 69 lines · Nicolas Dufresne
Le lundi 13 mai 2024 à 15:51 +0200, Maxime Ripard a écrit :
Quoted 36 lines · Nicolas Dufresne
Quoted 35 lines
Le lundi 13 mai 2024 à 10:29 +0200, Maxime Ripard a écrit :
Quoted 33 lines · Daniel Vetter
Quoted 22 lines · Nicolas Dufresne
Quoted 17 lines
Le mardi 07 mai 2024 à 21:36 +0300, Laurent Pinchart a écrit :
Quoted 5 lines
Shorter term, we have a problem to solve, and the best option we have found so far is to rely on dma-buf heaps as a backend for the frame buffer allocatro helper in libcamera for the use case described above. This won't work in 100% of the cases, clearly. It's a stop-gap measure until we can do better.
Considering the security concerned raised on this thread with dmabuf heap allocation not be restricted by quotas, you'd get what you want quickly with memfd + udmabuf instead (which is accounted already).
It was raised that distro don't enable udmabuf, but as stated there by Hans, in any cases distro needs to take action to make the softISP works. This alternative is easy and does not interfere in anyway with your future plan or the libcamera API. You could even have both dmabuf heap (for Raspbian) and the safer memfd+udmabuf for the distro with security concerns.
Quoted 3 lines
And for the long term plan, we can certainly get closer by fixing that issue with accounting. This issue also applied to v4l2 io-ops, so it would be nice to find common set of helpers to fix these exporters.
Quoted 6 lines
Yeah if this is just for softisp, then memfd + udmabuf is also what I was about to suggest. Not just as a stopgap, but as the real official thing.
udmabuf does kinda allow you to pin memory, but we can easily fix that by adding the right accounting and then either let mlock rlimits or cgroups kernel memory limits enforce good behavior.
I think the main drawback with memfd is that it'll be broken for devices without an IOMMU, and while you said that it's uncommon for GPUs, it's definitely not for codecs and display engines.
Quoted 7 lines
Quoted 5 lines
In the context of libcamera, the allocation and the alignment done to the video frame is done completely blindly. In that context, there is a lot more then just the allocation type that can go wrong and will lead to a memory copy. The upside of memfd, is that the read cache will help speeding up the copies if they are needed.
dma-heaps provide cacheable buffers too...
Yes, and why we have cache hints in V4L2 now. There is no clue that softISP code can read to make the right call. The required cache management in undefined until all the importer are known. I also don't think heaps currently care to adapt the dmabuf sync behaviour based on the different importers, or the addition of a new importer. On top of which, there is insufficient information on the device to really deduce what is needed.
Quoted 11 lines
Quoted 5 lines
Another important point is that this is only used if the application haven't provided frames. If your embedded application is non-generic, and you have permissions to access the right heap, the application can solve your specific issue. But in the generic Linux space, Linux kernel API are just insufficient for the "just work" scenario.
... but they also provide semantics around the memory buffers that no other allocation API do. There's at least the mediatek secure playback series and another one that I've started to work on to allocate ECC protected or unprotected buffers that are just the right use case for the heaps, and the target frameworks aren't.
Let's agree we are both off topic now. The libcamera softISP is currently purely software, and cannot write to any form of protected memory. As for ECC, I would hope this usage will be coded in the application and that this application has been authorized to access the appropriate heaps.
Quoted 4 lines
And finally, none of this fixes the issue that the heap allocation are not being accounted properly and allow of an easy memory DoS. So uaccess should be granted with care, meaning that defaulting a "desktop" library to that, means it will most of the time not work at all.
I think that issue should be fixed, regardless of whether or not we end up using dma heaps for libcamera. If we do use them, maybe there will be a higher incentive for somebody involved in this conversation to tackle that problem first :-) And maybe, as a result, the rest of the Linux community will consider with a more open mind usage of dma heaps on desktop systems.
Signature · 3 lines
Regards, Laurent Pinchart
Hi,
Le mardi 14 mai 2024 à 23:45 +0300, Laurent Pinchart a écrit :
Quoted 11 lines
Quoted 4 lines
And finally, none of this fixes the issue that the heap allocation are not being accounted properly and allow of an easy memory DoS. So uaccess should be granted with care, meaning that defaulting a "desktop" library to that, means it will most of the time not work at all.
I think that issue should be fixed, regardless of whether or not we end up using dma heaps for libcamera. If we do use them, maybe there will be a higher incentive for somebody involved in this conversation to tackle that problem first :-) And maybe, as a result, the rest of the Linux community will consider with a more open mind usage of dma heaps on desktop systems.
The strict reality is that if libcamera offer no alternatives, some OS will enable it and reduce their security. I totally agree this issue needs to be fixed regardless of libcamera, or even dma heaps. DMABuf allocation should be accounted and limited to quotas whether it comes from a GPU, Display, V4L2 or other type of supported devices. I would also not recommend dropping your heap support (or preventing it from being merged) in libcamera.
Nicolas