rust: DamFence: Remove warning about unsignaled drop
Subsystems (4)
Philipp Stanner
maintainerPATCH 1/1 rust: DamFence: Remove warning about unsignaled drop
DriverFence::drop() contains a pr_warn!() for when the fence drops unsignaled. This warning gets into the way of canonical teardown in
Rust: when you drop a lock, it unlocks itself, when you drop a KBox, it frees itself etc. Congruently, users should be able to tear down fences they don't care about anymore with ECANCELED, which is already the default error code.
Remove the warning.
--- rust/kernel/dma_buf/dma_fence.rs | 1 - 1 file changed, 1 deletion(-)
rust/kernel/dma_buf/dma_fence.rs +0 -1
@@ -942,7 +942,6 @@ fn drop(&mut self) {// SAFETY: `guard` is valid until the `call_rcu()` below. let signaled:bool =unsafe { bindings::dma_fence_test_signaled_flag (guard.as_raw ()) };if !signaled { -pr_err! ("DriverFence drops unsignaled. Danger of memory corruption!\n" );// SAFETY: `guard` is valid until the `call_rcu()` below. The fence // must not have been signaled yet, which we check directly above. unsafe { bindings::dma_fence_set_error (guard.as_raw (),ECANCELED .to_errno ()) };
Signature · 1 line
2.56.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] rust: dma_fence: stale DriverFence rustdoc after warning removal in drop() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] rust: dma_fence: stale DriverFence rustdoc after warning removal in drop()
--
commit 1fa69b16f405cf6ecb2511fccf990063ada22d3a
Author: Philipp Stanner <[email protected]>
rust: DamFence: Remove warning about unsignaled drop
This commit removes the error message printed when a DriverFence is dropped before being signaled.
rust/kernel/dma_buf/dma_fence.rs fn drop(&mut self) { · line 939
Quoted 12 lines
rust/kernel/dma_buf/dma_fence.rs +0
-1@@ -934,7 +934,6 @@ fn drop(&mut self) {// SAFETY: `guard` is valid until the `call_rcu()` below. let signaled:bool =unsafe { bindings::dma_fence_test_signaled_flag (guard.as_raw ()) };if !signaled { -pr_err! ("DriverFence drops unsignaled. Danger of memory corruption!\n" );// SAFETY: `guard` is valid until the `call_rcu()` below. The fence // must not have been signaled yet, which we check directly above. unsafe { bindings::dma_fence_set_error (guard.as_raw (),ECANCELED .to_errno ()) };
[Severity: Low]
Should the rustdoc comment on struct DriverFence also be updated now that dropping an unsignaled fence is canonical teardown and no longer logs a message?
rust/kernel/dma_buf/dma_fence.rs:
/// As a last resort, a [`DriverFence`] will signal itself if it drops
/// unsignaled and print a warning.
pub struct DriverFence<'a, T: FenceContextOps> {Also, the commit subject misspells the component as DamFence, and the commit message refers to the removed pr_err!() call as pr_warn!().
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Christian König
maintainerQuoted 6 lines · Philipp Stanner
DriverFence::drop()contains a pr_warn!() for when the fence drops unsignaled. This warning gets into the way of canonical teardown in
Rust: when you drop a lock, it unlocks itself, when you drop a KBox, it frees itself etc. Congruently, users should be able to tear down fences they don't care about anymore with ECANCELED, which is already the default error code.
Well a dma_fence should never automatically signal when the last reference drops.
Leaving it unsignaled is the most defensive path we can go. That can lead to deadlocks but it can at least not lead to memory corruption.
Regards,
Christian.
Quoted 21 lines
Remove the warning.
Suggested-by Danilo Krummrich <[email protected]>Signed-off-by Philipp Stanner <[email protected]>--- rust/kernel/dma_buf/dma_fence.rs | 1 - 1 file changed, 1 deletion(-)rust/kernel/dma_buf/dma_fence.rs +0
-1@@ -942,7 +942,6 @@ fn drop(&mut self) {// SAFETY: `guard` is valid until the `call_rcu()` below. let signaled:bool =unsafe { bindings::dma_fence_test_signaled_flag (guard.as_raw ()) };if !signaled { -pr_err! ("DriverFence drops unsignaled. Danger of memory corruption!\n" );// SAFETY: `guard` is valid until the `call_rcu()` below. The fence // must not have been signaled yet, which we check directly above. unsafe { bindings::dma_fence_set_error (guard.as_raw (),ECANCELED .to_errno ()) };
Quoted 8 lines · Christian König
Quoted 6 lines · Philipp Stanner
DriverFence::drop()contains a pr_warn!() for when the fence drops unsignaled. This warning gets into the way of canonical teardown in
Rust: when you drop a lock, it unlocks itself, when you drop a KBox, it frees itself etc. Congruently, users should be able to tear down fences they don't care about anymore with ECANCELED, which is already the default error code.
Quoted 5 lines
Well a
dma_fenceshould never automatically signal when the last reference drops.Leaving it unsignaled is the most defensive path we can go. That can lead to deadlocks but it can at least not lead to memory corruption.
True, you choose between the Unholy Trinity of DRM: deadlocks, UAF and memory corruption :]
In this particular case, however, Rust does not allow for preventing the user from signaling "forgotten" fences anyways, no matter how we write the fence core:
struct DriverFoo {
f: DriverFence<()>,
}impl Drop for Driver Foo {
pub fn drop(&mut self) {
// Signals fence implicitly & automatically.
self.f.signal(ECANCELED);
}
}So we decided to die this death because this way we get rid of the two of the three problems that *can* be solved in the API.
But it will please you to hear that all users of drm::JobQueue do not have access to fences directly anyways, they can only signal them through JobQueue. Thus, they can also not forget them (signaled or unsignaled), because they would have to forget / leak the entire JobQueue as a whole, so effectively forgetting also their GPU ring. If that would happen, the driver would be broken beyond repair anyways.
Regards
P.
Christian König
maintainerQuoted 26 lines · Philipp Stanner
Quoted 8 lines · Christian König
Quoted 6 lines · Philipp Stanner
DriverFence::drop()contains a pr_warn!() for when the fence drops unsignaled. This warning gets into the way of canonical teardown in
Rust: when you drop a lock, it unlocks itself, when you drop a KBox, it frees itself etc. Congruently, users should be able to tear down fences they don't care about anymore with ECANCELED, which is already the default error code.Quoted 5 lines
Well a
dma_fenceshould never automatically signal when the last reference drops.Leaving it unsignaled is the most defensive path we can go. That can lead to deadlocks but it can at least not lead to memory corruption.
True, you choose between the Unholy Trinity of DRM: deadlocks, UAF and memory corruption :]
In this particular case, however, Rust does not allow for preventing the user from signaling "forgotten" fences anyways, no matter how we write the fence core:
struct DriverFoo { f: DriverFence<()>, }
Quoted 6 lines
impl Drop for Driver Foo {
pub fn drop(&mut self) { // Signals fence implicitly & automatically. self.f.signal(ECANCELED); } }
Yeah, exactly that is still a bad idea :) See the HW eventually doesn't necessarily stop doing DMA if you unload the driver.
I even know about one case where a driver fully intentionally leaks memory on unload because the devs couldn't figure out which FW keeps writing into a buffer.
This was suggested as problem solving approach for AMDGPU as well, but we have been able to avoid it by clearing the DMA master flag in the PCI config dword.
The fundamental problem here is that a software object running out of scope is not an indicator that the HW stops doing something.
So just signaling all fences with ECANCELED is not enough. When the driver runs out of scope without all fences being signaling we seriously need at least some kind of warning in the system log, potentially even better call BUG() or maybe even panic().
Cheers,
Christian.
Quoted 15 lines
So we decided to die this death because this way we get rid of the two of the three problems that *can* be solved in the API.
But it will please you to hear that all users of
drm::JobQueuedo not have access to fences directly anyways, they can only signal them through JobQueue. Thus, they can also not forget them (signaled or unsignaled), because they would have to forget / leak the entire JobQueue as a whole, so effectively forgetting also their GPU ring. If that would happen, the driver would be broken beyond repair anyways.Regards
P.
Quoted 12 lines · Christian König
Quoted 10 lines · Philipp Stanner
struct DriverFoo { f: DriverFence<()>, }impl Drop for Driver Foo {
pub fn drop(&mut self) { // Signals fence implicitly & automatically. self.f.signal(ECANCELED); } }
Quoted 2 lines
Yeah, exactly that is still a bad idea :) See the HW eventually doesn't necessarily stop doing DMA if you unload the driver.
I know.
Quoted 1 line
So just signaling all fences with ECANCELED is not enough.
Yes. What I was trying to tell you as that you have no way to enforce this in Rust or C.
dma_fence_signal(f);free_driver_resources();
// Driver forgets to stop the hardware. Memory corruption occurs.
You see? Neither your BUG_ON() or WARN_ON() saves you if the driver does it like that, no matter the programming language. You cannot prevent it.
The only one who understands how the hardware works is the driver. The only one who can stop it is the driver.
In Rust, the place where a driver drops its JobQueue is the same place where it has to halt its GPU ring correctly. Enforcing this cannot be done by dma_fence code.
P.
Christian König
maintainerQuoted 32 lines · Philipp Stanner
Quoted 12 lines · Christian König
Quoted 10 lines · Philipp Stanner
struct DriverFoo { f: DriverFence<()>, }impl Drop for Driver Foo {
pub fn drop(&mut self) { // Signals fence implicitly & automatically. self.f.signal(ECANCELED); } }Quoted 2 lines
Yeah, exactly that is still a bad idea :) See the HW eventually doesn't necessarily stop doing DMA if you unload the driver.
I know.
Quoted 1 line
So just signaling all fences with ECANCELED is not enough.
Yes. What I was trying to tell you as that you have no way to enforce this in Rust or C.
dma_fence_signal(f);free_driver_resources();
// Driver forgets to stop the hardware. Memory corruption occurs.You see? Neither your
BUG_ON()orWARN_ON()saves you if the driver does it like that, no matter the programming language. You cannot prevent it.
Quoted 6 lines
The only one who understands how the hardware works is the driver. The only one who can stop it is the driver.
In Rust, the place where a driver drops its JobQueue is the same place where it has to halt its GPU ring correctly. Enforcing this cannot be done by
dma_fencecode.
I completely agree that we can't fix things from the dma_fence code, but it is our obligation to tell the user that something very very bad happened.
I mean that is what WARN_ON(), BUG_ON() and panic() are essentially good for, they never fix anything but just point out that the system should probably be rebooted and driver developers informed.
Regards,
Christian.
Quoted 3 lines
P.
Danilo Krummrich
maintainerQuoted 33 lines · Christian König
Quoted 32 lines · Philipp Stanner
Quoted 12 lines · Christian König
Quoted 10 lines · Philipp Stanner
struct DriverFoo { f: DriverFence<()>, }impl Drop for Driver Foo {
pub fn drop(&mut self) { // Signals fence implicitly & automatically. self.f.signal(ECANCELED); } }Quoted 2 lines
Yeah, exactly that is still a bad idea :) See the HW eventually doesn't necessarily stop doing DMA if you unload the driver.
I know.
Quoted 1 line
So just signaling all fences with ECANCELED is not enough.
Yes. What I was trying to tell you as that you have no way to enforce this in Rust or C.
dma_fence_signal(f);free_driver_resources();
// Driver forgets to stop the hardware. Memory corruption occurs.You see? Neither your
BUG_ON()orWARN_ON()saves you if the driver does it like that, no matter the programming language. You cannot prevent it.
Quoted 8 lines
Quoted 6 lines
The only one who understands how the hardware works is the driver. The only one who can stop it is the driver.
In Rust, the place where a driver drops its JobQueue is the same place where it has to halt its GPU ring correctly. Enforcing this cannot be done by
dma_fencecode.I completely agree that we can't fix things from the
dma_fencecode, but it is our obligation to tell the user that something very very bad happened.
That's the point, we can't infer that "something very bad happened" from dma::DriverFence being dropped unsignaled.
In Rust we use the RAII pattern, and drivers structurally embed a dma::DriverFence within the struct that represents the corresponding hardware / firmware queue (e.g. through Jobqueue).
Consequently, if dma::DriverFence drops unsignaled it means that the destructor of the structure that embeds it (which does the required HW teardown) did run already.
Christian König
maintainerQuoted 46 lines · Danilo Krummrich
Quoted 33 lines · Christian König
Quoted 32 lines · Philipp Stanner
Quoted 12 lines · Christian König
Quoted 10 lines · Philipp Stanner
struct DriverFoo { f: DriverFence<()>, }impl Drop for Driver Foo {
pub fn drop(&mut self) { // Signals fence implicitly & automatically. self.f.signal(ECANCELED); } }Quoted 2 lines
Yeah, exactly that is still a bad idea :) See the HW eventually doesn't necessarily stop doing DMA if you unload the driver.
I know.
Quoted 1 line
So just signaling all fences with ECANCELED is not enough.
Yes. What I was trying to tell you as that you have no way to enforce this in Rust or C.
dma_fence_signal(f);free_driver_resources();
// Driver forgets to stop the hardware. Memory corruption occurs.You see? Neither your
BUG_ON()orWARN_ON()saves you if the driver does it like that, no matter the programming language. You cannot prevent it.Quoted 8 lines
Quoted 6 lines
The only one who understands how the hardware works is the driver. The only one who can stop it is the driver.
In Rust, the place where a driver drops its JobQueue is the same place where it has to halt its GPU ring correctly. Enforcing this cannot be done by
dma_fencecode.I completely agree that we can't fix things from the
dma_fencecode, but it is our obligation to tell the user that something very very bad happened.That's the point, we can't infer that "something very bad happened" from
dma::DriverFencebeing dropped unsignaled.
Quoted 7 lines
In Rust we use the RAII pattern, and drivers structurally embed a
dma::DriverFencewithin the struct that represents the corresponding hardware / firmware queue (e.g. through Jobqueue).Consequently, if
dma::DriverFencedrops unsignaled it means that the destructor of the structure that embeds it (which does the required HW teardown) did run already.
Oh WOW! The signaling state of the fence should *NEVER* be bound to some object lifetime.
That sounds like an extremely bad practice to me. Just imagine anybody grabbing an extra reference to the HW state object to inspect it for debugging and the fence not signaling because of this.
Signaling a DMA-buf should always be an explicit operation drivers do because of some HW event and not something which happens as a side effect cleanup because your object runs out of scope.
Regards,
Christian.
Gary Guo
reviewerQuoted 55 lines · Christian König
Quoted 46 lines · Danilo Krummrich
Quoted 33 lines · Christian König
Quoted 32 lines · Philipp Stanner
Quoted 12 lines · Christian König
Quoted 10 lines · Philipp Stanner
struct DriverFoo { f: DriverFence<()>, }impl Drop for Driver Foo {
pub fn drop(&mut self) { // Signals fence implicitly & automatically. self.f.signal(ECANCELED); } }Quoted 2 lines
Yeah, exactly that is still a bad idea :) See the HW eventually doesn't necessarily stop doing DMA if you unload the driver.
I know.
Quoted 1 line
So just signaling all fences with ECANCELED is not enough.
Yes. What I was trying to tell you as that you have no way to enforce this in Rust or C.
dma_fence_signal(f);free_driver_resources();
// Driver forgets to stop the hardware. Memory corruption occurs.You see? Neither your
BUG_ON()orWARN_ON()saves you if the driver does it like that, no matter the programming language. You cannot prevent it.Quoted 8 lines
Quoted 6 lines
The only one who understands how the hardware works is the driver. The only one who can stop it is the driver.
In Rust, the place where a driver drops its JobQueue is the same place where it has to halt its GPU ring correctly. Enforcing this cannot be done by
dma_fencecode.I completely agree that we can't fix things from the
dma_fencecode, but it is our obligation to tell the user that something very very bad happened.That's the point, we can't infer that "something very bad happened" from
dma::DriverFencebeing dropped unsignaled.Quoted 7 lines
In Rust we use the RAII pattern, and drivers structurally embed a
dma::DriverFencewithin the struct that represents the corresponding hardware / firmware queue (e.g. through Jobqueue).Consequently, if
dma::DriverFencedrops unsignaled it means that the destructor of the structure that embeds it (which does the required HW teardown) did run already.
Quoted 5 lines
Oh WOW! The signaling state of the fence should *NEVER* be bound to some object lifetime.
That sounds like an extremely bad practice to me. Just imagine anybody grabbing an extra reference to the HW state object to inspect it for debugging and the fence not signaling because of this.
We do not allow "grab a reference" to prevent HW state from going away in Rust. All device resources are tied to the lifetime of bus device being bound. When device is unbound (device going away, sysfs unbind, parent unbind, or whatever), device private data goes it with it and all device resources must cease to be alive before then.
If driver's fence callbacks can access HW state, it must also be teared down at at that point and cannot live past that.
Best,
Gary
Quoted 7 lines
Signaling a DMA-buf should always be an explicit operation drivers do because of some HW event and not something which happens as a side effect cleanup because your object runs out of scope.
Regards,
Christian.
Danilo Krummrich
maintainerQuoted 12 lines · Christian König
Quoted 11 lines · Danilo Krummrich
Quoted 2 lines · Christian König
I completely agree that we can't fix things from the
dma_fencecode, but it is our obligation to tell the user that something very very bad happened.That's the point, we can't infer that "something very bad happened" from
dma::DriverFencebeing dropped unsignaled.In Rust we use the RAII pattern, and drivers structurally embed a
dma::DriverFencewithin the struct that represents the corresponding hardware / firmware queue (e.g. through Jobqueue).
Quoted 6 lines
Quoted 3 lines
Consequently, if
dma::DriverFencedrops unsignaled it means that the destructor of the structure that embeds it (which does the required HW teardown) did run already.Oh WOW! The signaling state of the fence should *NEVER* be bound to some object lifetime.
Let's engage with the argument and understand things before throwing overgeneralized statements like this around please.
Realitiy is usually much more nuanced and what you say doesn't hold in all cases. The case described above (and clarified below) is one of them.
I assume the fundamental reason why you say things like "software object running out of scope is not an indicator that the HW stops doing something" is because the lifetime model in a lot of existing C DRM drivers is backwards.
Drivers often reference count representations of hardware resources and hardware states and hence keep them alive beyond what their actual lifetime should be -- even beyond driver unbind.
This causes lot's of issues and I'm looking into fixing this up, see my talk in
[1] and an example from Xe [2] for reference.
That said, in Rust we don't have this kind of ambiguity, lifetime and ownership of resources are well defined and the core infrastructure guides drivers accordingly.
Consequently, it is perfectly reasonable to assume that a driver object that owns device resources (such as DMA memory) will, in its destructor, tear down the hardware to not access any of the resources it owns anymore.
For instance, think about how your logic would apply to e.g. DMA coherent allocations.
Those are allocated with e.g. dma::Coherent::zeroed() and are freed on drop(). They are then owned by whatever object represents the lifetime of the HW using this buffer.
Following your logic, we'd need to conclude that a dma::Coherent allocation needs an explicit free() method independent of its destructor as well, and have the destructor print a warning and then call free() anyway.
Now, eventually drivers would just call free() from the destructor of the object owning the dma::Coherent allocation, so we wouldn't win anything.
More in general, in Rust the object taking ownership of a device resource is responsible to deal with the device related state that makes use of this resource regardless.
Insisting on an explicit method call is not improving anything in this model.
To get back to dma::DriverFence: This is slightly different, as there's two different signal states, but the fundamental guarantee is the same: If it is dropped, the HW has stopped accessing the resource that it guards already *by design*.
Signaling with an error on drop is arguably the correct thing to do, since if the operation would have been completed successfully before already, it would have been signaled already -- if it drops unsignaled it means the hardware was forcefully torn down (e.g. on driver unbind).
[1] https://youtu.be/3A2vGbc_rxs?t=668
[2] https://lore.kernel.org/all/[email protected]/
Quoted 3 lines
That sounds like an extremely bad practice to me. Just imagine anybody grabbing an extra reference to the HW state object to inspect it for debugging and the fence not signaling because of this.
That's not how it works.
Rust ownership rules will guarantee that the driver fence will be signaled with an error (if not signaled already) if the HW state object is destroyed (i.e. its destructor runs that tears down the HW). But that doesn't mean that the fence can't be signaled (and subsequently be dropped) independently through the completion of a job before that.
IOW, the HW state object (for instance) owns Jobqueue, Jobqueue owns jobs, jobs own driver fences.
If the HW state object drops (e.g. on driver unbind), the Jobqueue drops, which makes their jobs drop, which makes the driver fences drop.
At this point the driver fence either has been signaled already because the job completed before teardown (but wasn't dropped yet), or it will be signaled with an error in its drop(), which is fine since at this point the HW state's drop() is completed already and hence the HW is stopped.
Besides that, reference counting HW state objects is bad practice. HW state objects have a bounded lifetime, so there's no point in reference counting them.
Danilo Krummrich
maintainerQuoted 1 line · Christian König
See the HW eventually doesn't necessarily stop doing DMA if you unload the driver.
No, if you can't stop the underlying device you must not unload the driver and leak resources. Device resources (such as DMA mappings) are not safe against this and it can cause UB in many layers of kernel, e.g. IOMMU.
Quoted 3 lines
I even know about one case where a driver fully intentionally leaks memory on unload because the devs couldn't figure out which FW keeps writing into a buffer.
Again, please don't finish unloading then; everything else will cause UB.