Skip to contentSkip to conversation

rust: DmaFence: Add better warning through Device reference

PATCH In review Philipp Stanner 5 participants 13 messages Sep 25 – Sep 28

Subsystems (4)
lore
Share & tools
lore

Philipp Stanner

author maintainer
#0 lore raw

PATCH 1/1 rust: DmaFence: Add better warning through Device reference

FenceContext::drop() contains a warning print that warns about possible memory corruptions if there are forgotten fences. However, precisely speaking, a forgotten fence is undefined behavior.

Moreover, forgotten fences hint at a severe design problem in the driver. In this context, the used pr_err!() does not provide very useful text output.

Replace the warning print with a dev_warn!(). To do so, have the FenceContext carry a reference to a Device, protected by the already present lifetime.

---
 rust/kernel/dma_buf/dma_fence.rs | 25 +++++++++++++++++++------
 1 file changed, 19 insertions(+), 6 deletions(-)
rust/kernel/dma_buf/dma_fence.rs +19 -6
index 18a43e1bb442..58ba678e91b5 100644
@@ -32,6 +32,10 @@
 };
 
 use kernel::{
+    device::{
+        Device,
+        Normal, //
+    },
     str::CString,
     sync::{
         aref::{
@@ -59,7 +63,7 @@ pub trait FenceContextOps {
 /// with each other, providing each with raising sequence numbers and a common
 /// identifier.
 #[pin_data(PinnedDrop)]
-pub struct FenceContext<T: FenceContextOps + Send + Sync> {
+pub struct FenceContext<'a, T: FenceContextOps + Send + Sync> {
     /// The fence context number.
     nr: u64,
     /// The sequence number for the next fence created.
@@ -81,12 +85,14 @@ pub struct FenceContext<T: FenceContextOps + Send + Sync> {
     // the lifetime which intends to enforce that all fences disappear before
     // their context.
     nr_of_unsignaled_fences: Atomic<usize>,
+    /// The device this fence context is associated with.
+    dev: &'a Device<Normal>,
     /// The user's data.
     #[pin]
     data: T,
 }
 
-impl<'a, T: Send + Sync + FenceContextOps> FenceContext<T> {
+impl<'a, T: Send + Sync + FenceContextOps + 'a> FenceContext<'a, T> {
     // This can later be extended as a vtable in case other parties need support
     // for the more "exotic" callbacks.
     const OPS: bindings::dma_fence_ops = bindings::dma_fence_ops {
@@ -106,6 +112,7 @@ pub fn new<E>(
         initial_seqno: u64,
         driver_name: &CStr,
         timeline_name: &CStr,
+        dev: &'a Device<Normal>,
         data: impl PinInit<T, E>,
     ) -> impl PinInit<Self, Error>
     where
@@ -122,6 +129,7 @@ pub fn new<E>(
             driver_name: driver_name?,
             timeline_name: timeline_name?,
             nr_of_unsignaled_fences: Atomic::new(0),
+            dev,
             data <- data,
         })
     }
@@ -211,7 +219,7 @@ unsafe fn from_raw_fence(ptr: *mut bindings::dma_fence) -> &'a Self {
 }
 
 #[pinned_drop]
-impl<T: FenceContextOps + Send + Sync> PinnedDrop for FenceContext<T> {
+impl<T: FenceContextOps + Send + Sync> PinnedDrop for FenceContext<'_, T> {
     fn drop(self: Pin<&mut Self>) {
         // Fence ops callbacks can be called on unsignaled fences. Since these
         // callbacks can access the fence context and its data, it needs to be
@@ -596,7 +604,7 @@ struct DriverFenceData<'a, T: Send + Sync + FenceContextOps> {
     /// Callback head for dropping this in a deferred manner through RCU.
     rcu_head: bindings::callback_head,
     /// Reference to access the FenceContext.
-    fctx: &'a FenceContext<T>,
+    fctx: &'a FenceContext<'a, T>,
     /// The API user's data. It is essential that the data only performs
     /// operations legal in atomic context in its [`Drop`] implementation.
     #[pin]
@@ -641,6 +649,7 @@ struct DriverFenceData<'a, T: Send + Sync + FenceContextOps> {
 ///
 /// ```
 /// use kernel::{
+///     device::Normal,
 ///     dma_buf::{
 ///         DriverFence,
 ///         FenceContext,
@@ -648,6 +657,7 @@ struct DriverFenceData<'a, T: Send + Sync + FenceContextOps> {
 ///         FenceCallback,
 ///         FenceCallbackRegistration,
 ///     },
+///     faux,
 ///     str::CString,
 ///     sync::aref::ARef, //
 /// };
@@ -676,9 +686,10 @@ struct DriverFenceData<'a, T: Send + Sync + FenceContextOps> {
 ///
 /// let fctx_data = FenceContextData::new();
 ///
+/// let reg = faux::Registration::new(c"DmaFence-KUnit-Test", None)?;
 ///
 /// let mut fctx = KBox::pin_init(
-///     FenceContext::new(0, c"dummy_driver", c"dummy_timeline", fctx_data),
+///     FenceContext::new(0, c"dummy_driver", c"dummy_timeline", reg.as_ref().as_ref(), fctx_data),
 ///     GFP_KERNEL
 /// )?;
 ///
@@ -934,7 +945,9 @@ fn drop(&mut self) {
         // SAFETY: `guard` is valid until the `call_rcu()` below.
         let signaled: bool = unsafe { bindings::dma_fence_test_signaled_flag(guard.as_raw()) };
         if !signaled {
-            pr_err!("DriverFence drops unsignaled. Danger of memory corruption!\n");
+            // SAFETY: `data` is valid because `self` is valid.
+            let dev = unsafe { self.data.as_ref().fctx.dev };
+            dev_warn!(dev, "DriverFence drops unsignaled.\n");
             // SAFETY: `guard` is valid until the `call_rcu()` below. The fence
             // must not have been signaled yet, which we check directly above.
             unsafe { bindings::dma_fence_set_error(guard.as_raw(), ECANCELED.to_errno()) };

base-commit: 896ed083362758b33c49a1b5e5a3423c5814d87e

Signature · 1 line
2.55.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] dma_fence: commit message incorrectly claims change is in FenceContext::drop() --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] dma_fence: commit message incorrectly claims change is in FenceContext::drop() --

commit 4a645fda82950d47b5bed116917aa2253db17d1f
Author: Philipp Stanner <[email protected]>

rust: DmaFence: Add better warning through Device reference

This commit replaces a pr_err warning about forgotten fences with a more informative dev_warn. It also adds a Device reference to FenceContext to support the dev_warn output.

Quoted 3 lines

FenceContext::drop() contains a warning print that warns about possible memory corruptions if there are forgotten fences. However, precisely speaking, a forgotten fence is undefined behavior.

[Severity: Medium]
Is this commit message accurate? The text states the warning print being replaced is located in FenceContext::drop(), but the diff shows the modification actually takes place in the Drop implementation for DriverFence:

rust/kernel/dma_buf/dma_fence.rs:drop() {
    ...
    let signaled: bool = unsafe { bindings::dma_fence_test_signaled_flag(guard.as_raw()) };
    if !signaled {
        // SAFETY: data is valid because self is valid.
        let dev = unsafe { self.data.as_ref().fctx.dev };
        dev_warn!(dev, "DriverFence drops unsignaled.\n");
    ...
}

FenceContext::drop() does not contain this warning print. Could the commit message be updated to correctly state that the warning is in DriverFence?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
reviewer
#2 lore raw
In reply to Philipp Stanner
Quoted 11 lines · Philipp Stanner

FenceContext::drop() contains a warning print that warns about possible memory corruptions if there are forgotten fences. However, precisely speaking, a forgotten fence is undefined behavior.

Moreover, forgotten fences hint at a severe design problem in the driver. In this context, the used pr_err!() does not provide very useful text output.

Replace the warning print with a dev_warn!(). To do so, have the FenceContext carry a reference to a Device, protected by the already present lifetime.

Add a lifetime just to do this a print doesn't sound ideal. You could use ARef<Device> instead?

Best,
Gary

Quoted 7 lines
---
 rust/kernel/dma_buf/dma_fence.rs | 25 +++++++++++++++++++------
 1 file changed, 19 insertions(+), 6 deletions(-)
maintainer
#3 lore raw
Quoted 7 lines · Gary Guo
Quoted 3 lines · Philipp Stanner

Replace the warning print with a dev_warn!(). To do so, have the FenceContext carry a reference to a Device, protected by the already present lifetime.

Add a lifetime just to do this a print doesn't sound ideal. You could use ARef<Device> instead?

Yes, that's what I recommend in general.

Another thing to consider for the DMA fence context specifically, there's two kinds of users. The absolute majority are drivers; there are very few non-driver ones (e.g. drm_crtc), which will remain to be C code for at least quite a while.

I think it wouldn't be unreasonable to keep the API driver centric, and in this case it also wouldn't be unreasonable to add the lifetime, as the T will most likely carry it anyway, and even if not it wouldn't make sense for the fence context to outlive driver unbind.

That said, we can also use WARN_ON() instead, which avoids the device dependency to begin with and still provides enough information to find the "offender".

But as I mentioned previously, I don't consider this that bad of a condition in the first place. Signaling with ECANCELED on drop() seems perfectly reasonable:

When a driver does a teardown of the channel (or more generically the execution context) it will follow the RAII pattern, so it will be very natural to just drop the Jobqueue, which will drop all jobs and hence all DriverFence objects.

IOW, driver will likely invent a new type that does the same thing on drop, just without the warning. :)

#4 lore raw
Quoted 10 lines · Danilo Krummrich
Quoted 7 lines · Gary Guo
Quoted 3 lines · Philipp Stanner

Replace the warning print with a dev_warn!(). To do so, have the FenceContext carry a reference to a Device, protected by the already present lifetime.

Add a lifetime just to do this a print doesn't sound ideal. You could use ARef<Device> instead?

Yes, that's what I recommend in general.

Weren't you super opposed to refcounting wherever it's avoidable?

[…]

Quoted 2 lines

That said, we can also use WARN_ON() instead, which avoids the device dependency to begin with and still provides enough information to find the "offender".

WARN_ON() is fine by me.

Quoted 10 lines

But as I mentioned previously, I don't consider this that bad of a condition in the first place. Signaling with ECANCELED on drop() seems perfectly reasonable:

When a driver does a teardown of the channel (or more generically the execution context) it will follow the RAII pattern, so it will be very natural to just drop the Jobqueue, which will drop all jobs and hence all DriverFence objects.

IOW, driver will likely invent a new type that does the same thing on drop, just without the warning. :)

The driver can avoid dropping half-forgotten stuff by calling jobqueue.complete_all_jobs(ECANCELED) immediately before dropping, which allows us for having the warning without false-positives.

P.

maintainer
#5 lore raw
Quoted 3 lines · Philipp Stanner

The driver can avoid dropping half-forgotten stuff by calling jobqueue.complete_all_jobs(ECANCELED) immediately before dropping, which allows us for having the warning without false-positives.

This would just move it up a layer from inventing a new type around DriverFence to invent a new type around Jobqueue which does that instead.

But the Jobqueue won't have any procedural teardown, it has to teardown stuff in drop() anyway, which includes dropping jobs already.

What I'm saying is that there's not much value in DriverFence::drop() having this warning in the first place, which then either the driver or the jobqueue has to work around on teardown.

#6 lore raw
Quoted 8 lines · Danilo Krummrich
Quoted 3 lines · Philipp Stanner

The driver can avoid dropping half-forgotten stuff by calling jobqueue.complete_all_jobs(ECANCELED) immediately before dropping, which allows us for having the warning without false-positives.

This would just move it up a layer from inventing a new type around DriverFence to invent a new type around Jobqueue which does that instead.

Quoted 6 lines

But the Jobqueue won't have any procedural teardown, it has to teardown stuff in drop() anyway, which includes dropping jobs already.

What I'm saying is that there's not much value in DriverFence::drop() having this warning in the first place, which then either the driver or the jobqueue has to work around on teardown.

I guess we agree that it would be a horrible bug if there's still a command buffer running on the GPU that can access memory which might have been freed once the associated fence signaled.

So I suppose what you are saying is more: there is not much value in the case of *JobQueue*, basically because all the jobs live inside of it anyways.

So I suppose we agree that a warning is fine. It won't fire in JQ anyways, but might benefit others.

P.

maintainer
#7 lore raw
Quoted 12 lines · Philipp Stanner
Quoted 11 lines · Danilo Krummrich
Quoted 3 lines · Philipp Stanner

The driver can avoid dropping half-forgotten stuff by calling jobqueue.complete_all_jobs(ECANCELED) immediately before dropping, which allows us for having the warning without false-positives.

This would just move it up a layer from inventing a new type around DriverFence to invent a new type around Jobqueue which does that instead.

But the Jobqueue won't have any procedural teardown, it has to teardown stuff in drop() anyway, which includes dropping jobs already.

Quoted 7 lines
Quoted 3 lines

What I'm saying is that there's not much value in DriverFence::drop() having this warning in the first place, which then either the driver or the jobqueue has to work around on teardown.

I guess we agree that it would be a horrible bug if there's still a command buffer running on the GPU that can access memory which might have been freed once the associated fence signaled.

Sure, but that's unrelated.

Quoted 3 lines

So I suppose what you are saying is more: there is not much value in the case of *JobQueue*, basically because all the jobs live inside of it anyways.

No, I'm saying there is not much value in general. Whatever thing owns the DriverFence has to represent the "device access" of some resource, which already naturally establishes the relationship.

IOW, whatever thing owns a DriverFence is also the thing that stops the hardware in its own drop() implementation; anything else would be rather questionable.

Quoted 2 lines

So I suppose we agree that a warning is fine. It won't fire in JQ anyways, but might benefit others.

What scenario are you thinking of?

#8 lore raw
Quoted 15 lines · Danilo Krummrich
Quoted 3 lines · Philipp Stanner

I guess we agree that it would be a horrible bug if there's still a command buffer running on the GPU that can access memory which might have been freed once the associated fence signaled.

Sure, but that's unrelated.

Quoted 3 lines

So I suppose what you are saying is more: there is not much value in the case of *JobQueue*, basically because all the jobs live inside of it anyways.

No, I'm saying there is not much value in general. Whatever thing owns the DriverFence has to represent the "device access" of some resource, which already naturally establishes the relationship.

Quoted 7 lines

IOW, whatever thing owns a DriverFence is also the thing that stops the hardware in its own drop() implementation; anything else would be rather questionable.

Quoted 2 lines

So I suppose we agree that a warning is fine. It won't fire in JQ anyways, but might benefit others.

What scenario are you thinking of?

Drivers doing "rather questionable" things, like we've seen a great many times already ;)

Note that the dma_fence backend fires a WARN_ON if a fence is freed unsignaled, too, for the same reason.

Life finds a way.

P.

maintainer
#9 lore raw
Quoted 24 lines · Philipp Stanner
Quoted 15 lines · Danilo Krummrich
Quoted 3 lines · Philipp Stanner

I guess we agree that it would be a horrible bug if there's still a command buffer running on the GPU that can access memory which might have been freed once the associated fence signaled.

Sure, but that's unrelated.

Quoted 3 lines

So I suppose what you are saying is more: there is not much value in the case of *JobQueue*, basically because all the jobs live inside of it anyways.

No, I'm saying there is not much value in general. Whatever thing owns the DriverFence has to represent the "device access" of some resource, which already naturally establishes the relationship.

Quoted 7 lines

IOW, whatever thing owns a DriverFence is also the thing that stops the hardware in its own drop() implementation; anything else would be rather questionable.

Quoted 2 lines

So I suppose we agree that a warning is fine. It won't fire in JQ anyways, but might benefit others.

What scenario are you thinking of?

Quoted 7 lines

Drivers doing "rather questionable" things, like we've seen a great many times already ;)

Note that the dma_fence backend fires a WARN_ON if a fence is freed unsignaled, too, for the same reason.

Life finds a way.

I'd rather you engage with the arguments I made above and give a concrete example of how it "might benefit others", instead of resorting to know-it-all platitudes.

The comparison with C does not address my point about ownership: C relies on explicit cleanup, whereas the Rust design I described ties cleanup to ownership through RAII.

Again, please address the arguments I've already made.

#10 lore raw
Quoted 9 lines · Danilo Krummrich
Quoted 8 lines · Philipp Stanner
Quoted 4 lines
Quoted 2 lines

So I suppose we agree that a warning is fine. It won't fire in JQ anyways, but might benefit others.

What scenario are you thinking of?

Drivers doing "rather questionable" things, like we've seen a great many times already ;)

Quoted 8 lines
Quoted 4 lines

Note that the dma_fence backend fires a WARN_ON if a fence is freed unsignaled, too, for the same reason.

Life finds a way.

I'd rather you engage with the arguments I made above and give a concrete example of how it "might benefit others", instead of resorting to know-it-all platitudes.

Stating that I cannot know nor conceive all possible patterns and misbehaviors is quite literally me acknowledging that I do *not* "know it all".

With this patch I was just trying to accommodate your post-merge request for replacing pr_ with dev_err() or WARN_ON. If you think neither is actually necessary, that's also fine by me.

P.

maintainer
#11 lore raw
Quoted 15 lines · Philipp Stanner
Quoted 14 lines · Danilo Krummrich
Quoted 12 lines · Philipp Stanner
Quoted 4 lines
Quoted 2 lines

So I suppose we agree that a warning is fine. It won't fire in JQ anyways, but might benefit others.

What scenario are you thinking of?

Drivers doing "rather questionable" things, like we've seen a great many times already ;)

Note that the dma_fence backend fires a WARN_ON if a fence is freed unsignaled, too, for the same reason.

Life finds a way.

Quoted 7 lines
Quoted 3 lines

I'd rather you engage with the arguments I made above and give a concrete example of how it "might benefit others", instead of resorting to know-it-all platitudes.

Stating that I cannot know nor conceive all possible patterns and misbehaviors is quite literally me acknowledging that I do *not* "know it all".

I mean, you expressed a concern about a fence being signaled before the hardware has been torn down accordingly. I provided arguments why I don't see that the concern holds and then you mentioned that the warning "might benefit others".

To me this sounded as if you had concrete scenarios in mind, which doesn't seem to be the case. If that's correct, and given that there was no reply to my arguments, it seems we can just remove it.

Maybe to further explain my reasoning:

The concern was that it could happen that the DriverFence is dropped while the hardware still utilizes the memory "protected" by the fence. As mentioned, to me this is not a concern because of the programming model we have in Rust. Take this analogous DMA memory example:

	struct Channel<'a> {
	    shared: dma::Coherent<'a, Data>,
	}
	
	impl Drop for Channel<'_> {
	    fn drop(&mut self) {
	        self.stop_hardware();
	        shared.free();
	    }
	}

We could use the exact same argument to require people to explicitly call free() on a dma::Coherent allocation as the underlying hardware (represented by the channel) could still use the dma::Coherent allocation if we "drop it accidentally".

But as you can see, the ownership is already properly expressed by Rust, the Channel owns the dma::Coherent allocation, so the memory can just be freed in Coherent::drop().

The same applies to DriverFence, the thing that takes ownership is the thing that operates the hardware.

Quoted 3 lines

With this patch I was just trying to accommodate your post-merge request for replacing pr_ with dev_err() or WARN_ON. If you think neither is actually necessary, that's also fine by me.

I did arrive at the conclusion to discuss whether we can't just get rid of the warning entirely in my first reply [1], which you refer to, already.

[1] https://lore.kernel.org/all/[email protected]/

maintainer
#12 lore raw
In reply to Philipp Stanner
Quoted 7 lines · Philipp Stanner
Quoted 6 lines · Danilo Krummrich
Quoted 5 lines · Gary Guo
Quoted 3 lines · Philipp Stanner

Replace the warning print with a dev_warn!(). To do so, have the FenceContext carry a reference to a Device, protected by the already present lifetime.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Add a lifetime just to do this a print doesn't sound ideal. You could use ARef<Device> instead?

Yes, that's what I recommend in general.

Weren't you super opposed to refcounting wherever it's avoidable?

Well, a device by itself is already reference counted (which is fine). The lifetime constraint does not come from the lifetime of a struct device, it comes from it being bound to a driver for bus devices; for class devices the lifetime constraint comes from the device being registered.

I'm opposed to refcount stuff that has a bounded lifetime that is defined by some scope it must not outlive.