gpu: nova-core: fb: make sure to unregister SysmemFlush on boot failure
Subsystems (4)
PATCH 1/1 gpu: nova-core: fb: make sure to unregister SysmemFlush on boot failure
Current Gpu::new will not unregister SysmemFlush if something fails after it is created, since it needs manual unregistering. Add a Drop implementation which will clean it up in that case. Maintain the manual unregister path because it can stay infallible, unlike the Drop path which depends on revocable access. In the case that Gpu::new fails the access is guaranteed to succeed, however.
--- drivers/gpu/nova-core/fb.rs | 29 ++++++++++++++++++++--------- drivers/gpu/nova-core/gpu.rs | 7 ++++++- 2 files changed, 26 insertions(+), 10 deletions(-)
drivers/gpu/nova-core/fb.rs +20 -9
@@ -7,6 +7,7 @@use kernel::{ device, + devres::Devres , dma::CoherentHandle , fmt, io::Io ,
@@ -16,7 +17,10 @@ Alignment, // }, sizes::*, - sync::aref::ARef, // + sync::{ + aref::ARef, + Arc, // + }, };use crate ::{
@@ -46,12 +50,14 @@/// Because of this, the sysmem flush memory page must be registered as early as possible during /// driver initialization, and before any falcon is reset. /// -/// Users are responsible for manually calling [`Self::unregister`] before dropping this object, -/// otherwise the GPU might still use it even after it has been freed. +/// Users should call [`Self::unregister`] before unloading to ensure unregistering is infallible. +/// [`Drop`] performs a best-effort fallback using revocable BAR access. pub (crate )struct SysmemFlush {/// Chipset we are operating on. chipset:Chipset , device:ARef <device::Device >, +/// MMIO mapping of PCI BAR 0. + bar:Arc <Devres <Bar0 >>,/// Keep the page alive as long as we need it. page:CoherentHandle , }
@@ -60,6 +66,7 @@ impl SysmemFlush {/// Allocate a memory page and register it as the sysmem flush page. pub (crate )fn register ( dev: &device::Device <device::Bound >, + devres_bar:Arc <Devres <Bar0 >>, bar: &Bar0 , chipset:Chipset , ) ->Result <Self > {
@@ -70,18 +77,17 @@ pub(crate) fn register(Ok (Self { chipset, device: dev.into (), + bar: devres_bar, page, }) }/// Unregister the managed sysmem flush page. -/// -/// In order to gracefully tear down the GPU, users must make sure to call this method before -/// dropping the object. pub(crate) fn unregister(&self, bar: &Bar0) { let hal = hal::fb_hal(self.chipset); + let registered_dma_handle = hal.read_sysmem_flush_page(bar); - if hal.read_sysmem_flush_page(bar) == self.page.dma_handle() { + if registered_dma_handle == self.page.dma_handle() { let _ = hal.write_sysmem_flush_page(bar, 0).inspect_err(|e| { dev_warn!( &self.device,
@@ -89,8 +95,7 @@ pub(crate) fn unregister(&self, bar: &Bar0) { e ) }); - }else { -// Another page has been registered after us for some reason - warn as this is a bug. + }else if registered_dma_handle !=0 {dev_warn! ( &self.device,"attempt to unregister a sysmem flush page that is not active\n"
@@ -99,6 +104,12 @@ pub(crate) fn unregister(&self, bar: &Bar0) { } } +impl Drop for SysmemFlush { +fn drop (&mut self) { +let _ = self.bar.try_access_with (|bar| self.unregister (bar)); + } +} +pub (crate )struct FbRange (Range <u64 >);impl FbRange {
drivers/gpu/nova-core/gpu.rs +6 -1
@@ -257,7 +257,12 @@ pub(crate) fn new<'a>( .inspect_err(|_| dev_err!(pdev, "GFW boot did not complete\n"))?; }, - sysmem_flush: SysmemFlush::register(pdev.as_ref(), bar, spec.chipset)?, + sysmem_flush: SysmemFlush::register( + pdev.as_ref(), + devres_bar.clone(), + bar, + spec.chipset, + )?, gsp_falcon: Falcon::new( pdev.as_ref(),
---
base-commit: a7a080bb4236ebe577b6776d940d1717912ff6dd
change-id: 20260409-fix-systemflush-de66dc90378a
Best regards,
Signature · 1 line
Eliot Courtney <[email protected]>
Quoted 6 lines · Eliot Courtney
Current
Gpu::newwill not unregister SysmemFlush if something fails after it is created, since it needs manual unregistering. Add aDropimplementation which will clean it up in that case. Maintain the manual unregister path because it can stay infallible, unlike the Drop path which depends on revocable access. In the case thatGpu::newfails the access is guaranteed to succeed, however.
Hi Eliot,
The code looks exactly correct to me. I just have some tiny commit message suggestions:
1. The subject line could be tightened up slightly, to:
gpu: nova-core: fb: unregister SysmemFlush on boot failure
2. And I'd like to rewrite the commit message body, to approximately this:
If Gpu::new fails after SysmemFlush::register succeeds, the registered sysmem flush page is never unregistered because SysmemFlush has no Drop impl and try_pin_init! only drops already-initialized fields on failure.
Add a Drop impl that unregisters through revocable BAR access, which covers the init-failure path. The manual unregister in Gpu::unbind is still needed because by the time Drop runs during normal teardown, devres has already revoked the BAR.
With that,
thanks,
Signature · 1 line
John Hubbard
Gary Guo
reviewerQuoted 96 lines · Eliot Courtney
Current
Gpu::newwill not unregister SysmemFlush if something fails after it is created, since it needs manual unregistering. Add aDropimplementation which will clean it up in that case. Maintain the manual unregister path because it can stay infallible, unlike the Drop path which depends on revocable access. In the case thatGpu::newfails the access is guaranteed to succeed, however.Fixes 6554ad65b589 ("gpu: nova-core: register sysmem flush page")Signed-off-by Eliot Courtney <[email protected]>--- drivers/gpu/nova-core/fb.rs | 29 ++++++++++++++++++++--------- drivers/gpu/nova-core/gpu.rs | 7 ++++++- 2 files changed, 26 insertions(+), 10 deletions(-)drivers/gpu/nova-core/fb.rs +14
-9@@ -7,6 +7,7 @@use kernel::{ device, + devres::Devres , dma::CoherentHandle , fmt, io::Io ,@@ -16,7 +17,10 @@ Alignment, // }, sizes::*, - sync::aref::ARef, // + sync::{ + aref::ARef, + Arc, // + }, };use crate ::{@@ -46,12 +50,14 @@/// Because of this, the sysmem flush memory page must be registered as early as possible during /// driver initialization, and before any falcon is reset. /// -/// Users are responsible for manually calling [`Self::unregister`] before dropping this object, -/// otherwise the GPU might still use it even after it has been freed. +/// Users should call [`Self::unregister`] before unloading to ensure unregistering is infallible. +/// [`Drop`] performs a best-effort fallback using revocable BAR access. pub (crate )struct SysmemFlush {/// Chipset we are operating on. chipset:Chipset , device:ARef <device::Device >, +/// MMIO mapping of PCI BAR 0. + bar:Arc <Devres <Bar0 >>,/// Keep the page alive as long as we need it. page:CoherentHandle , }@@ -60,6 +66,7 @@ impl SysmemFlush {/// Allocate a memory page and register it as the sysmem flush page. pub (crate )fn register ( dev: &device::Device <device::Bound >, + devres_bar:Arc <Devres <Bar0 >>, bar: &Bar0 , chipset:Chipset , ) ->Result <Self > {@@ -70,18 +77,17 @@ pub(crate) fn register(Ok (Self { chipset, device: dev.into (), + bar: devres_bar, page, }) }/// Unregister the managed sysmem flush page. -/// -/// In order to gracefully tear down the GPU, users must make sure to call this method before -/// dropping the object. pub(crate) fn unregister(&self, bar: &Bar0) { let hal = hal::fb_hal(self.chipset); + let registered_dma_handle = hal.read_sysmem_flush_page(bar); - if hal.read_sysmem_flush_page(bar) == self.page.dma_handle() { + if registered_dma_handle == self.page.dma_handle() { let _ = hal.write_sysmem_flush_page(bar, 0).inspect_err(|e| { dev_warn!( &self.device,@@ -89,8 +95,7 @@ pub(crate) fn unregister(&self, bar: &Bar0) { e ) }); - }else { -// Another page has been registered after us for some reason - warn as this is a bug. + }else if registered_dma_handle !=0 {dev_warn! ( &self.device,"attempt to unregister a sysmem flush page that is not active\n"
drivers/gpu/nova-core/fb.rs pub(crate) fn unregister(&self, bar: &Bar0) { · line 109
Quoted 6 lines
drivers/gpu/nova-core/fb.rs +3
-0@@ -99,6 +104,12 @@ pub(crate) fn unregister(&self, bar: &Bar0) { } } +impl Drop for SysmemFlush { +fn drop (&mut self) { +let _ = self.bar.try_access_with (|bar| self.unregister (bar));
I feel that this is the wrong solution to the problem.
The thing we want is to *ensure* that SysmemFlush Drop is called with device still being bound.
It's not yet fully clear to me how we'd want to guarantee that, but one API that might make sense is to create a DevRes API that allows you to reference an existing DevRes and have driver-core making sure that the tear down happens in reverse order. So inside the Drop the bar can still be unconditionally access.
Best,
Gary
Quoted 32 lines
drivers/gpu/nova-core/fb.rs +3
-0+ } +} +pub (crate )struct FbRange (Range <u64 >);impl FbRange {drivers/gpu/nova-core/gpu.rs +6
-2@@ -257,7 +257,12 @@ pub(crate) fn new<'a>( .inspect_err(|_| dev_err!(pdev, "GFW boot did not complete\n"))?; }, - sysmem_flush: SysmemFlush::register(pdev.as_ref(), bar, spec.chipset)?, + sysmem_flush: SysmemFlush::register( + pdev.as_ref(), + devres_bar.clone(), + bar, + spec.chipset, + )?, gsp_falcon: Falcon::new( pdev.as_ref(), ---base-commit:
a7a080bb4236ebe577b6776d940d1717912ff6dd
change-id: 20260409-fix-systemflush-de66dc90378aBest regards,
Signature · 1 line
Eliot Courtney <[email protected]>
Danilo Krummrich
maintainerQuoted 5 lines · Gary Guo
drivers/gpu/nova-core/fb.rs pub(crate) fn unregister(&self, bar: &Bar0) { · line 109
Quoted 3 lines
drivers/gpu/nova-core/fb.rs +3
-0+impl Drop for SysmemFlush { +fn drop (&mut self) { +let _ = self.bar.try_access_with (|bar| self.unregister (bar));I feel that this is the wrong solution to the problem.
Yeah, it is pretty fragile as it relies on outer implementation details, such as the fact that SysmemFlush is part of the device private data and hence try_access_with() will fail when not dropped in an unwind path.
Quoted 8 lines
The thing we want is to *ensure* that
SysmemFlushDrop is called with device still being bound.It's not yet fully clear to me how we'd want to guarantee that, but one API that might make sense is to create a DevRes API that allows you to reference an existing
DevResand have driver-core making sure that the tear down happens in reverse order. So inside theDropthebarcan still be unconditionally access.
Yes, I have something like this on my list of things I want to look into for while (my list entry calls it DevresChain).
I want to leverage the internal reference count of Devres for this, which is not exactly straight forward, but should be possible.
I will prioritize this and have a look.
Thanks,
Danilo
Quoted 20 lines · Danilo Krummrich
Quoted 5 lines · Gary Guo
drivers/gpu/nova-core/fb.rs pub(crate) fn unregister(&self, bar: &Bar0) { · line 109
Quoted 3 lines
drivers/gpu/nova-core/fb.rs +3
-0+impl Drop for SysmemFlush { +fn drop (&mut self) { +let _ = self.bar.try_access_with (|bar| self.unregister (bar));I feel that this is the wrong solution to the problem.
Yeah, it is pretty fragile as it relies on outer implementation details, such as the fact that SysmemFlush is part of the device private data and hence
try_access_with()will fail when not dropped in an unwind path.Quoted 8 lines
The thing we want is to *ensure* that
SysmemFlushDrop is called with device still being bound.It's not yet fully clear to me how we'd want to guarantee that, but one API that might make sense is to create a DevRes API that allows you to reference an existing
DevResand have driver-core making sure that the tear down happens in reverse order. So inside theDropthebarcan still be unconditionally access.
Quoted 7 lines
Yes, I have something like this on my list of things I want to look into for while (my list entry calls it DevresChain).
I want to leverage the internal reference count of Devres for this, which is not exactly straight forward, but should be possible.
I will prioritize this and have a look.
Yeah, I agree that this patch is the wrong long term solution. If we don't have the infra for the proper way to do it soon, it might be worth taking it anyway since it adds minimal complexity and fixes a real issue.
I had a brief look into the Devres chain stuff that exists on the C side and it looks like it doesn't provide any actual guarantees about the lifetime (it seems possible to delete from the middle of the chain, so descendants can't assume ancestors exist in general, AFAICT), so the the translation of that into lifetimes in rust might get interesting
thanks
Quoted 3 lines
Thanks,
Danilo
Danilo Krummrich
maintainerQuoted 4 lines · Eliot Courtney
Yeah, I agree that this patch is the wrong long term solution. If we don't have the infra for the proper way to do it soon, it might be worth taking it anyway since it adds minimal complexity and fixes a real issue.
Yeah, let's see how it goes the next weeks.
Quoted 5 lines
I had a brief look into the Devres chain stuff that exists on the C side and it looks like it doesn't provide any actual guarantees about the lifetime (it seems possible to delete from the middle of the chain, so descendants can't assume ancestors exist in general, AFAICT), so the the translation of that into lifetimes in rust might get interesting
There is no such API that does what we want here on the C side. In C it simply is the responsibility of drivers to take care of this. So, I'm not sure what you are referring to.
In case you are talking about devres groups, they have a different purpose. They are used in cases where you e.g. have some middle layer API between drivers and a subsystem, so the middle layer can acquire resources on the driver's behalf and manage them. If the middle layer fails on optional features, it can simply release the group of resources acquired for the purpose of the optional feature.
I.e. devres groups are simply markers for a certain range within the list of devres nodes of a specific device.