Add a Rust GPUVM abstraction
Subsystems (5)
Daniel Almeida
reviewercover Add a Rust GPUVM abstraction
Changes from v0:
This version changes how LockedGpuVm operates. The previous code assumed
that the interval tree would be protected if all the associated GEM's resvs were locked, but this is completely unrelated. In fact, this initial version only aims to support the core VA range management feature of GPUVM, and not any of the "convenience" functions like "exec_lock()" and friends, so this function was removed accordingly.
LockedGpuVm is now produced by a call to GpuVm::lock(). This takes a generic
guard type to signal that the driver-specific locks have been acquired in order to protect the VMs tree, and hence its view of the address space. This approach is somewhat similar to CondVar, but also incurs in the same pitfall: it is up to the caller to give meaning to the guard by ensuring that it actually protects against concurrent access to the VM. Maybe this is a good candidate to the upcoming "Pitfall" docs section?
Note that LockedGpuVm::obj was removed. I'm not really sure why this field
was there in the first place, but callers should indicate the object through the sm_map() and sm_unmap() APIs instead.
I am not sure why GpuVm::inner uses UnsafeCell, though. I did not touch this
so that we can first discuss whether UnsafeCell is really needed.
The docs were also updated. Care was taken to reuse the C documentation as
much as possible.
Itemized changes:
- Rebased on top of nova-drm
- Use srctree in the docs
- Add docs as appropriate and remove #[allow(missing_docs)]
- Remove impl DriverGpuVa for (). Drivers can to that themselves, if they want.
- Added #[inline] to getters, as Rust cannot inline across crates otherwise (unless this changed recently?)
- Exposed drm_gpuva_op_unmap::keep.
- Removed pub(super) from unsafe fns meant to be called from the C code.
- Added "# Safety" blocks to the above.
- Removed exec_lock_gem_object().
- Removed exec_lock()
- Removed LockedGpuVm::vm_exec. This initial version does not support exec calls at all.
- By the same token, removed LockedGpuVm::drop()
- Removed LockedGpuVm::obj. This object has to be passed explicitly.
---
Asahi Lina (2):
rust: helpers: Add bindings/wrappers for dma_resv
rust: drm: Add GPUVM abstraction
rust/bindings/bindings_helper.h | 5 +- rust/helpers/dma-resv.c | 13 + rust/helpers/drm_gpuvm.c | 29 ++ rust/helpers/helpers.c | 2 + rust/kernel/drm/gpuvm.rs | 790 ++++++++++++++++++++++++++++++++++++++++ rust/kernel/drm/mod.rs | 2 + 6 files changed, 840 insertions(+), 1 deletion(-)
---
base-commit: e7de41cc4b01dd5500a0c2533c64bdb3f5360567
change-id: 20250320-gpuvm-29d3e0726f34
Best regards,
Signature · 1 line
Daniel Almeida <[email protected]>
Daniel Almeida
reviewerPATCH 1/2 rust: helpers: Add bindings/wrappers for dma_resv
From: Asahi Lina <[email protected]>
This is just for basic usage in the DRM shmem abstractions for implied locking, not intended as a full DMA Reservation abstraction yet.
--- rust/bindings/bindings_helper.h | 4 +++- rust/helpers/dma-resv.c | 13 +++++++++++++ rust/helpers/helpers.c | 1 + 3 files changed, 17 insertions(+), 1 deletion(-)
rust/bindings/bindings_helper.h +3 -1
@@ -18,6 +18,7 @@#include <linux/blkdev.h> #include <linux/cred.h> #include <linux/device/faux.h> +#include <linux/dma-resv.h> #include <linux/errname.h> #include <linux/ethtool.h> #include <linux/file.h>
@@ -53,5 +54,6 @@ const gfp_t RUST_CONST_HELPER_GFP_NOWAIT = GFP_NOWAIT;const gfp_t RUST_CONST_HELPER___GFP_ZERO = __GFP_ZERO;const gfp_t RUST_CONST_HELPER___GFP_HIGHMEM = ___GFP_HIGHMEM;const gfp_t RUST_CONST_HELPER___GFP_NOWARN = ___GFP_NOWARN; -const blk_features_t RUST_CONST_HELPER_BLK_FEAT_ROTATIONAL =BLK_FEAT_ROTATIONAL ; +const blk_features_t RUST_CONST_HELPER_BLK_FEAT_ROTATIONAL = +BLK_FEAT_ROTATIONAL ;const fop_flags_t RUST_CONST_HELPER_FOP_UNSIGNED_OFFSET =FOP_UNSIGNED_OFFSET ;
rust/helpers/dma-resv.c new file +13 -0
@@ -0,0 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <linux/dma-resv.h> + +int rust_helper_dma_resv_lock (struct dma_resv *obj,struct ww_acquire_ctx *ctx) +{ +return dma_resv_lock (obj, ctx); +} + +void rust_helper_dma_resv_unlock (struct dma_resv *obj) +{ +dma_resv_unlock (obj); +}
rust/helpers/helpers.c +1 -0
@@ -14,6 +14,7 @@#include "build_bug.c" #include "cred.c" #include "device.c" +#include "dma-resv.c" #include "drm.c" #include "err.c" #include "fs.c"
Signature · 1 line
2.48.1
Re: [PATCH 1/2] rust: helpers: Add bindings/wrappers for dma_resv
rust/bindings/bindings_helper.h const gfp_t RUST_CONST_HELPER_GFP_NOWAIT = GFP_NOWAIT; · line 58
Quoted 4 lines · Daniel Almeida
rust/bindings/bindings_helper.h +2
-1-const blk_features_t RUST_CONST_HELPER_BLK_FEAT_ROTATIONAL =BLK_FEAT_ROTATIONAL ; +const blk_features_t RUST_CONST_HELPER_BLK_FEAT_ROTATIONAL = +BLK_FEAT_ROTATIONAL ;
Spurious change?
Ah, this is probably the result of applying clang-format, which is something that perhaps we should consider, but separately... :)
Cheers,
Miguel
Daniel Almeida
reviewerPATCH 2/2 rust: drm: Add GPUVM abstraction
In reply to Daniel AlmeidaFrom: Asahi Lina <[email protected]>
Add a GPUVM abstraction to be used by Rust GPU drivers.
GPUVM keeps track of a GPU's virtual address (VA) space and manages the corresponding virtual mappings represented by "GPU VA" objects. It also keeps track of the mapping's backing GEM buffers.
This initial version only support synchronous operations. In other words, we do not support the use case where the operations are pre-built and handed over to the driver.
We also do not support using a driver-specific lock in order to serialize the gpuva list for a given GEM object. This has to be the GEM object resv, which is the default behavior in C.
Similarly to the C code, locking is left to the driver. This means that the driver should make sure that the VM's interval tree is protected against concurrent access. In Rust, we encode this requirement by requiring a generic Guard type in GpuVm::lock(), which is a similar approach as the one taken by CondVar.
It is up to drivers to make sure that the guard indeed provides the required locking. Any operations that modifies the interval tree is only available after the VM has been locked as per above.
--- rust/bindings/bindings_helper.h | 1 + rust/helpers/drm_gpuvm.c | 29 ++ rust/helpers/helpers.c | 1 + rust/kernel/drm/gpuvm.rs | 790 ++++++++++++++++++++++++++++++++++++++++ rust/kernel/drm/mod.rs | 2 + 5 files changed, 823 insertions(+)
rust/bindings/bindings_helper.h +1 -0
@@ -10,6 +10,7 @@#include <drm/drm_drv.h> #include <drm/drm_file.h> #include <drm/drm_gem.h> +#include <drm/drm_gpuvm.h> #include <drm/drm_ioctl.h> #include <kunit/test.h> #include <linux/auxiliary_bus.h>
rust/helpers/drm_gpuvm.c new file +29 -0
@@ -0,0 +1,29 @@ +// SPDX-License-Identifier: GPL-2.0 or MIT + +#include <drm/drm_gpuvm.h> + +#ifdef CONFIG_DRM +#ifdef CONFIG_DRM_GPUVM + +struct drm_gpuvm *rust_helper_drm_gpuvm_get (struct drm_gpuvm *obj) +{ +return drm_gpuvm_get (obj); +} + +void rust_helper_drm_gpuva_init_from_op (struct drm_gpuva *va,struct drm_gpuva_op_map *op) +{ +drm_gpuva_init_from_op (va, op); +} + +struct drm_gpuvm_bo *rust_helper_drm_gpuvm_bo_get (struct drm_gpuvm_bo *vm_bo) +{ +return drm_gpuvm_bo_get (vm_bo); +} + +bool rust_helper_drm_gpuvm_is_extobj (struct drm_gpuvm *gpuvm,struct drm_gem_object *obj) +{ +return drm_gpuvm_is_extobj (gpuvm, obj); +} + +#endif /* CONFIG_DRM_GPUVM */ +#endif /* CONFIG_DRM */
rust/helpers/helpers.c +1 -0
@@ -16,6 +16,7 @@#include "device.c" #include "dma-resv.c" #include "drm.c" +#include "drm_gpuvm.c" #include "err.c" #include "fs.c" #include "io.c"
rust/kernel/drm/gpuvm.rs new file +790 -0
790 lines not on this page · raw message
rust/kernel/drm/mod.rs +2 -0
@@ -6,4 +6,6 @@pub mod drv;pub mod file;pub mod gem; +#[cfg( CONFIG_DRM_GPUVM = "y" )] +pub mod gpuvm;pub mod ioctl;
Signature · 1 line
2.48.1
Re: [PATCH 2/2] rust: drm: Add GPUVM abstraction
Hi Daniel,
A few quick notes for future versions on style/docs to try to keep things consistent upstream -- not an actual review.
Quoted 2 lines · Daniel Almeida
+#[allow(type_alias_bounds)]
The documentation says this is highly discouraged -- it may be good to mention why it is OK in this instance in a comment or similar.
Also, could this be expect? (e.g. if it triggers in all compiler versions we support)
Quoted 1 line
+// A convenience type for the driver's GEM object.
Should this be a /// comment, i.e. docs?
Quoted 1 line
+/// Trait that must be implemented by DRM drivers to represent a DRM GpuVm (a GPU address space).
(Throughout the file) Markdown in documentation, e.g. GpuVm.
(Throughout the file) Intra-doc links where they work, e.g. [GpuVm] (assuming it works this one).
Quoted 1 line
+ // - Ok(()) is always returned.
(Throughout the file) Markdown in normal comments too.
Quoted 1 line
+/// A transparent wrapper over
drm_gpuva_op_map.
(Throughout the file) A link to C definitions is always nice if there is a good one, e.g.
[`drm_gpuva_op_map`]:
https://docs.kernel.org/gpu/drm-mm.html#c.drm_gpuva_op_map
Ideally we will eventually have a better way to link these automatically, but for the time being, this helps (and later we can do a replace easier).
rust/kernel/drm/gpuvm.rs line 130
Quoted 3 lines
rust/kernel/drm/gpuvm.rs new file +3
-0+/// `None`. + +/// Note: the reason for a dedicated remap operation, rather than arbitrary
Missing /// (?).
rust/kernel/drm/gpuvm.rs line 264
Quoted 5 lines
rust/kernel/drm/gpuvm.rs new file +5
-0+#[repr( C )] +#[pin_data] +/// A GPU VA range. +/// +/// Drivers can use `inner` to store additional data.
(Throughout the file) We typically place attributes go below the documentation -- or is there a reason to do it like this?
We had cases with e.g. Clippy bugs regarding safety comments that could be workarounded with "attribute movement", but it does not seem to be the case here.
Review
Quoted 2 lines
(quoted diff) +2
-0+ if p.is_null() { + Err(ENOMEM)
For error cases, we typically try to do early returns instead.
rust/kernel/drm/gpuvm.rs line 664
Quoted 3 lines
rust/kernel/drm/gpuvm.rs new file +3
-0+/// Iterates the given range of the GPU VA space. It utilizes +/// [`DriverGpuVm`] to call back into the driver providing the split and +/// merge steps.
This title (and the next one) may be a bit too long (or not -- please check in the rendered docs), i.e. the first paragraph is the "title", which is used differently in the rendered docs. If there is a way to have a shorter title that still differentiates between the two methods, that would be nice.
rust/kernel/drm/gpuvm.rs line 681
Quoted 7 lines
rust/kernel/drm/gpuvm.rs new file +7
-0+/// # Arguments +/// +/// - `ctx`: A driver-specific context. +/// - `req_obj`: The GEM object to map. +/// - `req_addr`: The start address of the new mapping. +/// - `req_range`: The range of the mapping. +/// - `req_offset`: The offset into the GEM object.
Normally we try to avoid this kind of sections and instead reference the arguments from the text (e.g. "...the range of the mapping (req_range)...") -- but if there is no good way to do it, then it is OK.
Quoted 1 line
+// SAFETY: All our trait methods take locks
(Throughout the file) Period at the end.
Thanks!
Cheers,
Miguel
Daniel Almeida
reviewerRe: [PATCH 2/2] rust: drm: Add GPUVM abstraction
Hi Miguel, thanks for having a look at this:
Quoted 7 lines · Miguel Ojeda
Hi Daniel,
A few quick notes for future versions on style/docs to try to keep things consistent upstream -- not an actual review.
Quoted 7 lines
Quoted 2 lines · Daniel Almeida
+#[allow(type_alias_bounds)]
The documentation says this is highly discouraged -- it may be good to mention why it is OK in this instance in a comment or similar.
Someone correct me here, but I see no issue with this warning. That’s because we need the bound to make <T::Driver as drv::Driver> work in the first place. Otherwise, we’d get a compiler error saying that there’s no Driver associated type (assuming the case where a random T gets passed in)
So, for this to be a problem, we would need to mix this up with something that also has a Driver associated type, and this associated type would also need a drv::Driver bound.
In other words, we would need a lot of things to align for this to actually have a chance of being misused. When you consider that this is then only used in a few places, the balance tips heavily in favor of the convenience of having the type alias IMHO.
In fact, the docs point to the exact thing I am trying to do, i.e.:
Quoted 8 lines
these bounds may have secondary effects such as enabling the use of “shorthand” associated type paths
I.e., paths of the form
T::Assocwhere T is a type parameter bounded by trait Trait which defines an associated type called Assoc as opposed to a fully qualified path of the form <T as Trait>::Assoc.Also, could this be
expect? (e.g. if it triggers in all compiler versions we support)
Quoted 7 lines
Quoted 1 line
+// A convenience type for the driver's GEM object.
Should this be a
///comment, i.e. docs?Quoted 1 line
+/// Trait that must be implemented by DRM drivers to represent a DRM GpuVm (a GPU address space).
(Throughout the file) Markdown in documentation, e.g.
GpuVm.
By the way, maybe we should have a lint for CamelCase in docs? I tried my best to cover all of these, but some slip through :/
i.e.: if you write something in CamelCase somewhere in the docs, there's a high chance that you should actually use Markdown and link as appropriate.
I have no idea whether this would actually work in practice, to be honest. It’s just a random suggestion (that I'd be willing to help with).
Quoted 26 lines
(Throughout the file) Intra-doc links where they work, e.g. [
GpuVm] (assuming it works this one).Quoted 1 line
+ // - Ok(()) is always returned.
(Throughout the file) Markdown in normal comments too.
Quoted 1 line
+/// A transparent wrapper over
drm_gpuva_op_map.(Throughout the file) A link to C definitions is always nice if there is a good one, e.g.
[`drm_gpuva_op_map`]:https://docs.kernel.org/gpu/drm-mm.html#c.drm_gpuva_op_map
Ideally we will eventually have a better way to link these automatically, but for the time being, this helps (and later we can do a replace easier).
rust/kernel/drm/gpuvm.rs line 130
Quoted 3 lines
rust/kernel/drm/gpuvm.rs new file +3
-0+/// `None`. + +/// Note: the reason for a dedicated remap operation, rather than arbitrary Missing
///(?).
Quoted 8 lines
rust/kernel/drm/gpuvm.rs line 264
Quoted 5 lines
rust/kernel/drm/gpuvm.rs new file +5
-0+#[repr( C )] +#[pin_data] +/// A GPU VA range. +/// +/// Drivers can use `inner` to store additional data. (Throughout the file) We typically place attributes go below the documentation -- or is there a reason to do it like this?
I will be honest with you here: I never remember the right order for docs and attributes.
I’ll fix this.
Quoted 28 lines
We had cases with e.g. Clippy bugs regarding safety comments that could be workarounded with "attribute movement", but it does not seem to be the case here.
Review
Quoted 2 lines
(quoted diff) +2
-0+ if p.is_null() { + Err(ENOMEM)For error cases, we typically try to do early returns instead.
rust/kernel/drm/gpuvm.rs line 664
Quoted 3 lines
rust/kernel/drm/gpuvm.rs new file +3
-0+/// Iterates the given range of the GPU VA space. It utilizes +/// [`DriverGpuVm`] to call back into the driver providing the split and +/// merge steps. This title (and the next one) may be a bit too long (or not -- please check in the rendered docs), i.e. the first paragraph is the "title", which is used differently in the rendered docs. If there is a way to have a shorter title that still differentiates between the two methods, that would be nice.
rust/kernel/drm/gpuvm.rs line 681
Quoted 7 lines
rust/kernel/drm/gpuvm.rs new file +7
-0+/// # Arguments +/// +/// - `ctx`: A driver-specific context. +/// - `req_obj`: The GEM object to map. +/// - `req_addr`: The start address of the new mapping. +/// - `req_range`: The range of the mapping. +/// - `req_offset`: The offset into the GEM object.
Quoted 4 lines
Normally we try to avoid this kind of sections and instead reference the arguments from the text (e.g. "...the range of the mapping (
req_range)...") -- but if there is no good way to do it, then it is OK.
Ack.
Quoted 9 lines
Quoted 2 lines
+// SAFETY: All our trait methods take locks
(Throughout the file) Period at the end.
Thanks!
Cheers,
Miguel
— Daniel
Re: [PATCH 2/2] rust: drm: Add GPUVM abstraction
Quoted 9 lines · Daniel Almeida
By the way, maybe we should have a lint for CamelCase in docs? I tried my best to cover all of these, but some slip through :/
i.e.: if you write something in CamelCase somewhere in the docs, there's a high chance that you should actually use Markdown and link as appropriate.
I have no idea whether this would actually work in practice, to be honest. It’s just a random suggestion (that I'd be willing to help with).
Yeah, I asked upstream if we could have something that at least detects what could have been intra-doc links, because that should not have too many false positives since it is a subset:
https://github.com/rust-lang/rust/issues/131510
It is still tricky, because e.g. Some or None would need to be excluded, even if you require the exact same case. So it probably still requires extra heuristics.
Then there is:
https://rust-lang.github.io/rust-clippy/master/index.html#doc_markdown
which is closer to your idea. It has a few false positives (e.g. KUnit) from a quick try, so probably we cannot enable it for everything, unless we commit to maintain a list of terms in the config. It spots a handful of things we should fix, though -- I can send a few patches or create good first issues.
Cheers,
Miguel
Re: [PATCH 2/2] rust: drm: Add GPUVM abstraction
Quoted 13 lines · Miguel Ojeda
Yeah, I asked upstream if we could have something that at least detects what could have been intra-doc links, because that should not have too many false positives since it is a subset:
https://github.com/rust-lang/rust/issues/131510It is still tricky, because e.g.
SomeorNonewould need to be excluded, even if you require the exact same case. So it probably still requires extra heuristics.Then there is:
Quoted 7 lines
https://rust-lang.github.io/rust-clippy/master/index.html#doc_markdown
which is closer to your idea. It has a few false positives (e.g. KUnit) from a quick try, so probably we cannot enable it for everything, unless we commit to maintain a list of terms in the config. It spots a handful of things we should fix, though -- I can send a few patches or create good first issues.
I went ahead and sent a quick series for the latter:
https://lore.kernel.org/rust-for-linux/[email protected]/
Cheers,
Miguel
Re: [PATCH 2/2] rust: drm: Add GPUVM abstraction
In reply to Daniel AlmeidaQuoted 25 lines · Daniel Almeida
Quoted 7 lines · Miguel Ojeda
Hi Daniel,
A few quick notes for future versions on style/docs to try to keep things consistent upstream -- not an actual review.
Quoted 7 lines
Quoted 2 lines · Daniel Almeida
+#[allow(type_alias_bounds)]
The documentation says this is highly discouraged -- it may be good to mention why it is OK in this instance in a comment or similar.
Someone correct me here, but I see no issue with this warning. That’s because we need the bound to make
<T::Driver as drv::Driver>work in the first place. Otherwise, we’d get a compiler error saying that there’s noDriverassociated type (assuming the case where a random T gets passed in)So, for this to be a problem, we would need to mix this up with something that also has a
Driverassociated type, and this associated type would also need adrv::Driverbound.
Quoted 10 lines
In other words, we would need a lot of things to align for this to actually have a chance of being misused. When you consider that this is then only used in a few places, the balance tips heavily in favor of the convenience of having the type alias IMHO.
In fact, the docs point to the exact thing I am trying to do, i.e.:
Quoted 3 lines
these bounds may have secondary effects such as enabling the use of “shorthand” associated type paths
I.e., paths of the form
T::Assocwhere T is a type parameter bounded by trait Trait which defines an associated type called Assoc as opposed to a fully qualified path of the form <T as Trait>::Assoc.
You can avoid the allow by using:
type DriverObject<T> = <<T as DriverGpuVm>::Driver as drv::Driver>::Object;
That is more wordy, but avoids the allow (it still errors when you put in something that doesn't implement DriverGpuVm).
---
Cheers,
Benno
Re: [PATCH 2/2] rust: drm: Add GPUVM abstraction
In reply to Daniel AlmeidaQuoted 19 lines · Daniel Almeida
rust/kernel/drm/gpuvm.rs new file +19
-0+/// The object returned after a call to [`GpuVm::lock`]. +/// +/// This object has access to operations that modify the VM's interval tree. +pub struct LockedGpuVm <'a,T :DriverGpuVm > { + gpuvm: &'aGpuVm <T >, +} + +impl <T :DriverGpuVm >LockedGpuVm <'_,T > { +/// Finds the [`GpuVmBo`] object that connects `obj` to this VM. +/// +/// If found, increases the reference count of the GpuVmBo object +/// accordingly. +pub fn find_bo (&mut self, obj: &DriverObject <T >) ->Option <ARef <GpuVmBo <T >>> { +// SAFETY: LockedGpuVm implies the right locks are held. +let p =unsafe { + bindings::drm_gpuvm_bo_find ( + self.gpuvm.gpuvm ()as *mut _ , + obj.gem_obj ()as *const _ as *mut _ , + )
rust/kernel/drm/gpuvm.rs line 635
Quoted 10 lines
rust/kernel/drm/gpuvm.rs new file +10
-0+ }; +if p.is_null () { +None + }else { +// SAFETY: All the drm_gpuvm_bo objects in this GpuVm are always allocated by us as GpuVmBo<T>. +let p =unsafe {crate ::container_of!(p,GpuVmBo <T >, bo)as *mut GpuVmBo <T > }; +// SAFETY: We checked for NULL above, and the types ensure that +// this object was created by vm_bo_alloc_callback<T>. +Some (unsafe {ARef ::from_raw (NonNull ::new_unchecked (p)) }) + }
Hi Daniel,
This is mostly eye candy—maybe we can simplify it to just:
if p.is_null() {
return None;
}
// SAFETY: All the drm_gpuvm_bo objects in this GpuVm are always allocated by us as GpuVmBo<T>.
let p = unsafe { crate::container_of!(p, GpuVmBo<T>, bo) as *mut GpuVmBo<T> };
// SAFETY: We checked for NULL above, and the types ensure that
// this object was created by vm_bo_alloc_callback<T>.
Some(unsafe { ARef::from_raw(NonNull::new_unchecked(p)) })Same with fn obtain_bo?
--
C. Mitrodimas
Quoted 25 lines
rust/kernel/drm/gpuvm.rs new file +25
-0+ } + +/// Obtains the [`GpuVmBo`] object that connects `obj` to this VM. +/// +/// This connection is unique, so an instane of [`GpuVmBo`] will be +/// allocated for `obj` once, and that instance will be returned from that +/// point forward. +pub fn obtain_bo (&mut self, obj: &DriverObject <T >) ->Result <ARef <GpuVmBo <T >>> { +// SAFETY: LockedGpuVm implies the right locks are held. +let p =unsafe { + bindings::drm_gpuvm_bo_obtain ( + self.gpuvm.gpuvm ()as *mut _ , + obj.gem_obj ()as *const _ as *mut _ , + ) + }; +if p.is_null () { +Err (ENOMEM ) + }else { +// SAFETY: Container invariant is guaranteed for GpuVmBo objects for this GpuVm. +let p =unsafe {crate ::container_of!(p,GpuVmBo <T >, bo)as *mut GpuVmBo <T > }; +// SAFETY: We checked for NULL above, and the types ensure that +// this object was created by vm_bo_alloc_callback<T>. +Ok (unsafe {ARef ::from_raw (NonNull ::new_unchecked (p)) }) + } + }