gpu: nova-core: add ChannelIdPool
Subsystems (5)
PATCH 1/1 gpu: nova-core: add ChannelIdPool
Add ChannelIdPool which does allocation of channel IDs on top of a maple tree. This is necessary for allotting ranges of channel IDs to be used in e.g. vGPU and also later for allocating single channel IDs for the host.
Pre-blackwell needs to allocate channel IDs with a particular alignment. For this case, loop on MapleTreeAlloc::alloc_range while aligning up. For unaligned cases, this always terminates on the first iteration. Use a Mutex to ensure that the multi step insert-check-erase can't race.
---
Add tracking of channel ID allocations, needed for allotting ranges of channel IDs for vGPU guests and also (later) for regular host channel ID reservation. Allocating channel IDs is not on a hot path (absolute peak ~100 per second, on average < 1 per second).
For pre-blackwell we need to be able to allocate a range of channel IDs with a particular alignment. The approach I took here is to wrap MapleTreeAlloc in a Mutex and loop on regular alloc_range lookups. An alternative would be adding rust bindings for maple tree mas_empty_area and doing a try+restart kind of thing.
--- drivers/gpu/nova-core/gpu.rs | 2 + drivers/gpu/nova-core/gpu/channel.rs | 174 +++++++++++++++++++++++++++++++++++ 2 files changed, 176 insertions(+)
drivers/gpu/nova-core/gpu.rs +2 -0
@@ -31,6 +31,8 @@ regs, }; +#[cfg_attr(not( CONFIG_KUNIT = "y" ), expect(dead_code))] +mod channel;mod hal;macro_rules! define_chipset {
drivers/gpu/nova-core/gpu/channel.rs new file +174 -0
174 lines not on this page · raw message
---
base-commit: 78900738d981c97aad929e33620340d0d9bcfc82
change-id: 20260709-chid-maple-3d696554e0a3
Best regards,
Signature · 1 line
Eliot Courtney <[email protected]>
On 7/10/26 6:42 AM, Eliot Courtney wrote:
...
drivers/gpu/nova-core/gpu/channel.rs line 15
Quoted 2 lines
drivers/gpu/nova-core/gpu/channel.rs new file +2
-0+use kernel::{ + maple_tree::MapleTreeAlloc ,
Hi Eliot, Alice, all,
Eliot already laid out the maple costs (the alloc_range+erase loop replicating find_next_zero_area, the extra mutex, allocation on alloc and free), so I won't rehash them. What I can add is why the alignment is a hard requirement, since that is what seems to make the decision clearer to me at least.
Pre-Blackwell, USERD sits in BAR1 at page granularity, 8 channels to a 4K page (chid = page*8 + slot). Giving a VM its own channels means giving it whole USERD pages, so a VM's chid range has to be 8-aligned and a multiple of 8. Blackwell moves submit to a per-function doorbell keyed by (runlist, chid), and the constraint goes away. So it's a constraint we're stuck with pre-Blackwell, not one we can design away.
Given that, the bitmap version is hard to argue with. The aligned allocation is the one call the API is built around, roughly:
let mut ids = self.inner.lock();
let area = ids.find_unused_area(0, count, align_mask)
.ok_or(ENOSPC)?;
// area.acquire() reserves and returns the range, drop clears itOne lock, no retry, and release is a bitmap_clear that can't fail. For 2048 IDs the backing store is a 256-byte array. find_next_zero_area() is also the existing idiom for this (IOMMU, DMA, IRQ), so it answers Greg's reuse-what-exists point as well.
I'd go with the bitmap id_pool.
thanks,
Signature · 1 line
John Hubbard
Quoted 28 lines · John Hubbard
On 7/10/26 6:42 AM, Eliot Courtney wrote:
...drivers/gpu/nova-core/gpu/channel.rs line 15
Quoted 2 lines
drivers/gpu/nova-core/gpu/channel.rs new file +2
-0+use kernel::{ + maple_tree::MapleTreeAlloc ,Hi Eliot, Alice, all,
Eliot already laid out the maple costs (the alloc_range+erase loop replicating
find_next_zero_area, the extra mutex, allocation on alloc and free), so I won't rehash them. What I can add is why the alignment is a hard requirement, since that is what seems to make the decision clearer to me at least.Pre-Blackwell, USERD sits in BAR1 at page granularity, 8 channels to a 4K page (chid = page*8 + slot). Giving a VM its own channels means giving it whole USERD pages, so a VM's chid range has to be 8-aligned and a multiple of 8. Blackwell moves submit to a per-function doorbell keyed by (runlist, chid), and the constraint goes away. So it's a constraint we're stuck with pre-Blackwell, not one we can design away.
Given that, the bitmap version is hard to argue with. The aligned allocation is the one call the API is built around, roughly:
let mut ids = self.inner.lock(); let area = ids.find_unused_area(0, count, align_mask) .ok_or(ENOSPC)?; // area.acquire() reserves and returns the range, drop clears it
Quoted 6 lines
One lock, no retry, and release is a
bitmap_clearthat can't fail. For 2048 IDs the backing store is a 256-byte array.find_next_zero_area()is also the existing idiom for this (IOMMU, DMA, IRQ), so it answers Greg's reuse-what-exists point as well.I'd go with the bitmap
id_pool.
Hi John,
I agree. Bitmaps are hard to beat, especially when it comes to just a couple thousands of bits.
Eliot,
If you consider something more complicated, I'd suggest to estimate memory and time consumption in a typical scenario, then weight the cost of maintenance of one solution vs another.
If you're hard-limited with 2048 bits, bitmap is your natural choice.
Quoted 4 lines
Pre-Blackwell, USERD sits in BAR1 at page granularity, 8 channels to a 4K page (chid = page*8 + slot). Giving a VM its own channels means giving it whole USERD pages, so a VM's chid range has to be 8-aligned and a multiple of 8.
If you always allocate 8-bit aligned area and never partially release IDs, you can have just 256-bit bitmap, where each bit represents 8 IDs. For example, bit #1 represents IDs 8..15.
There's no real code using it in the patch, unfortunately, so it's hard to say if that's doable.
Thanks,
Yury
Alice Ryhl
maintainerQuoted 28 lines · John Hubbard
On 7/10/26 6:42 AM, Eliot Courtney wrote:
...drivers/gpu/nova-core/gpu/channel.rs line 15
Quoted 2 lines
drivers/gpu/nova-core/gpu/channel.rs new file +2
-0+use kernel::{ + maple_tree::MapleTreeAlloc ,Hi Eliot, Alice, all,
Eliot already laid out the maple costs (the alloc_range+erase loop replicating
find_next_zero_area, the extra mutex, allocation on alloc and free), so I won't rehash them. What I can add is why the alignment is a hard requirement, since that is what seems to make the decision clearer to me at least.Pre-Blackwell, USERD sits in BAR1 at page granularity, 8 channels to a 4K page (chid = page*8 + slot). Giving a VM its own channels means giving it whole USERD pages, so a VM's chid range has to be 8-aligned and a multiple of 8. Blackwell moves submit to a per-function doorbell keyed by (runlist, chid), and the constraint goes away. So it's a constraint we're stuck with pre-Blackwell, not one we can design away.
Given that, the bitmap version is hard to argue with. The aligned allocation is the one call the API is built around, roughly:
let mut ids = self.inner.lock(); let area = ids.find_unused_area(0, count, align_mask) .ok_or(ENOSPC)?; // area.acquire() reserves and returns the range, drop clears it
Quoted 6 lines
One lock, no retry, and release is a
bitmap_clearthat can't fail. For 2048 IDs the backing store is a 256-byte array.find_next_zero_area()is also the existing idiom for this (IOMMU, DMA, IRQ), so it answers Greg's reuse-what-exists point as well.I'd go with the bitmap
id_pool.
Completely agreed. The loop in this patch seems like a really bad idea to me. If the maple tree doesn't natively provide a way to allocate the range with an alignment requirement, we should not attempt to hack in support for alignment by looping. Please use the bitmap.
(Even without the alignment requirement, I think that the 2048 size limit is also a good reason to use a bitmap, but the argument is less strong in that case.)
Alice
Gary Guo
reviewerQuoted 36 lines · Alice Ryhl
Quoted 28 lines · John Hubbard
On 7/10/26 6:42 AM, Eliot Courtney wrote:
...drivers/gpu/nova-core/gpu/channel.rs line 15
Quoted 2 lines
drivers/gpu/nova-core/gpu/channel.rs new file +2
-0+use kernel::{ + maple_tree::MapleTreeAlloc ,Hi Eliot, Alice, all,
Eliot already laid out the maple costs (the alloc_range+erase loop replicating
find_next_zero_area, the extra mutex, allocation on alloc and free), so I won't rehash them. What I can add is why the alignment is a hard requirement, since that is what seems to make the decision clearer to me at least.Pre-Blackwell, USERD sits in BAR1 at page granularity, 8 channels to a 4K page (chid = page*8 + slot). Giving a VM its own channels means giving it whole USERD pages, so a VM's chid range has to be 8-aligned and a multiple of 8. Blackwell moves submit to a per-function doorbell keyed by (runlist, chid), and the constraint goes away. So it's a constraint we're stuck with pre-Blackwell, not one we can design away.
Given that, the bitmap version is hard to argue with. The aligned allocation is the one call the API is built around, roughly:
let mut ids = self.inner.lock(); let area = ids.find_unused_area(0, count, align_mask) .ok_or(ENOSPC)?; // area.acquire() reserves and returns the range, drop clears itQuoted 6 lines
One lock, no retry, and release is a
bitmap_clearthat can't fail. For 2048 IDs the backing store is a 256-byte array.find_next_zero_area()is also the existing idiom for this (IOMMU, DMA, IRQ), so it answers Greg's reuse-what-exists point as well.I'd go with the bitmap
id_pool.
Quoted 8 lines
Completely agreed. The loop in this patch seems like a really bad idea to me. If the maple tree doesn't natively provide a way to allocate the range with an alignment requirement, we should not attempt to hack in support for alignment by looping. Please use the bitmap.
(Even without the alignment requirement, I think that the 2048 size limit is also a good reason to use a bitmap, but the argument is less strong in that case.)
I think the reasoning is strong, you get everything fit within 4 cache lines and with even a dumb prefetcher you'd have no more than 2 cache misses . It's going to be faster than data structures that need indirection.
Best,
Gary