Skip to contentSkip to conversation

gpu: nova-core: add ChannelIdPool

PATCH In review Eliot Courtney 5 participants 5 messages Jul 10 – Jul 11

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 gpu: nova-core: add ChannelIdPool

Add ChannelIdPool which does allocation of channel IDs on top of a maple tree. This is necessary for allotting ranges of channel IDs to be used in e.g. vGPU and also later for allocating single channel IDs for the host.

Pre-blackwell needs to allocate channel IDs with a particular alignment. For this case, loop on MapleTreeAlloc::alloc_range while aligning up. For unaligned cases, this always terminates on the first iteration. Use a Mutex to ensure that the multi step insert-check-erase can't race.

---
Add tracking of channel ID allocations, needed for allotting ranges of channel IDs for vGPU guests and also (later) for regular host channel ID reservation. Allocating channel IDs is not on a hot path (absolute peak ~100 per second, on average < 1 per second).

For pre-blackwell we need to be able to allocate a range of channel IDs with a particular alignment. The approach I took here is to wrap MapleTreeAlloc in a Mutex and loop on regular alloc_range lookups. An alternative would be adding rust bindings for maple tree mas_empty_area and doing a try+restart kind of thing.

---
 drivers/gpu/nova-core/gpu.rs         |   2 +
 drivers/gpu/nova-core/gpu/channel.rs | 174 +++++++++++++++++++++++++++++++++++
 2 files changed, 176 insertions(+)
drivers/gpu/nova-core/gpu.rs +2 -0
index 43c3f4f8df71..0c3b5de7d849 100644
@@ -31,6 +31,8 @@
     regs,
 };
 
+#[cfg_attr(not(CONFIG_KUNIT = "y"), expect(dead_code))]
+mod channel;
 mod hal;
 
 macro_rules! define_chipset {
drivers/gpu/nova-core/gpu/channel.rs new file +174 -0

174 lines not on this page · raw message

---
base-commit: 78900738d981c97aad929e33620340d0d9bcfc82
change-id: 20260709-chid-maple-3d696554e0a3

Best regards,

Signature · 1 line
Eliot Courtney <[email protected]>
#1 lore raw

On 7/10/26 6:42 AM, Eliot Courtney wrote:
...

drivers/gpu/nova-core/gpu/channel.rs line 15

Quoted 2 lines
drivers/gpu/nova-core/gpu/channel.rs new file +2 -0
+use kernel::{
+    maple_tree::MapleTreeAlloc,

Hi Eliot, Alice, all,

Eliot already laid out the maple costs (the alloc_range+erase loop replicating find_next_zero_area, the extra mutex, allocation on alloc and free), so I won't rehash them. What I can add is why the alignment is a hard requirement, since that is what seems to make the decision clearer to me at least.

Pre-Blackwell, USERD sits in BAR1 at page granularity, 8 channels to a 4K page (chid = page*8 + slot). Giving a VM its own channels means giving it whole USERD pages, so a VM's chid range has to be 8-aligned and a multiple of 8. Blackwell moves submit to a per-function doorbell keyed by (runlist, chid), and the constraint goes away. So it's a constraint we're stuck with pre-Blackwell, not one we can design away.

Given that, the bitmap version is hard to argue with. The aligned allocation is the one call the API is built around, roughly:

    let mut ids = self.inner.lock();
    let area = ids.find_unused_area(0, count, align_mask)
                  .ok_or(ENOSPC)?;
    // area.acquire() reserves and returns the range, drop clears it

One lock, no retry, and release is a bitmap_clear that can't fail. For 2048 IDs the backing store is a 256-byte array. find_next_zero_area() is also the existing idiom for this (IOMMU, DMA, IRQ), so it answers Greg's reuse-what-exists point as well.

I'd go with the bitmap id_pool.

thanks,

Signature · 1 line
John Hubbard
#2 lore raw
Quoted 28 lines · John Hubbard

On 7/10/26 6:42 AM, Eliot Courtney wrote:
...

drivers/gpu/nova-core/gpu/channel.rs line 15

Quoted 2 lines
drivers/gpu/nova-core/gpu/channel.rs new file +2 -0
+use kernel::{
+    maple_tree::MapleTreeAlloc,

Hi Eliot, Alice, all,

Eliot already laid out the maple costs (the alloc_range+erase loop replicating find_next_zero_area, the extra mutex, allocation on alloc and free), so I won't rehash them. What I can add is why the alignment is a hard requirement, since that is what seems to make the decision clearer to me at least.

Pre-Blackwell, USERD sits in BAR1 at page granularity, 8 channels to a 4K page (chid = page*8 + slot). Giving a VM its own channels means giving it whole USERD pages, so a VM's chid range has to be 8-aligned and a multiple of 8. Blackwell moves submit to a per-function doorbell keyed by (runlist, chid), and the constraint goes away. So it's a constraint we're stuck with pre-Blackwell, not one we can design away.

Given that, the bitmap version is hard to argue with. The aligned allocation is the one call the API is built around, roughly:

    let mut ids = self.inner.lock();
    let area = ids.find_unused_area(0, count, align_mask)
                  .ok_or(ENOSPC)?;
    // area.acquire() reserves and returns the range, drop clears it
Quoted 6 lines

One lock, no retry, and release is a bitmap_clear that can't fail. For 2048 IDs the backing store is a 256-byte array. find_next_zero_area() is also the existing idiom for this (IOMMU, DMA, IRQ), so it answers Greg's reuse-what-exists point as well.

I'd go with the bitmap id_pool.

Hi John,

I agree. Bitmaps are hard to beat, especially when it comes to just a couple thousands of bits.

Eliot,

If you consider something more complicated, I'd suggest to estimate memory and time consumption in a typical scenario, then weight the cost of maintenance of one solution vs another.

If you're hard-limited with 2048 bits, bitmap is your natural choice.

Quoted 4 lines

Pre-Blackwell, USERD sits in BAR1 at page granularity, 8 channels to a 4K page (chid = page*8 + slot). Giving a VM its own channels means giving it whole USERD pages, so a VM's chid range has to be 8-aligned and a multiple of 8.

If you always allocate 8-bit aligned area and never partially release IDs, you can have just 256-bit bitmap, where each bit represents 8 IDs. For example, bit #1 represents IDs 8..15.

There's no real code using it in the patch, unfortunately, so it's hard to say if that's doable.

Thanks,
Yury

maintainer
#3 lore raw
In reply to John Hubbard
Quoted 28 lines · John Hubbard

On 7/10/26 6:42 AM, Eliot Courtney wrote:
...

drivers/gpu/nova-core/gpu/channel.rs line 15

Quoted 2 lines
drivers/gpu/nova-core/gpu/channel.rs new file +2 -0
+use kernel::{
+    maple_tree::MapleTreeAlloc,

Hi Eliot, Alice, all,

Eliot already laid out the maple costs (the alloc_range+erase loop replicating find_next_zero_area, the extra mutex, allocation on alloc and free), so I won't rehash them. What I can add is why the alignment is a hard requirement, since that is what seems to make the decision clearer to me at least.

Pre-Blackwell, USERD sits in BAR1 at page granularity, 8 channels to a 4K page (chid = page*8 + slot). Giving a VM its own channels means giving it whole USERD pages, so a VM's chid range has to be 8-aligned and a multiple of 8. Blackwell moves submit to a per-function doorbell keyed by (runlist, chid), and the constraint goes away. So it's a constraint we're stuck with pre-Blackwell, not one we can design away.

Given that, the bitmap version is hard to argue with. The aligned allocation is the one call the API is built around, roughly:

    let mut ids = self.inner.lock();
    let area = ids.find_unused_area(0, count, align_mask)
                  .ok_or(ENOSPC)?;
    // area.acquire() reserves and returns the range, drop clears it
Quoted 6 lines

One lock, no retry, and release is a bitmap_clear that can't fail. For 2048 IDs the backing store is a 256-byte array. find_next_zero_area() is also the existing idiom for this (IOMMU, DMA, IRQ), so it answers Greg's reuse-what-exists point as well.

I'd go with the bitmap id_pool.

Completely agreed. The loop in this patch seems like a really bad idea to me. If the maple tree doesn't natively provide a way to allocate the range with an alignment requirement, we should not attempt to hack in support for alignment by looping. Please use the bitmap.

(Even without the alignment requirement, I think that the 2048 size limit is also a good reason to use a bitmap, but the argument is less strong in that case.)

Alice

reviewer
#4 lore raw
Quoted 36 lines · Alice Ryhl
Quoted 28 lines · John Hubbard

On 7/10/26 6:42 AM, Eliot Courtney wrote:
...

drivers/gpu/nova-core/gpu/channel.rs line 15

Quoted 2 lines
drivers/gpu/nova-core/gpu/channel.rs new file +2 -0
+use kernel::{
+    maple_tree::MapleTreeAlloc,

Hi Eliot, Alice, all,

Eliot already laid out the maple costs (the alloc_range+erase loop replicating find_next_zero_area, the extra mutex, allocation on alloc and free), so I won't rehash them. What I can add is why the alignment is a hard requirement, since that is what seems to make the decision clearer to me at least.

Pre-Blackwell, USERD sits in BAR1 at page granularity, 8 channels to a 4K page (chid = page*8 + slot). Giving a VM its own channels means giving it whole USERD pages, so a VM's chid range has to be 8-aligned and a multiple of 8. Blackwell moves submit to a per-function doorbell keyed by (runlist, chid), and the constraint goes away. So it's a constraint we're stuck with pre-Blackwell, not one we can design away.

Given that, the bitmap version is hard to argue with. The aligned allocation is the one call the API is built around, roughly:

    let mut ids = self.inner.lock();
    let area = ids.find_unused_area(0, count, align_mask)
                  .ok_or(ENOSPC)?;
    // area.acquire() reserves and returns the range, drop clears it
Quoted 6 lines

One lock, no retry, and release is a bitmap_clear that can't fail. For 2048 IDs the backing store is a 256-byte array. find_next_zero_area() is also the existing idiom for this (IOMMU, DMA, IRQ), so it answers Greg's reuse-what-exists point as well.

I'd go with the bitmap id_pool.

Quoted 8 lines

Completely agreed. The loop in this patch seems like a really bad idea to me. If the maple tree doesn't natively provide a way to allocate the range with an alignment requirement, we should not attempt to hack in support for alignment by looping. Please use the bitmap.

(Even without the alignment requirement, I think that the 2048 size limit is also a good reason to use a bitmap, but the argument is less strong in that case.)

I think the reasoning is strong, you get everything fit within 4 cache lines and with even a dumb prefetcher you'd have no more than 2 cache misses . It's going to be faster than data structures that need indirection.

Best,
Gary