Changing radeon KMS cs+gem ioctl to merge read & write domain
Hi,
I think we should merge the read & write domain of radeon KMS into a single domains information. I don't think there is a good reason for separate read & write domain, we did copy intel model for that and intel use this mostly for cache coherency & cache flushing as far as i understand. We make no good use of this inside the kernel. In order to make this change less disruptive and easier to introduce i propose we keep libdrm-radeon api intact thus userspace xf86video-ati & mesa 3d driver doesn't need a single line patch to adapt. Attached is a proof of concept, a patch against libdrm which merge read & write domain and only use the read domain to communicate with the kernel. I am still in process of stress testing this patch but so far neither X or 3D had any glitches.
I want to take advantage of this change to the cs reloc to the following:
struct drm_radeon_cs_reloc { »·······uint32_t»·······»·······handle; »·······uint32_t»·······»·······domains; »·······uint32_t»·······»·······unused; »·······uint32_t»·······»·······flags;
};
With the following rules: a domain is a 4bit value (more than enough i believe). Userspace can then provide domain preference for each relocation. For instance :
0 Invalid|CPU
1 VRAM
2 GTT
domains = (VRAM << 0) | (GTT << 4)
would mean try to place object in VRAM first, if not enough VRAM place it in GTT.
domains = (GTT << 0)
object can only be in GTT
...
I believe this would be a lot more useful information that read|write domain. We would also now assume that userspace knows what it's doing inside a single submited cs and that userspace issue necessary flush if a bo is used in different way. Which is what the ddx does.
I believe the only argument in favor of read & write split is broken AGP chipset where GPU can't write to GART. So far we don't use this information to work around the issue, we don't even always test AGP writeback. Thus i believe this change won't impact current user. Note that i am working on code to work around bad AGP chipset (fallback to PCI GART for GPU write + detection of broken writeback).
I really think we should take advantage of being in staging driver to get the ioctl right before we have to freeze them.
Cheers,
Jerome Glisse
Quoted 1 line
From
4dcdd0b5fce6d8ca5a6b8ae08cfbb0c80af6613eMon Sep 17 00:00:00 2001
From: Jerome Glisse <[email protected]>
Date: Thu, 22 Oct 2009 00:26:32 +0200
Subject: [PATCH] libdrm/libradeon: Unify read & write domain into a single one
This unify read & write domain into a single domain without changing the API or ABI of libdrm_radeon.
--- libdrm/radeon/radeon_cs.h | 11 +-- libdrm/radeon/radeon_cs_gem.c | 35 +++----- libdrm/radeon/radeon_cs_space.c | 169 +++++++++++++-------------------------- 3 files changed, 71 insertions(+), 144 deletions(-)
libdrm/radeon/radeon_cs.h +4 -7
@@ -40,8 +40,7 @@struct radeon_cs_reloc {struct radeon_bo *bo; -uint32_t read_domain; -uint32_t write_domain; +uint32_t domains;uint32_t flags; };
@@ -52,8 +51,7 @@ struct radeon_cs_reloc {struct radeon_cs_space_check {struct radeon_bo *bo; -uint32_t read_domains; -uint32_t write_domain; +uint32_t domains;uint32_t new_accounted; };
@@ -109,9 +107,8 @@ struct radeon_cs_funcs {struct radeon_cs_manager {struct radeon_cs_funcs *funcs;int fd; -int32_t vram_limit, gart_limit; -int32_t vram_write_used, gart_write_used; -int32_t read_used; +int32_t vram_limit, gart_limit; +int32_t vram_free, gart_free; }; static inline struct radeon_cs *radeon_cs_create(struct radeon_cs_manager *csm,
libdrm/radeon/radeon_cs_gem.c +12 -23
@@ -115,11 +115,12 @@ static int cs_gem_write_reloc(struct radeon_cs *cs, {struct cs_gem *csg = (struct cs_gem *)cs;struct cs_reloc_gem *reloc; -uint32_t idx; +uint32_t idx, domains;unsigned i;assert (bo->space_accounted); - + domains = read_domain | write_domain; +#if 0 /* check domains */ if ((read_domain && write_domain) || (!read_domain && !write_domain)) {/* in one CS a bo can only be in read or write domain but not
@@ -127,10 +128,11 @@ static int cs_gem_write_reloc(struct radeon_cs *cs,*/ return -EINVAL; } -if (read_domain ==RADEON_GEM_DOMAIN_CPU ) { +#endif +if (read_domain &RADEON_GEM_DOMAIN_CPU ) {return -EINVAL ; } -if (write_domain ==RADEON_GEM_DOMAIN_CPU ) { +if (write_domain &RADEON_GEM_DOMAIN_CPU ) {return -EINVAL ; }/* check if bo is already referenced */
@@ -145,20 +147,8 @@ static int cs_gem_write_reloc(struct radeon_cs *cs,* new relocation. */ /* the DDX expects to read and write from same pixmap */ -if (write_domain && (reloc->read_domain & write_domain)) { - reloc->read_domain =0 ; - reloc->write_domain = write_domain; - }else if (read_domain & reloc->write_domain) { - reloc->read_domain =0 ; - }else { -if (write_domain != reloc->write_domain) -return -EINVAL ; -if (read_domain != reloc->read_domain) -return -EINVAL ; - } - - reloc->read_domain |= read_domain; - reloc->write_domain |= write_domain; + reloc->read_domain |= domains; + reloc->write_domain =0 ;/* update flags */ reloc->flags |= (flags & reloc->flags);/* write relocation packet */
@@ -190,8 +180,8 @@ static int cs_gem_write_reloc(struct radeon_cs *cs, idx = (csg->base.crelocs++) *RELOC_SIZE ; reloc = (struct cs_reloc_gem *)&csg->relocs[idx]; reloc->handle = bo->handle; - reloc->read_domain = read_domain; - reloc->write_domain = write_domain; + reloc->read_domain = domains; + reloc->write_domain =0 ; reloc->flags = flags; csg->chunks[1 ].length_dw +=RELOC_SIZE ;radeon_bo_ref (bo);
@@ -283,9 +273,8 @@ static int cs_gem_emit(struct radeon_cs *cs) csg->relocs_bo[i] =NULL ; } - cs->csm->read_used =0 ; - cs->csm->vram_write_used =0 ; - cs->csm->gart_write_used =0 ; + cs->csm->vram_free = cs->csm->vram_limit; + cs->csm->gart_free = cs->csm->gart_limit;return r; }
libdrm/radeon/radeon_cs_space.c +55 -114
243 lines not on this page · raw message
Signature · 1 line
1.6.5.rc2
Quoted 54 lines · Jerome Glisse
Hi,
I think we should merge the read & write domain of radeon KMS into a single domains information. I don't think there is a good reason for separate read & write domain, we did copy intel model for that and intel use this mostly for cache coherency & cache flushing as far as i understand. We make no good use of this inside the kernel. In order to make this change less disruptive and easier to introduce i propose we keep libdrm-radeon api intact thus userspace xf86video-ati & mesa 3d driver doesn't need a single line patch to adapt. Attached is a proof of concept, a patch against libdrm which merge read & write domain and only use the read domain to communicate with the kernel. I am still in process of stress testing this patch but so far neither X or 3D had any glitches.
I want to take advantage of this change to the cs reloc to the following:
structdrm_radeon_cs_reloc{ »·······uint32_t»·······»·······handle; »·······uint32_t»·······»·······domains; »·······uint32_t»·······»·······unused; »·······uint32_t»·······»·······flags;
};With the following rules: a domain is a 4bit value (more than enough i believe). Userspace can then provide domain preference for each relocation. For instance :
0 Invalid|CPU
1 VRAM
2 GTTdomains = (VRAM << 0) | (GTT << 4)
would mean try to place object in VRAM first, if not enough VRAM place it in GTT.domains = (GTT << 0)
object can only be in GTT
...I believe this would be a lot more useful information that read|write domain. We would also now assume that userspace knows what it's doing inside a single submited cs and that userspace issue necessary flush if a bo is used in different way. Which is what the ddx does.
I believe the only argument in favor of read & write split is broken AGP chipset where GPU can't write to GART. So far we don't use this information to work around the issue, we don't even always test AGP writeback. Thus i believe this change won't impact current user. Note that i am working on code to work around bad AGP chipset (fallback to PCI GART for GPU write + detection of broken writeback).
Quoted 2 lines
I really think we should take advantage of being in staging driver to get the ioctl right before we have to freeze them.
No objections from me. If you have further ioctl changes, raising them sooner rather than later would be *greatly* appreciated since I'm probably the only person touching them in Gallium.
~ C.
Quoted 55 lines · Corbin Simpson
Quoted 54 lines · Jerome Glisse
Hi,
I think we should merge the read & write domain of radeon KMS into a single domains information. I don't think there is a good reason for separate read & write domain, we did copy intel model for that and intel use this mostly for cache coherency & cache flushing as far as i understand. We make no good use of this inside the kernel. In order to make this change less disruptive and easier to introduce i propose we keep libdrm-radeon api intact thus userspace xf86video-ati & mesa 3d driver doesn't need a single line patch to adapt. Attached is a proof of concept, a patch against libdrm which merge read & write domain and only use the read domain to communicate with the kernel. I am still in process of stress testing this patch but so far neither X or 3D had any glitches.
I want to take advantage of this change to the cs reloc to the following:
structdrm_radeon_cs_reloc{ »·······uint32_t»·······»·······handle; »·······uint32_t»·······»·······domains; »·······uint32_t»·······»·······unused; »·······uint32_t»·······»·······flags;
};With the following rules: a domain is a 4bit value (more than enough i believe). Userspace can then provide domain preference for each relocation. For instance :
0 Invalid|CPU
1 VRAM
2 GTTdomains = (VRAM << 0) | (GTT << 4)
would mean try to place object in VRAM first, if not enough VRAM place it in GTT.domains = (GTT << 0)
object can only be in GTT
...I believe this would be a lot more useful information that read|write domain. We would also now assume that userspace knows what it's doing inside a single submited cs and that userspace issue necessary flush if a bo is used in different way. Which is what the ddx does.
I believe the only argument in favor of read & write split is broken AGP chipset where GPU can't write to GART. So far we don't use this information to work around the issue, we don't even always test AGP writeback. Thus i believe this change won't impact current user. Note that i am working on code to work around bad AGP chipset (fallback to PCI GART for GPU write + detection of broken writeback).
Quoted 8 lines
Quoted 2 lines
I really think we should take advantage of being in staging driver to get the ioctl right before we have to freeze them.
No objections from me. If you have further ioctl changes, raising them sooner rather than later would be *greatly* appreciated since I'm probably the only person touching them in Gallium.
~ C.
This change should work without any update to gallium code. But to take advantages of the placement list you would need to use a new libdrm-radeon API for reloc.
Beside this merge i don't have in mind any other API change. Others things i am working on are mostly kernel side only.
Cheers,
Jerome
Quoted 16 lines · Jerome Glisse
Hi,
I think we should merge the read & write domain of radeon KMS into a single domains information. I don't think there is a good reason for separate read & write domain, we did copy intel model for that and intel use this mostly for cache coherency & cache flushing as far as i understand. We make no good use of this inside the kernel. In order to make this change less disruptive and easier to introduce i propose we keep libdrm-radeon api intact thus userspace xf86video-ati & mesa 3d driver doesn't need a single line patch to adapt. Attached is a proof of concept, a patch against libdrm which merge read & write domain and only use the read domain to communicate with the kernel. I am still in process of stress testing this patch but so far neither X or 3D had any glitches.
Can you list the advantages (speed, complexity reduction)?, I really really don't like this patch at this point in the development process, yes the API has warts does fixing it now help any or just increase the chance of regressions.
Like I don't think we've hit any of the limitations yet, and I suspect the API limitations we hit will require a new revision of the API, which I'd rather do once, than just hack away functionality because the current underlying implementation doesn't use it yet.
I'd really like to use the read/write information to help decide VRAM/GTT migration priorities in the future, I've mentioned this a few times and I've haven't heard how your scheme addresses this.
Some issues I can see are you haven't really defined how userspace users of this API should look, like who decides the buffer placement? userspace only? can the kernel override? how does the kernel know which allocs it can override and which it can't? how does space checking work for the VRAM but maybe GTT buffers?
I don't think the API we have is perfect by any means I just don't think investing in tweaking the corners for no real benefit is worth it. Just make the gallium winsys API clean and then you don't need to look at this stuff, and in a year or two a new API that actually provides benefits and speedups after we've learned a bit from this one.
Dave.
Quoted 18 lines · Dave Airlie
Quoted 16 lines · Jerome Glisse
Hi,
I think we should merge the read & write domain of radeon KMS into a single domains information. I don't think there is a good reason for separate read & write domain, we did copy intel model for that and intel use this mostly for cache coherency & cache flushing as far as i understand. We make no good use of this inside the kernel. In order to make this change less disruptive and easier to introduce i propose we keep libdrm-radeon api intact thus userspace xf86video-ati & mesa 3d driver doesn't need a single line patch to adapt. Attached is a proof of concept, a patch against libdrm which merge read & write domain and only use the read domain to communicate with the kernel. I am still in process of stress testing this patch but so far neither X or 3D had any glitches.
Quoted 4 lines
Can you list the advantages (speed, complexity reduction)?, I really really don't like this patch at this point in the development process, yes the API has warts does fixing it now help any or just increase the chance of regressions.
See below
Quoted 8 lines
Like I don't think we've hit any of the limitations yet, and I suspect the API limitations we hit will require a new revision of the API, which I'd rather do once, than just hack away functionality because the current underlying implementation doesn't use it yet.
I'd really like to use the read/write information to help decide VRAM/GTT migration priorities in the future, I've mentioned this a few times and I've haven't heard how your scheme addresses this.
New API gives opportunity to give a list of prefered placement, this would be a lot better than read/write. Userspace has better view of what is the usage of a buffer (used one frame ? big texture used several frame ? big texture of which only few texel will be use ? ...) Read/Write domain fails to transmit any such information to the kernel.
Quoted 5 lines
Some issues I can see are you haven't really defined how userspace users of this API should look, like who decides the buffer placement? userspace only? can the kernel override? how does the kernel know which allocs it can override and which it can't? how does space checking work for the VRAM but maybe GTT buffers?
Userspace decide list of placement and kernel can't override it, it has to pick one placement in the list (mostly because of PPC and vertex buffer & swap utility). So this doesn't change from what we have, right now only read buffer can be in vram or gtt.
For space checking, it works pretty much as it does now, we got 2 pool vram+gtt when a new buffer comes in you check if there is enough room for it in any of the pool it's valid to put it in. If there isn't enough then you ask for flush. You first pick the prefered pool and decrease it if enough space, if not then try second pool if any.
Quoted 8 lines
I don't think the API we have is perfect by any means I just don't think investing in tweaking the corners for no real benefit is worth it. Just make the gallium winsys API clean and then you don't need to look at this stuff, and in a year or two a new API that actually provides benefits and speedups after we've learned a bit from this one.
Dave.
My fear here is that we will have to support 3 different API (old DRI1, KMS CS+read/write, KMS CS+newapi). It's painful, of course we could always let the code rot and don't touch it once it works.
GPU are really different from other HW we don't have a "simple" high level common API we can't export to userspace (like network or anyother device i am aware of) :(.
Cheers,
Jerome