Revert "drm/amdkfd: Relocate TBA/TMA to opposite side of VM hole"
Subsystems (3)
PATCH 1/1 Revert "drm/amdkfd: Relocate TBA/TMA to opposite side of VM hole"
That commit causes NULL pointer dereferences in dmesgs when running applications using ROCm, including clinfo, blender, and PyTorch, since v6.6.1. Revert it to fix blender again.
This reverts commit 96c211f1f9ef82183493f4ceed4e347b52849149.
--- drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 26 ++++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +13 -13
@@ -330,12 +330,6 @@ static void kfd_init_apertures_vi(struct kfd_process_device *pdd, uint8_t id) pdd->gpuvm_limit = pdd->dev->kfd->shared_resources.gpuvm_size -1 ; -/* dGPUs: the reserved space for kernel -* before SVM -*/ - pdd->qpd.cwsr_base =SVM_CWSR_BASE ; - pdd->qpd.ib_base =SVM_IB_BASE ; - pdd->scratch_base =MAKE_SCRATCH_APP_BASE_VI (); pdd->scratch_limit =MAKE_SCRATCH_APP_LIMIT (pdd->scratch_base); }
@@ -345,18 +339,18 @@ static void kfd_init_apertures_v9(struct kfd_process_device *pdd, uint8_t id) pdd->lds_base =MAKE_LDS_APP_BASE_V9 (); pdd->lds_limit =MAKE_LDS_APP_LIMIT (pdd->lds_base); - pdd->gpuvm_base =PAGE_SIZE ; +/* Raven needs SVM to support graphic handle, etc. Leave the small +* reserved space before SVM on Raven as well, even though we don't +* have to. +* Set gpuvm_base and gpuvm_limit to CANONICAL addresses so that they +* are used in Thunk to reserve SVM. +*/ + pdd->gpuvm_base =SVM_USER_BASE ; pdd->gpuvm_limit = pdd->dev->kfd->shared_resources.gpuvm_size -1 ; pdd->scratch_base =MAKE_SCRATCH_APP_BASE_V9 (); pdd->scratch_limit =MAKE_SCRATCH_APP_LIMIT (pdd->scratch_base); - -/* -* Place TBA/TMA on opposite side of VM hole to prevent -* stray faults from triggering SVM on these pages. -*/ - pdd->qpd.cwsr_base = pdd->dev->kfd->shared_resources.gpuvm_size; }int kfd_init_apertures (struct kfd_process *process)
@@ -413,6 +407,12 @@ int kfd_init_apertures(struct kfd_process *process)return -EINVAL ; } } + +/* dGPUs: the reserved space for kernel +* before SVM +*/ + pdd->qpd.cwsr_base =SVM_CWSR_BASE ; + pdd->qpd.ib_base =SVM_IB_BASE ; }dev_dbg (kfd_device,"node id %u\n" , id);
Signature · 1 line
2.42.0
+ Jay, Felix
Quoted 73 lines · Kaibo Ma
That commit causes NULL pointer dereferences in dmesgs when running applications using ROCm, including clinfo, blender, and PyTorch, since v6.6.1. Revert it to fix blender again.
This reverts commit
96c211f1f9ef82183493f4ceed4e347b52849149.Signed-off-by Kaibo Ma <[email protected]>--- drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 26 ++++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-)drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +13
-14@@ -330,12 +330,6 @@ static void kfd_init_apertures_vi(struct kfd_process_device *pdd, uint8_t id) pdd->gpuvm_limit = pdd->dev->kfd->shared_resources.gpuvm_size -1 ; -/* dGPUs: the reserved space for kernel -* before SVM -*/ - pdd->qpd.cwsr_base =SVM_CWSR_BASE ; - pdd->qpd.ib_base =SVM_IB_BASE ; - pdd->scratch_base =MAKE_SCRATCH_APP_BASE_VI (); pdd->scratch_limit =MAKE_SCRATCH_APP_LIMIT (pdd->scratch_base); }@@ -345,18 +339,18 @@ static void kfd_init_apertures_v9(struct kfd_process_device *pdd, uint8_t id) pdd->lds_base =MAKE_LDS_APP_BASE_V9 (); pdd->lds_limit =MAKE_LDS_APP_LIMIT (pdd->lds_base); - pdd->gpuvm_base =PAGE_SIZE ; +/* Raven needs SVM to support graphic handle, etc. Leave the small +* reserved space before SVM on Raven as well, even though we don't +* have to. +* Set gpuvm_base and gpuvm_limit to CANONICAL addresses so that they +* are used in Thunk to reserve SVM. +*/ + pdd->gpuvm_base =SVM_USER_BASE ; pdd->gpuvm_limit = pdd->dev->kfd->shared_resources.gpuvm_size -1 ; pdd->scratch_base =MAKE_SCRATCH_APP_BASE_V9 (); pdd->scratch_limit =MAKE_SCRATCH_APP_LIMIT (pdd->scratch_base); - -/* -* Place TBA/TMA on opposite side of VM hole to prevent -* stray faults from triggering SVM on these pages. -*/ - pdd->qpd.cwsr_base = pdd->dev->kfd->shared_resources.gpuvm_size; }int kfd_init_apertures (struct kfd_process *process)@@ -413,6 +407,12 @@ int kfd_init_apertures(struct kfd_process *process)return -EINVAL ; } } + +/* dGPUs: the reserved space for kernel +* before SVM +*/ + pdd->qpd.cwsr_base =SVM_CWSR_BASE ; + pdd->qpd.ib_base =SVM_IB_BASE ; }dev_dbg (kfd_device,"node id %u\n" , id); --2.42.0
Jay Cornwall
Reviewed-byQuoted 72 lines · Alex Deucher
+ Jay, Felix
Quoted 69 lines · Kaibo Ma
That commit causes NULL pointer dereferences in dmesgs when running applications using ROCm, including clinfo, blender, and PyTorch, since v6.6.1. Revert it to fix blender again.
This reverts commit
96c211f1f9ef82183493f4ceed4e347b52849149.Signed-off-by Kaibo Ma <[email protected]>--- drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 26 ++++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-)drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +13
-13@@ -330,12 +330,6 @@ static void kfd_init_apertures_vi(struct kfd_process_device *pdd, uint8_t id) pdd->gpuvm_limit = pdd->dev->kfd->shared_resources.gpuvm_size -1 ; -/* dGPUs: the reserved space for kernel -* before SVM -*/ - pdd->qpd.cwsr_base =SVM_CWSR_BASE ; - pdd->qpd.ib_base =SVM_IB_BASE ; - pdd->scratch_base =MAKE_SCRATCH_APP_BASE_VI (); pdd->scratch_limit =MAKE_SCRATCH_APP_LIMIT (pdd->scratch_base); }@@ -345,18 +339,18 @@ static void kfd_init_apertures_v9(struct kfd_process_device *pdd, uint8_t id) pdd->lds_base =MAKE_LDS_APP_BASE_V9 (); pdd->lds_limit =MAKE_LDS_APP_LIMIT (pdd->lds_base); - pdd->gpuvm_base =PAGE_SIZE ; +/* Raven needs SVM to support graphic handle, etc. Leave the small +* reserved space before SVM on Raven as well, even though we don't +* have to. +* Set gpuvm_base and gpuvm_limit to CANONICAL addresses so that they +* are used in Thunk to reserve SVM. +*/ + pdd->gpuvm_base =SVM_USER_BASE ; pdd->gpuvm_limit = pdd->dev->kfd->shared_resources.gpuvm_size -1 ; pdd->scratch_base =MAKE_SCRATCH_APP_BASE_V9 (); pdd->scratch_limit =MAKE_SCRATCH_APP_LIMIT (pdd->scratch_base); - -/* -* Place TBA/TMA on opposite side of VM hole to prevent -* stray faults from triggering SVM on these pages. -*/ - pdd->qpd.cwsr_base = pdd->dev->kfd->shared_resources.gpuvm_size; }int kfd_init_apertures (struct kfd_process *process)@@ -413,6 +407,12 @@ int kfd_init_apertures(struct kfd_process *process)return -EINVAL ; } } + +/* dGPUs: the reserved space for kernel +* before SVM +*/ + pdd->qpd.cwsr_base =SVM_CWSR_BASE ; + pdd->qpd.ib_base =SVM_IB_BASE ; }
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +0
-1dev_dbg (kfd_device,"node id %u\n" , id); --2.42.0
I saw a segfault issue in Mesa yesterday. Not sure about the others, but I don't know how to make this change while compatibility with older UMDs.
So I agree, let's revert it.
Alex Deucher
AppliedApplied. Thanks!
Alex
Quoted 84 lines · Jay Cornwall
Quoted 72 lines · Alex Deucher
+ Jay, Felix
Quoted 69 lines · Kaibo Ma
That commit causes NULL pointer dereferences in dmesgs when running applications using ROCm, including clinfo, blender, and PyTorch, since v6.6.1. Revert it to fix blender again.
This reverts commit
96c211f1f9ef82183493f4ceed4e347b52849149.Signed-off-by Kaibo Ma <[email protected]>--- drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 26 ++++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-)drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +13
-13@@ -330,12 +330,6 @@ static void kfd_init_apertures_vi(struct kfd_process_device *pdd, uint8_t id) pdd->gpuvm_limit = pdd->dev->kfd->shared_resources.gpuvm_size -1 ; -/* dGPUs: the reserved space for kernel -* before SVM -*/ - pdd->qpd.cwsr_base =SVM_CWSR_BASE ; - pdd->qpd.ib_base =SVM_IB_BASE ; - pdd->scratch_base =MAKE_SCRATCH_APP_BASE_VI (); pdd->scratch_limit =MAKE_SCRATCH_APP_LIMIT (pdd->scratch_base); }@@ -345,18 +339,18 @@ static void kfd_init_apertures_v9(struct kfd_process_device *pdd, uint8_t id) pdd->lds_base =MAKE_LDS_APP_BASE_V9 (); pdd->lds_limit =MAKE_LDS_APP_LIMIT (pdd->lds_base); - pdd->gpuvm_base =PAGE_SIZE ; +/* Raven needs SVM to support graphic handle, etc. Leave the small +* reserved space before SVM on Raven as well, even though we don't +* have to. +* Set gpuvm_base and gpuvm_limit to CANONICAL addresses so that they +* are used in Thunk to reserve SVM. +*/ + pdd->gpuvm_base =SVM_USER_BASE ; pdd->gpuvm_limit = pdd->dev->kfd->shared_resources.gpuvm_size -1 ; pdd->scratch_base =MAKE_SCRATCH_APP_BASE_V9 (); pdd->scratch_limit =MAKE_SCRATCH_APP_LIMIT (pdd->scratch_base); - -/* -* Place TBA/TMA on opposite side of VM hole to prevent -* stray faults from triggering SVM on these pages. -*/ - pdd->qpd.cwsr_base = pdd->dev->kfd->shared_resources.gpuvm_size; }int kfd_init_apertures (struct kfd_process *process)@@ -413,6 +407,12 @@ int kfd_init_apertures(struct kfd_process *process)return -EINVAL ; } } + +/* dGPUs: the reserved space for kernel +* before SVM +*/ + pdd->qpd.cwsr_base =SVM_CWSR_BASE ; + pdd->qpd.ib_base =SVM_IB_BASE ; }Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +0
-1dev_dbg (kfd_device,"node id %u\n" , id); --2.42.0
I saw a segfault issue in Mesa yesterday. Not sure about the others, but I don't know how to make this change while compatibility with older UMDs.
So I agree, let's revert it.
Reviewed-by Jay Cornwall <[email protected]>
Quoted 72 lines · Jay Cornwall
Quoted 71 lines · Alex Deucher
+ Jay, Felix
Quoted 68 lines · Kaibo Ma
That commit causes NULL pointer dereferences in dmesgs when running applications using ROCm, including clinfo, blender, and PyTorch, since v6.6.1. Revert it to fix blender again.
This reverts commit
96c211f1f9ef82183493f4ceed4e347b52849149.Signed-off-by Kaibo Ma <[email protected]>--- drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 26 ++++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-)drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +13
-13@@ -330,12 +330,6 @@ static void kfd_init_apertures_vi(struct kfd_process_device *pdd, uint8_t id) pdd->gpuvm_limit = pdd->dev->kfd->shared_resources.gpuvm_size -1 ; -/* dGPUs: the reserved space for kernel -* before SVM -*/ - pdd->qpd.cwsr_base =SVM_CWSR_BASE ; - pdd->qpd.ib_base =SVM_IB_BASE ; - pdd->scratch_base =MAKE_SCRATCH_APP_BASE_VI (); pdd->scratch_limit =MAKE_SCRATCH_APP_LIMIT (pdd->scratch_base); }@@ -345,18 +339,18 @@ static void kfd_init_apertures_v9(struct kfd_process_device *pdd, uint8_t id) pdd->lds_base =MAKE_LDS_APP_BASE_V9 (); pdd->lds_limit =MAKE_LDS_APP_LIMIT (pdd->lds_base); - pdd->gpuvm_base =PAGE_SIZE ; +/* Raven needs SVM to support graphic handle, etc. Leave the small +* reserved space before SVM on Raven as well, even though we don't +* have to. +* Set gpuvm_base and gpuvm_limit to CANONICAL addresses so that they +* are used in Thunk to reserve SVM. +*/ + pdd->gpuvm_base =SVM_USER_BASE ; pdd->gpuvm_limit = pdd->dev->kfd->shared_resources.gpuvm_size -1 ; pdd->scratch_base =MAKE_SCRATCH_APP_BASE_V9 (); pdd->scratch_limit =MAKE_SCRATCH_APP_LIMIT (pdd->scratch_base); - -/* -* Place TBA/TMA on opposite side of VM hole to prevent -* stray faults from triggering SVM on these pages. -*/ - pdd->qpd.cwsr_base = pdd->dev->kfd->shared_resources.gpuvm_size; }int kfd_init_apertures (struct kfd_process *process)@@ -413,6 +407,12 @@ int kfd_init_apertures(struct kfd_process *process)return -EINVAL ; } } + +/* dGPUs: the reserved space for kernel +* before SVM +*/ + pdd->qpd.cwsr_base =SVM_CWSR_BASE ; + pdd->qpd.ib_base =SVM_IB_BASE ; }
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +0
-1dev_dbg (kfd_device,"node id %u\n" , id); --2.42.0
I saw a segfault issue in Mesa yesterday. Not sure about the others, but I don't know how to make this change while compatibility with older UMDs.
A segfault in Mesa seems to be a different issue from what's mentioned in the commit message. I'd let Christian or Marek comment on compatibility with graphics UMDs. I'm not sure why this patch would affect them at all.
Looking at the logs in the tickets, it looks like a fence reference counting error. I don't see how Jay's patch could have caused that. I made another change in that code recently that could make a difference for this issue:
commit 8f08c5b24ced1be7eb49692e4816c1916233c79b
Author: Felix Kuehling <[email protected]> Date: Fri Oct 27 18:21:55 2023 -0400
drm/amdkfd: Run restore_workers on freezable WQs
Make restore workers freezable so we don't have to explicitly
flush them
in suspend and GPU reset code paths, and we don't accidentally
try to
restore BOs while the GPU is suspended. Not having to flush
restore_work
also helps avoid lock/fence dependencies in the GPU reset case
where we're
not allowed to wait for fences.
A side effect of this is, that we can now have multiple
concurrent threads
trying to signal the same eviction fence. Rework eviction fence
signaling
and replacement to account for that.
The GPU reset path can no longer rely on restore_process_worker
to resume
queues because evict/restore workers can run independently of
it. Instead
call a new restore_process_helper directly.
This is an RFC and request for testing.
v2:
- Reworked eviction fence signaling
- Introduced restore_process_helper
v3:
- Handle unsignaled eviction fences in restore_process_bos
FWIW, I built a plain 6.6 kernel, and was not able to reproduce the crash with some simple tests.
Regards,
Felix
Quoted 4 lines
So I agree, let's revert it.
Reviewed-by Jay Cornwall <[email protected]>
Quoted 4 lines · Felix Kuehling
A segfault in Mesa seems to be a different issue from what's mentioned in the commit message. I'd let Christian or Marek comment on compatibility with graphics UMDs. I'm not sure why this patch would affect them at all.
I was referencing this issue in OpenCL/OpenGL interop, which certainly looked related:
[ 91.769002] amdgpu 0000:0a:00.0: amdgpu: bo 000000009bba4692 va 0x0800000000-0x08000001ff conflict with 0x0800000000-0x0800000002 [ 91.769141] ocltst[2781]: segfault at b2 ip 00007f3fb90a7c39 sp 00007ffd3c011ba0 error 4 in radeonsi_dri.so[7f3fb888e000+1196000] likely on CPU 15 (core 7, socket 0)
Quoted 60 lines
Looking at the logs in the tickets, it looks like a fence reference counting error. I don't see how Jay's patch could have caused that. I made another change in that code recently that could make a difference for this issue:
commit
8f08c5b24ced1be7eb49692e4816c1916233c79b
Author: Felix Kuehling <[email protected]> Date: Fri Oct 27 18:21:55 2023 -0400drm/amdkfd: Run
restore_workerson freezable WQsMake restore workers freezable so we don't have to explicitly
flush them
in suspend and GPU reset code paths, and we don't accidentally
try to
restore BOs while the GPU is suspended. Not having to flush
restore_work
also helps avoid lock/fence dependencies in the GPU reset case
where we're
not allowed to wait for fences.A side effect of this is, that we can now have multiple
concurrent threads
trying to signal the same eviction fence. Rework eviction fence
signaling
and replacement to account for that.The GPU reset path can no longer rely on
restore_process_worker
to resume
queues because evict/restore workers can run independently of
it. Instead
call a newrestore_process_helperdirectly.This is an RFC and request for testing.
v2:
- Reworked eviction fence signaling
- Introducedrestore_process_helperv3:
- Handle unsignaled eviction fences inrestore_process_bosSigned-off-by Felix Kuehling <[email protected]>Acked-by Christian König <[email protected]>Tested-by Emily Deng <[email protected]>Signed-off-by Alex Deucher <[email protected]>FWIW, I built a plain 6.6 kernel, and was not able to reproduce the crash with some simple tests.
Regards,
FelixQuoted 4 lines
So I agree, let's revert it.
Reviewed-by Jay Cornwall <[email protected]>
Hi,
I have received information that the original commit makes all 32-bit userspace VA allocations fail, so UMDs like Mesa can't even initialize and they either crash or fail to load. If TBA/TMA was relocated to the 32-bit address range, it would explain why UMDs can't allocate anything in that range.
Marek
Quoted 74 lines · Jay Cornwall
Quoted 4 lines · Felix Kuehling
A segfault in Mesa seems to be a different issue from what's mentioned in the commit message. I'd let Christian or Marek comment on compatibility with graphics UMDs. I'm not sure why this patch would affect them at all.
I was referencing this issue in OpenCL/OpenGL interop, which certainly looked related:
[ 91.769002] amdgpu 0000:0a:00.0: amdgpu: bo 000000009bba4692 va 0x0800000000-0x08000001ff conflict with 0x0800000000-0x0800000002 [ 91.769141] ocltst[2781]: segfault at b2 ip 00007f3fb90a7c39 sp 00007ffd3c011ba0 error 4 in radeonsi_dri.so[7f3fb888e000+1196000] likely on CPU 15 (core 7, socket 0)Quoted 60 lines
Looking at the logs in the tickets, it looks like a fence reference counting error. I don't see how Jay's patch could have caused that. I made another change in that code recently that could make a difference for this issue:
commit
8f08c5b24ced1be7eb49692e4816c1916233c79b
Author: Felix Kuehling <[email protected]>
Date: Fri Oct 27 18:21:55 2023 -0400drm/amdkfd: Run
restore_workerson freezable WQsMake restore workers freezable so we don't have to explicitly
flush them
in suspend and GPU reset code paths, and we don't accidentally
try to
restore BOs while the GPU is suspended. Not having to flush
restore_work
also helps avoid lock/fence dependencies in the GPU reset case
where we're
not allowed to wait for fences.A side effect of this is, that we can now have multiple
concurrent threads
trying to signal the same eviction fence. Rework eviction fence
signaling
and replacement to account for that.The GPU reset path can no longer rely on
restore_process_worker
to resume
queues because evict/restore workers can run independently of
it. Instead
call a newrestore_process_helperdirectly.This is an RFC and request for testing.
v2:
- Reworked eviction fence signaling
- Introducedrestore_process_helperv3:
- Handle unsignaled eviction fences inrestore_process_bosSigned-off-by Felix Kuehling <[email protected]>Acked-by Christian König <[email protected]>Tested-by Emily Deng <[email protected]>Signed-off-by Alex Deucher <[email protected]>FWIW, I built a plain 6.6 kernel, and was not able to reproduce the crash with some simple tests.
Regards,
FelixQuoted 4 lines
So I agree, let's revert it.
Reviewed-by Jay Cornwall <[email protected]>
Felix Kuehling
maintainerTBA/TMA were relocated to the upper half of the canonical address space. I don't think that qualifies as 32-bit by definition. But maybe you're using a different definition.
That said, if Mesa manages its own virtual address space in user mode, and KFD maps the TMA/TBA at an address that Mesa believes to be free, I can see how that would lead to problems.
That said, the fence refcount bug is another problem that may have been exposed by the way that a crashing Mesa application shuts down. Reverting Jay's patch certainly didn't fix that, but only hides the problem.
Regards,
Felix
Quoted 80 lines · Marek Olšák
Hi,
I have received information that the original commit makes all 32-bit userspace VA allocations fail, so UMDs like Mesa can't even initialize and they either crash or fail to load. If TBA/TMA was relocated to the 32-bit address range, it would explain why UMDs can't allocate anything in that range.
Marek
Quoted 69 lines · Jay Cornwall
Quoted 4 lines · Felix Kuehling
A segfault in Mesa seems to be a different issue from what's mentioned in the commit message. I'd let Christian or Marek comment on compatibility with graphics UMDs. I'm not sure why this patch would affect them at all.
I was referencing this issue in OpenCL/OpenGL interop, which certainly looked related:
[ 91.769002] amdgpu 0000:0a:00.0: amdgpu: bo 000000009bba4692 va 0x0800000000-0x08000001ff conflict with 0x0800000000-0x0800000002 [ 91.769141] ocltst[2781]: segfault at b2 ip 00007f3fb90a7c39 sp 00007ffd3c011ba0 error 4 in radeonsi_dri.so[7f3fb888e000+1196000] likely on CPU 15 (core 7, socket 0)Quoted 58 lines
Looking at the logs in the tickets, it looks like a fence reference counting error. I don't see how Jay's patch could have caused that. I made another change in that code recently that could make a difference for this issue:
commit
8f08c5b24ced1be7eb49692e4816c1916233c79b
Author: Felix Kuehling <[email protected]>
Date: Fri Oct 27 18:21:55 2023 -0400drm/amdkfd: Run
restore_workerson freezable WQsMake restore workers freezable so we don't have to explicitly
flush them
in suspend and GPU reset code paths, and we don't accidentally
try to
restore BOs while the GPU is suspended. Not having to flush
restore_work
also helps avoid lock/fence dependencies in the GPU reset case
where we're
not allowed to wait for fences.A side effect of this is, that we can now have multiple
concurrent threads
trying to signal the same eviction fence. Rework eviction fence
signaling
and replacement to account for that.The GPU reset path can no longer rely on
restore_process_worker
to resume
queues because evict/restore workers can run independently of
it. Instead
call a newrestore_process_helperdirectly.This is an RFC and request for testing.
v2:
- Reworked eviction fence signaling
- Introducedrestore_process_helperv3:
- Handle unsignaled eviction fences inrestore_process_bosSigned-off-by Felix Kuehling <[email protected]>Acked-by Christian König <[email protected]>Tested-by Emily Deng <[email protected]>Signed-off-by Alex Deucher <[email protected]>FWIW, I built a plain 6.6 kernel, and was not able to reproduce the crash with some simple tests.
Regards,
FelixQuoted 3 lines
So I agree, let's revert it.
Reviewed-by Jay Cornwall <[email protected]>
The 32-bit address space means the high 32 bits are constant and predetermined and it's definitely somewhere in the upper range of the address space. If ROCm or KFD occupy that space, even accidentally, other UMDs that use libdrm for VA allocation won't be able to start. The VA range allocator is in libdrm.
Marek
Quoted 108 lines · On Fri, Jan 5, 2024, 15:20 Felix Kuehling <[email protected]> wrote:
TBA/TMA were relocated to the upper half of the canonical address space. I don't think that qualifies as 32-bit by definition. But maybe you're using a different definition.
That said, if Mesa manages its own virtual address space in user mode, and KFD maps the TMA/TBA at an address that Mesa believes to be free, I can see how that would lead to problems.
That said, the fence refcount bug is another problem that may have been exposed by the way that a crashing Mesa application shuts down. Reverting Jay's patch certainly didn't fix that, but only hides the problem.
Regards,
FelixQuoted 11 lines · Marek Olšák
Hi,
I have received information that the original commit makes all 32-bit userspace VA allocations fail, so UMDs like Mesa can't even initialize and they either crash or fail to load. If TBA/TMA was relocated to the 32-bit address range, it would explain why UMDs can't allocate anything in that range.
Marek
On Wed, Jan 3, 2024 at 2:50 PM Jay Cornwall <[email protected]>
Quoted 7 lines · wrote:
Quoted 7 lines · Felix Kuehling
Quoted 5 lines
A segfault in Mesa seems to be a different issue from what's mentioned in the commit message. I'd let Christian or Marek comment on compatibility with graphics UMDs. I'm not sure why this patch would affect them at all.
I was referencing this issue in OpenCL/OpenGL interop, which certainly
looked related:
Quoted 2 lines
Quoted 2 lines
[ 91.769002] amdgpu 0000:0a:00.0: amdgpu: bo 000000009bba4692 va0x0800000000-0x08000001ff conflict with 0x0800000000-0x0800000002
Quoted 1 line
Quoted 1 line
[ 91.769141] ocltst[2781]: segfault at b2 ip 00007f3fb90a7c39 sp
00007ffd3c011ba0error 4 in radeonsi_dri.so[7f3fb888e000+1196000] likely on CPU 15 (core 7, socket 0)Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Looking at the logs in the tickets, it looks like a fence reference counting error. I don't see how Jay's patch could have caused that. I made another change in that code recently that could make a difference for this issue:
commit 8f08c5b24ced1be7eb49692e4816c1916233c79b Author: Felix Kuehling <[email protected]> Date: Fri Oct 27 18:21:55 2023 -0400Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drm/amdkfd: Run
restore_workerson freezable WQsMake restore workers freezable so we don't have to explicitly
flush them
in suspend and GPU reset code paths, and we don'taccidentally
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
try to
restore BOs while the GPU is suspended. Not having to flush
restore_work
also helps avoid lock/fence dependencies in the GPU resetcase
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
where we're
not allowed to wait for fences.A side effect of this is, that we can now have multiple
concurrent threads
trying to signal the same eviction fence. Rework evictionfence
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
signaling
and replacement to account for that.The GPU reset path can no longer rely on
restore_process_workerQuoted 30 lines
Quoted 30 lines
Quoted 30 lines
to resume
queues because evict/restore workers can run independently of
it. Instead
call a newrestore_process_helperdirectly.This is an RFC and request for testing.
v2:
- Reworked eviction fence signaling
- Introducedrestore_process_helperv3:
- Handle unsignaled eviction fences inrestore_process_bosSigned-off-by Felix Kuehling <[email protected]>Acked-by Christian König <[email protected]>Tested-by Emily Deng <[email protected]>Signed-off-by Alex Deucher <[email protected]>FWIW, I built a plain 6.6 kernel, and was not able to reproduce the crash with some simple tests.
Regards,
FelixQuoted 3 lines
So I agree, let's revert it.
Reviewed-by Jay Cornwall <[email protected]>
It looks like this would cause failures even with regular 64-bit allocations because the virtual address range allocator in libdrm asks the kernel what ranges of addresses are free, and the kernel doesn't exclude the KFD allocation from that.
Basically, no VM allocations can be done by the kernel outside the ranges reserved for the kernel.
Marek
Quoted 104 lines · Marek Olšák
The 32-bit address space means the high 32 bits are constant and predetermined and it's definitely somewhere in the upper range of the address space. If ROCm or KFD occupy that space, even accidentally, other UMDs that use libdrm for VA allocation won't be able to start. The VA range allocator is in libdrm.
Marek
Quoted 98 lines · On Fri, Jan 5, 2024, 15:20 Felix Kuehling <[email protected]> wrote:
TBA/TMA were relocated to the upper half of the canonical address space. I don't think that qualifies as 32-bit by definition. But maybe you're using a different definition.
That said, if Mesa manages its own virtual address space in user mode, and KFD maps the TMA/TBA at an address that Mesa believes to be free, I can see how that would lead to problems.
That said, the fence refcount bug is another problem that may have been exposed by the way that a crashing Mesa application shuts down. Reverting Jay's patch certainly didn't fix that, but only hides the problem.
Regards,
FelixQuoted 80 lines · Marek Olšák
Hi,
I have received information that the original commit makes all 32-bit userspace VA allocations fail, so UMDs like Mesa can't even initialize and they either crash or fail to load. If TBA/TMA was relocated to the 32-bit address range, it would explain why UMDs can't allocate anything in that range.
Marek
Quoted 69 lines · Jay Cornwall
Quoted 4 lines · Felix Kuehling
A segfault in Mesa seems to be a different issue from what's mentioned in the commit message. I'd let Christian or Marek comment on compatibility with graphics UMDs. I'm not sure why this patch would affect them at all.
I was referencing this issue in OpenCL/OpenGL interop, which certainly looked related:
[ 91.769002] amdgpu 0000:0a:00.0: amdgpu: bo 000000009bba4692 va 0x0800000000-0x08000001ff conflict with 0x0800000000-0x0800000002 [ 91.769141] ocltst[2781]: segfault at b2 ip 00007f3fb90a7c39 sp 00007ffd3c011ba0 error 4 in radeonsi_dri.so[7f3fb888e000+1196000] likely on CPU 15 (core 7, socket 0)Quoted 58 lines
Looking at the logs in the tickets, it looks like a fence reference counting error. I don't see how Jay's patch could have caused that. I made another change in that code recently that could make a difference for this issue:
commit
8f08c5b24ced1be7eb49692e4816c1916233c79b
Author: Felix Kuehling <[email protected]>
Date: Fri Oct 27 18:21:55 2023 -0400drm/amdkfd: Run
restore_workerson freezable WQsMake restore workers freezable so we don't have to explicitly
flush them
in suspend and GPU reset code paths, and we don't accidentally
try to
restore BOs while the GPU is suspended. Not having to flush
restore_work
also helps avoid lock/fence dependencies in the GPU reset case
where we're
not allowed to wait for fences.A side effect of this is, that we can now have multiple
concurrent threads
trying to signal the same eviction fence. Rework eviction fence
signaling
and replacement to account for that.The GPU reset path can no longer rely on
restore_process_worker
to resume
queues because evict/restore workers can run independently of
it. Instead
call a newrestore_process_helperdirectly.This is an RFC and request for testing.
v2:
- Reworked eviction fence signaling
- Introducedrestore_process_helperv3:
- Handle unsignaled eviction fences inrestore_process_bosSigned-off-by Felix Kuehling <[email protected]>Acked-by Christian König <[email protected]>Tested-by Emily Deng <[email protected]>Signed-off-by Alex Deucher <[email protected]>FWIW, I built a plain 6.6 kernel, and was not able to reproduce the crash with some simple tests.
Regards,
FelixQuoted 3 lines
So I agree, let's revert it.
Reviewed-by Jay Cornwall <[email protected]>
Felix Kuehling
maintainer[+Christian]
I'm looking into virtual address reservations in amdgpu and what's reported by AMDGPU_INFO_DEV_INFO. So far I only found AMDGPU_VA_RESERVED_SIZE, which is reserved both at the start of the lower virtual address range and the end of the upper virtual address range.
The reservation size is currently 2MB. The upper reservation is used by the CSA, which is currently 128KB. That leaves plenty of room for the TMA/TBA above the CSA. I'll create a patch for that.
Regards,
Felix
Quoted 113 lines · Marek Olšák
It looks like this would cause failures even with regular 64-bit allocations because the virtual address range allocator in libdrm asks the kernel what ranges of addresses are free, and the kernel doesn't exclude the KFD allocation from that.
Basically, no VM allocations can be done by the kernel outside the ranges reserved for the kernel.
Marek
Quoted 102 lines · Marek Olšák
The 32-bit address space means the high 32 bits are constant and predetermined and it's definitely somewhere in the upper range of the address space. If ROCm or KFD occupy that space, even accidentally, other UMDs that use libdrm for VA allocation won't be able to start. The VA range allocator is in libdrm.
Marek
Quoted 97 lines · On Fri, Jan 5, 2024, 15:20 Felix Kuehling <[email protected]> wrote:
TBA/TMA were relocated to the upper half of the canonical address space. I don't think that qualifies as 32-bit by definition. But maybe you're using a different definition.
That said, if Mesa manages its own virtual address space in user mode, and KFD maps the TMA/TBA at an address that Mesa believes to be free, I can see how that would lead to problems.
That said, the fence refcount bug is another problem that may have been exposed by the way that a crashing Mesa application shuts down. Reverting Jay's patch certainly didn't fix that, but only hides the problem.
Regards,
FelixQuoted 80 lines · Marek Olšák
Hi,
I have received information that the original commit makes all 32-bit userspace VA allocations fail, so UMDs like Mesa can't even initialize and they either crash or fail to load. If TBA/TMA was relocated to the 32-bit address range, it would explain why UMDs can't allocate anything in that range.
Marek
Quoted 69 lines · Jay Cornwall
Quoted 4 lines · Felix Kuehling
A segfault in Mesa seems to be a different issue from what's mentioned in the commit message. I'd let Christian or Marek comment on compatibility with graphics UMDs. I'm not sure why this patch would affect them at all.
I was referencing this issue in OpenCL/OpenGL interop, which certainly looked related:
[ 91.769002] amdgpu 0000:0a:00.0: amdgpu: bo 000000009bba4692 va 0x0800000000-0x08000001ff conflict with 0x0800000000-0x0800000002 [ 91.769141] ocltst[2781]: segfault at b2 ip 00007f3fb90a7c39 sp 00007ffd3c011ba0 error 4 in radeonsi_dri.so[7f3fb888e000+1196000] likely on CPU 15 (core 7, socket 0)Quoted 58 lines
Looking at the logs in the tickets, it looks like a fence reference counting error. I don't see how Jay's patch could have caused that. I made another change in that code recently that could make a difference for this issue:
commit
8f08c5b24ced1be7eb49692e4816c1916233c79b
Author: Felix Kuehling <[email protected]>
Date: Fri Oct 27 18:21:55 2023 -0400drm/amdkfd: Run
restore_workerson freezable WQsMake restore workers freezable so we don't have to explicitly
flush them
in suspend and GPU reset code paths, and we don't accidentally
try to
restore BOs while the GPU is suspended. Not having to flush
restore_work
also helps avoid lock/fence dependencies in the GPU reset case
where we're
not allowed to wait for fences.A side effect of this is, that we can now have multiple
concurrent threads
trying to signal the same eviction fence. Rework eviction fence
signaling
and replacement to account for that.The GPU reset path can no longer rely on
restore_process_worker
to resume
queues because evict/restore workers can run independently of
it. Instead
call a newrestore_process_helperdirectly.This is an RFC and request for testing.
v2:
- Reworked eviction fence signaling
- Introducedrestore_process_helperv3:
- Handle unsignaled eviction fences inrestore_process_bosSigned-off-by Felix Kuehling <[email protected]>Acked-by Christian König <[email protected]>Tested-by Emily Deng <[email protected]>Signed-off-by Alex Deucher <[email protected]>FWIW, I built a plain 6.6 kernel, and was not able to reproduce the crash with some simple tests.
Regards,
FelixQuoted 3 lines
So I agree, let's revert it.
Reviewed-by Jay Cornwall <[email protected]>
Christian König
maintainerArun and I have been working on a patch to split up AMDGPU_VA_RESERVED_SIZE into two separate defines.
I assumed that Arun landed this by now, but that doesn't seem to be the case.
Going to take another look, but yeah the CSA area isn't large enough to fill the upper reserved area so moving the TMA/TBA there should work.
Regards,
Christian.
Am 11.01.24 um 23:41 schrieb Felix Kuehling:
Quoted 124 lines
[+Christian]
I'm looking into virtual address reservations in amdgpu and what's reported by
AMDGPU_INFO_DEV_INFO. So far I only foundAMDGPU_VA_RESERVED_SIZE, which is reserved both at the start of the lower virtual address range and the end of the upper virtual address range.The reservation size is currently 2MB. The upper reservation is used by the CSA, which is currently 128KB. That leaves plenty of room for the TMA/TBA above the CSA. I'll create a patch for that.
Regards,
FelixQuoted 113 lines · Marek Olšák
It looks like this would cause failures even with regular 64-bit allocations because the virtual address range allocator in libdrm asks the kernel what ranges of addresses are free, and the kernel doesn't exclude the KFD allocation from that.
Basically, no VM allocations can be done by the kernel outside the ranges reserved for the kernel.
Marek
Quoted 102 lines · Marek Olšák
The 32-bit address space means the high 32 bits are constant and predetermined and it's definitely somewhere in the upper range of the address space. If ROCm or KFD occupy that space, even accidentally, other UMDs that use libdrm for VA allocation won't be able to start. The VA range allocator is in libdrm.
Marek
Quoted 97 lines · On Fri, Jan 5, 2024, 15:20 Felix Kuehling <[email protected]> wrote:
TBA/TMA were relocated to the upper half of the canonical address space. I don't think that qualifies as 32-bit by definition. But maybe you're using a different definition.
That said, if Mesa manages its own virtual address space in user mode, and KFD maps the TMA/TBA at an address that Mesa believes to be free, I can see how that would lead to problems.
That said, the fence refcount bug is another problem that may have been exposed by the way that a crashing Mesa application shuts down. Reverting Jay's patch certainly didn't fix that, but only hides the problem.
Regards,
FelixQuoted 80 lines · Marek Olšák
Hi,
I have received information that the original commit makes all 32-bit userspace VA allocations fail, so UMDs like Mesa can't even initialize and they either crash or fail to load. If TBA/TMA was relocated to the 32-bit address range, it would explain why UMDs can't allocate anything in that range.
Marek
Quoted 69 lines · Jay Cornwall
Quoted 4 lines · Felix Kuehling
A segfault in Mesa seems to be a different issue from what's mentioned in the commit message. I'd let Christian or Marek comment on compatibility with graphics UMDs. I'm not sure why this patch would affect them at all.
I was referencing this issue in OpenCL/OpenGL interop, which certainly looked related:
[ 91.769002] amdgpu 0000:0a:00.0: amdgpu: bo
000000009bba4692va 0x0800000000-0x08000001ff conflict with 0x0800000000-0x0800000002
[ 91.769141] ocltst[2781]: segfault at b2 ip00007f3fb90a7c39sp00007ffd3c011ba0error 4 in radeonsi_dri.so[7f3fb888e000+1196000] likely on CPU 15 (core 7, socket 0)Quoted 58 lines
Looking at the logs in the tickets, it looks like a fence reference counting error. I don't see how Jay's patch could have caused that. I made another change in that code recently that could make a difference for this issue:
commit
8f08c5b24ced1be7eb49692e4816c1916233c79b
Author: Felix Kuehling <[email protected]> Date: Fri Oct 27 18:21:55 2023 -0400drm/amdkfd: Run
restore_workerson freezable WQsMake restore workers freezable so we don't have to explicitly flush them
in suspend and GPU reset code paths, and we don't accidentally try to
restore BOs while the GPU is suspended. Not having to flushrestore_work
also helps avoid lock/fence dependencies in the GPU reset case where we're
not allowed to wait for fences.A side effect of this is, that we can now have multiple concurrent threads
trying to signal the same eviction fence. Rework eviction fence signaling
and replacement to account for that.The GPU reset path can no longer rely on
restore_process_workerto resume
queues because evict/restore workers can run independently of it. Instead
call a newrestore_process_helperdirectly.This is an RFC and request for testing.
v2:
- Reworked eviction fence signaling
- Introducedrestore_process_helperv3:
- Handle unsignaled eviction fences inrestore_process_bosSigned-off-by Felix Kuehling <[email protected]>Acked-by Christian König <[email protected]>Tested-by Emily Deng <[email protected]>Signed-off-by Alex Deucher <[email protected]>FWIW, I built a plain 6.6 kernel, and was not able to reproduce the crash with some simple tests.
Regards,
FelixQuoted 3 lines
So I agree, let's revert it.
Reviewed-by Jay Cornwall <[email protected]>