Skip to contentSkip to conversation

Revert "drm/amdkfd: Relocate TBA/TMA to opposite side of VM hole"

PATCH Applied Kaibo Ma 6 participants 12 messages Jan 3, 2024 – Jan 12, 2024

Subsystems (3)
lore
Share & tools
lore

Kaibo Ma

author
#0 lore raw

PATCH 1/1 Revert "drm/amdkfd: Relocate TBA/TMA to opposite side of VM hole"

That commit causes NULL pointer dereferences in dmesgs when running applications using ROCm, including clinfo, blender, and PyTorch, since v6.6.1. Revert it to fix blender again.

This reverts commit 96c211f1f9ef82183493f4ceed4e347b52849149.

---
 drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 26 ++++++++++----------
 1 file changed, 13 insertions(+), 13 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +13 -13
index 62b205dac..6604a3f99 100644
@@ -330,12 +330,6 @@ static void kfd_init_apertures_vi(struct kfd_process_device *pdd, uint8_t id)
 	pdd->gpuvm_limit =
 		pdd->dev->kfd->shared_resources.gpuvm_size - 1;
 
-	/* dGPUs: the reserved space for kernel
-	 * before SVM
-	 */
-	pdd->qpd.cwsr_base = SVM_CWSR_BASE;
-	pdd->qpd.ib_base = SVM_IB_BASE;
-
 	pdd->scratch_base = MAKE_SCRATCH_APP_BASE_VI();
 	pdd->scratch_limit = MAKE_SCRATCH_APP_LIMIT(pdd->scratch_base);
 }
@@ -345,18 +339,18 @@ static void kfd_init_apertures_v9(struct kfd_process_device *pdd, uint8_t id)
 	pdd->lds_base = MAKE_LDS_APP_BASE_V9();
 	pdd->lds_limit = MAKE_LDS_APP_LIMIT(pdd->lds_base);
 
-	pdd->gpuvm_base = PAGE_SIZE;
+        /* Raven needs SVM to support graphic handle, etc. Leave the small
+         * reserved space before SVM on Raven as well, even though we don't
+         * have to.
+         * Set gpuvm_base and gpuvm_limit to CANONICAL addresses so that they
+         * are used in Thunk to reserve SVM.
+         */
+        pdd->gpuvm_base = SVM_USER_BASE;
 	pdd->gpuvm_limit =
 		pdd->dev->kfd->shared_resources.gpuvm_size - 1;
 
 	pdd->scratch_base = MAKE_SCRATCH_APP_BASE_V9();
 	pdd->scratch_limit = MAKE_SCRATCH_APP_LIMIT(pdd->scratch_base);
-
-	/*
-	 * Place TBA/TMA on opposite side of VM hole to prevent
-	 * stray faults from triggering SVM on these pages.
-	 */
-	pdd->qpd.cwsr_base = pdd->dev->kfd->shared_resources.gpuvm_size;
 }
 
 int kfd_init_apertures(struct kfd_process *process)
@@ -413,6 +407,12 @@ int kfd_init_apertures(struct kfd_process *process)
 					return -EINVAL;
 				}
 			}
+
+                        /* dGPUs: the reserved space for kernel
+                         * before SVM
+                         */
+                        pdd->qpd.cwsr_base = SVM_CWSR_BASE;
+                        pdd->qpd.ib_base = SVM_IB_BASE;
 		}
 
 		dev_dbg(kfd_device, "node id %u\n", id);
Signature · 1 line
2.42.0
#1 lore raw

+ Jay, Felix

Quoted 73 lines · Kaibo Ma

That commit causes NULL pointer dereferences in dmesgs when running applications using ROCm, including clinfo, blender, and PyTorch, since v6.6.1. Revert it to fix blender again.

This reverts commit 96c211f1f9ef82183493f4ceed4e347b52849149.

---
 drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 26 ++++++++++----------
 1 file changed, 13 insertions(+), 13 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +13 -14
index 62b205dac..6604a3f99 100644
@@ -330,12 +330,6 @@ static void kfd_init_apertures_vi(struct kfd_process_device *pdd, uint8_t id)
        pdd->gpuvm_limit =
                pdd->dev->kfd->shared_resources.gpuvm_size - 1;
 
-       /* dGPUs: the reserved space for kernel
-        * before SVM
-        */
-       pdd->qpd.cwsr_base = SVM_CWSR_BASE;
-       pdd->qpd.ib_base = SVM_IB_BASE;
-
        pdd->scratch_base = MAKE_SCRATCH_APP_BASE_VI();
        pdd->scratch_limit = MAKE_SCRATCH_APP_LIMIT(pdd->scratch_base);
 }
@@ -345,18 +339,18 @@ static void kfd_init_apertures_v9(struct kfd_process_device *pdd, uint8_t id)
        pdd->lds_base = MAKE_LDS_APP_BASE_V9();
        pdd->lds_limit = MAKE_LDS_APP_LIMIT(pdd->lds_base);
 
-       pdd->gpuvm_base = PAGE_SIZE;
+        /* Raven needs SVM to support graphic handle, etc. Leave the small
+         * reserved space before SVM on Raven as well, even though we don't
+         * have to.
+         * Set gpuvm_base and gpuvm_limit to CANONICAL addresses so that they
+         * are used in Thunk to reserve SVM.
+         */
+        pdd->gpuvm_base = SVM_USER_BASE;
        pdd->gpuvm_limit =
                pdd->dev->kfd->shared_resources.gpuvm_size - 1;
 
        pdd->scratch_base = MAKE_SCRATCH_APP_BASE_V9();
        pdd->scratch_limit = MAKE_SCRATCH_APP_LIMIT(pdd->scratch_base);
-
-       /*
-        * Place TBA/TMA on opposite side of VM hole to prevent
-        * stray faults from triggering SVM on these pages.
-        */
-       pdd->qpd.cwsr_base = pdd->dev->kfd->shared_resources.gpuvm_size;
 }
 
 int kfd_init_apertures(struct kfd_process *process)
@@ -413,6 +407,12 @@ int kfd_init_apertures(struct kfd_process *process)
                                        return -EINVAL;
                                }
                        }
+
+                        /* dGPUs: the reserved space for kernel
+                         * before SVM
+                         */
+                        pdd->qpd.cwsr_base = SVM_CWSR_BASE;
+                        pdd->qpd.ib_base = SVM_IB_BASE;
                }
 
                dev_dbg(kfd_device, "node id %u\n", id);
--

2.42.0

Reviewed-by
#2 lore raw
Quoted 72 lines · Alex Deucher

+ Jay, Felix

Quoted 69 lines · Kaibo Ma

That commit causes NULL pointer dereferences in dmesgs when running applications using ROCm, including clinfo, blender, and PyTorch, since v6.6.1. Revert it to fix blender again.

This reverts commit 96c211f1f9ef82183493f4ceed4e347b52849149.

---
 drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 26 ++++++++++----------
 1 file changed, 13 insertions(+), 13 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +13 -13
index 62b205dac..6604a3f99 100644
@@ -330,12 +330,6 @@ static void kfd_init_apertures_vi(struct kfd_process_device *pdd, uint8_t id)
        pdd->gpuvm_limit =
                pdd->dev->kfd->shared_resources.gpuvm_size - 1;
 
-       /* dGPUs: the reserved space for kernel
-        * before SVM
-        */
-       pdd->qpd.cwsr_base = SVM_CWSR_BASE;
-       pdd->qpd.ib_base = SVM_IB_BASE;
-
        pdd->scratch_base = MAKE_SCRATCH_APP_BASE_VI();
        pdd->scratch_limit = MAKE_SCRATCH_APP_LIMIT(pdd->scratch_base);
 }
@@ -345,18 +339,18 @@ static void kfd_init_apertures_v9(struct kfd_process_device *pdd, uint8_t id)
        pdd->lds_base = MAKE_LDS_APP_BASE_V9();
        pdd->lds_limit = MAKE_LDS_APP_LIMIT(pdd->lds_base);
 
-       pdd->gpuvm_base = PAGE_SIZE;
+        /* Raven needs SVM to support graphic handle, etc. Leave the small
+         * reserved space before SVM on Raven as well, even though we don't
+         * have to.
+         * Set gpuvm_base and gpuvm_limit to CANONICAL addresses so that they
+         * are used in Thunk to reserve SVM.
+         */
+        pdd->gpuvm_base = SVM_USER_BASE;
        pdd->gpuvm_limit =
                pdd->dev->kfd->shared_resources.gpuvm_size - 1;
 
        pdd->scratch_base = MAKE_SCRATCH_APP_BASE_V9();
        pdd->scratch_limit = MAKE_SCRATCH_APP_LIMIT(pdd->scratch_base);
-
-       /*
-        * Place TBA/TMA on opposite side of VM hole to prevent
-        * stray faults from triggering SVM on these pages.
-        */
-       pdd->qpd.cwsr_base = pdd->dev->kfd->shared_resources.gpuvm_size;
 }
 
 int kfd_init_apertures(struct kfd_process *process)
@@ -413,6 +407,12 @@ int kfd_init_apertures(struct kfd_process *process)
                                        return -EINVAL;
                                }
                        }
+
+                        /* dGPUs: the reserved space for kernel
+                         * before SVM
+                         */
+                        pdd->qpd.cwsr_base = SVM_CWSR_BASE;
+                        pdd->qpd.ib_base = SVM_IB_BASE;
                }
 
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +0 -1
                dev_dbg(kfd_device, "node id %u\n", id);
--

2.42.0

I saw a segfault issue in Mesa yesterday. Not sure about the others, but I don't know how to make this change while compatibility with older UMDs.

So I agree, let's revert it.

Applied
#3 lore raw

Applied. Thanks!

Alex

Quoted 84 lines · Jay Cornwall
Quoted 72 lines · Alex Deucher

+ Jay, Felix

Quoted 69 lines · Kaibo Ma

That commit causes NULL pointer dereferences in dmesgs when running applications using ROCm, including clinfo, blender, and PyTorch, since v6.6.1. Revert it to fix blender again.

This reverts commit 96c211f1f9ef82183493f4ceed4e347b52849149.

---
 drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 26 ++++++++++----------
 1 file changed, 13 insertions(+), 13 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +13 -13
index 62b205dac..6604a3f99 100644
@@ -330,12 +330,6 @@ static void kfd_init_apertures_vi(struct kfd_process_device *pdd, uint8_t id)
        pdd->gpuvm_limit =
                pdd->dev->kfd->shared_resources.gpuvm_size - 1;
 
-       /* dGPUs: the reserved space for kernel
-        * before SVM
-        */
-       pdd->qpd.cwsr_base = SVM_CWSR_BASE;
-       pdd->qpd.ib_base = SVM_IB_BASE;
-
        pdd->scratch_base = MAKE_SCRATCH_APP_BASE_VI();
        pdd->scratch_limit = MAKE_SCRATCH_APP_LIMIT(pdd->scratch_base);
 }
@@ -345,18 +339,18 @@ static void kfd_init_apertures_v9(struct kfd_process_device *pdd, uint8_t id)
        pdd->lds_base = MAKE_LDS_APP_BASE_V9();
        pdd->lds_limit = MAKE_LDS_APP_LIMIT(pdd->lds_base);
 
-       pdd->gpuvm_base = PAGE_SIZE;
+        /* Raven needs SVM to support graphic handle, etc. Leave the small
+         * reserved space before SVM on Raven as well, even though we don't
+         * have to.
+         * Set gpuvm_base and gpuvm_limit to CANONICAL addresses so that they
+         * are used in Thunk to reserve SVM.
+         */
+        pdd->gpuvm_base = SVM_USER_BASE;
        pdd->gpuvm_limit =
                pdd->dev->kfd->shared_resources.gpuvm_size - 1;
 
        pdd->scratch_base = MAKE_SCRATCH_APP_BASE_V9();
        pdd->scratch_limit = MAKE_SCRATCH_APP_LIMIT(pdd->scratch_base);
-
-       /*
-        * Place TBA/TMA on opposite side of VM hole to prevent
-        * stray faults from triggering SVM on these pages.
-        */
-       pdd->qpd.cwsr_base = pdd->dev->kfd->shared_resources.gpuvm_size;
 }
 
 int kfd_init_apertures(struct kfd_process *process)
@@ -413,6 +407,12 @@ int kfd_init_apertures(struct kfd_process *process)
                                        return -EINVAL;
                                }
                        }
+
+                        /* dGPUs: the reserved space for kernel
+                         * before SVM
+                         */
+                        pdd->qpd.cwsr_base = SVM_CWSR_BASE;
+                        pdd->qpd.ib_base = SVM_IB_BASE;
                }
 
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +0 -1
                dev_dbg(kfd_device, "node id %u\n", id);
--

2.42.0

I saw a segfault issue in Mesa yesterday. Not sure about the others, but I don't know how to make this change while compatibility with older UMDs.

So I agree, let's revert it.

maintainer Acked-by Christian König Tested-by Emily Deng
#4 lore raw
In reply to Jay Cornwall
Quoted 72 lines · Jay Cornwall
Quoted 71 lines · Alex Deucher

+ Jay, Felix

Quoted 68 lines · Kaibo Ma

That commit causes NULL pointer dereferences in dmesgs when running applications using ROCm, including clinfo, blender, and PyTorch, since v6.6.1. Revert it to fix blender again.

This reverts commit 96c211f1f9ef82183493f4ceed4e347b52849149.

---
  drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 26 ++++++++++----------
  1 file changed, 13 insertions(+), 13 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +13 -13
index 62b205dac..6604a3f99 100644
@@ -330,12 +330,6 @@ static void kfd_init_apertures_vi(struct kfd_process_device *pdd, uint8_t id)
         pdd->gpuvm_limit =
                 pdd->dev->kfd->shared_resources.gpuvm_size - 1;
 
-       /* dGPUs: the reserved space for kernel
-        * before SVM
-        */
-       pdd->qpd.cwsr_base = SVM_CWSR_BASE;
-       pdd->qpd.ib_base = SVM_IB_BASE;
-
         pdd->scratch_base = MAKE_SCRATCH_APP_BASE_VI();
         pdd->scratch_limit = MAKE_SCRATCH_APP_LIMIT(pdd->scratch_base);
  }
@@ -345,18 +339,18 @@ static void kfd_init_apertures_v9(struct kfd_process_device *pdd, uint8_t id)
         pdd->lds_base = MAKE_LDS_APP_BASE_V9();
         pdd->lds_limit = MAKE_LDS_APP_LIMIT(pdd->lds_base);
 
-       pdd->gpuvm_base = PAGE_SIZE;
+        /* Raven needs SVM to support graphic handle, etc. Leave the small
+         * reserved space before SVM on Raven as well, even though we don't
+         * have to.
+         * Set gpuvm_base and gpuvm_limit to CANONICAL addresses so that they
+         * are used in Thunk to reserve SVM.
+         */
+        pdd->gpuvm_base = SVM_USER_BASE;
         pdd->gpuvm_limit =
                 pdd->dev->kfd->shared_resources.gpuvm_size - 1;
 
         pdd->scratch_base = MAKE_SCRATCH_APP_BASE_V9();
         pdd->scratch_limit = MAKE_SCRATCH_APP_LIMIT(pdd->scratch_base);
-
-       /*
-        * Place TBA/TMA on opposite side of VM hole to prevent
-        * stray faults from triggering SVM on these pages.
-        */
-       pdd->qpd.cwsr_base = pdd->dev->kfd->shared_resources.gpuvm_size;
  }
 
  int kfd_init_apertures(struct kfd_process *process)
@@ -413,6 +407,12 @@ int kfd_init_apertures(struct kfd_process *process)
                                         return -EINVAL;
                                 }
                         }
+
+                        /* dGPUs: the reserved space for kernel
+                         * before SVM
+                         */
+                        pdd->qpd.cwsr_base = SVM_CWSR_BASE;
+                        pdd->qpd.ib_base = SVM_IB_BASE;
                 }
 
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +0 -1
                 dev_dbg(kfd_device, "node id %u\n", id);
--

2.42.0

I saw a segfault issue in Mesa yesterday. Not sure about the others, but I don't know how to make this change while compatibility with older UMDs.

A segfault in Mesa seems to be a different issue from what's mentioned in the commit message. I'd let Christian or Marek comment on compatibility with graphics UMDs. I'm not sure why this patch would affect them at all.

Looking at the logs in the tickets, it looks like a fence reference counting error. I don't see how Jay's patch could have caused that. I made another change in that code recently that could make a difference for this issue:

commit 8f08c5b24ced1be7eb49692e4816c1916233c79b
Author: Felix Kuehling <[email protected]> Date:   Fri Oct 27 18:21:55 2023 -0400

    drm/amdkfd: Run restore_workers on freezable WQs

    Make restore workers freezable so we don't have to explicitly
flush them
    in suspend and GPU reset code paths, and we don't accidentally
try to
    restore BOs while the GPU is suspended. Not having to flush
restore_work
    also helps avoid lock/fence dependencies in the GPU reset case
where we're
    not allowed to wait for fences.

    A side effect of this is, that we can now have multiple
concurrent threads
    trying to signal the same eviction fence. Rework eviction fence
signaling
    and replacement to account for that.

    The GPU reset path can no longer rely on restore_process_worker
to resume
    queues because evict/restore workers can run independently of
it. Instead
    call a new restore_process_helper directly.

    This is an RFC and request for testing.

    v2:
    - Reworked eviction fence signaling
    - Introduced restore_process_helper

    v3:
    - Handle unsignaled eviction fences in restore_process_bos

FWIW, I built a plain 6.6 kernel, and was not able to reproduce the crash with some simple tests.

Regards,
  Felix

Quoted 4 lines

So I agree, let's revert it.

#5 lore raw
Quoted 4 lines · Felix Kuehling

A segfault in Mesa seems to be a different issue from what's mentioned in the commit message. I'd let Christian or Marek comment on compatibility with graphics UMDs. I'm not sure why this patch would affect them at all.

I was referencing this issue in OpenCL/OpenGL interop, which certainly looked related:

[   91.769002] amdgpu 0000:0a:00.0: amdgpu: bo 000000009bba4692 va 0x0800000000-0x08000001ff conflict with 0x0800000000-0x0800000002
[   91.769141] ocltst[2781]: segfault at b2 ip 00007f3fb90a7c39 sp 00007ffd3c011ba0 error 4 in radeonsi_dri.so[7f3fb888e000+1196000] likely on CPU 15 (core 7, socket 0)
Quoted 60 lines

Looking at the logs in the tickets, it looks like a fence reference counting error. I don't see how Jay's patch could have caused that. I made another change in that code recently that could make a difference for this issue:

commit 8f08c5b24ced1be7eb49692e4816c1916233c79b
Author: Felix Kuehling <[email protected]> Date:   Fri Oct 27 18:21:55 2023 -0400

    drm/amdkfd: Run restore_workers on freezable WQs

    Make restore workers freezable so we don't have to explicitly
flush them
    in suspend and GPU reset code paths, and we don't accidentally
try to
    restore BOs while the GPU is suspended. Not having to flush
restore_work
    also helps avoid lock/fence dependencies in the GPU reset case
where we're
    not allowed to wait for fences.

    A side effect of this is, that we can now have multiple
concurrent threads
    trying to signal the same eviction fence. Rework eviction fence
signaling
    and replacement to account for that.

    The GPU reset path can no longer rely on restore_process_worker
to resume
    queues because evict/restore workers can run independently of
it. Instead
    call a new restore_process_helper directly.

    This is an RFC and request for testing.

    v2:
    - Reworked eviction fence signaling
    - Introduced restore_process_helper

    v3:
    - Handle unsignaled eviction fences in restore_process_bos

FWIW, I built a plain 6.6 kernel, and was not able to reproduce the crash with some simple tests.

Regards,
  Felix

Quoted 4 lines

So I agree, let's revert it.

#6 lore raw

Hi,

I have received information that the original commit makes all 32-bit userspace VA allocations fail, so UMDs like Mesa can't even initialize and they either crash or fail to load. If TBA/TMA was relocated to the 32-bit address range, it would explain why UMDs can't allocate anything in that range.

Marek

Quoted 74 lines · Jay Cornwall
Quoted 4 lines · Felix Kuehling

A segfault in Mesa seems to be a different issue from what's mentioned in the commit message. I'd let Christian or Marek comment on compatibility with graphics UMDs. I'm not sure why this patch would affect them at all.

I was referencing this issue in OpenCL/OpenGL interop, which certainly looked related:

[   91.769002] amdgpu 0000:0a:00.0: amdgpu: bo 000000009bba4692 va 0x0800000000-0x08000001ff conflict with 0x0800000000-0x0800000002
[   91.769141] ocltst[2781]: segfault at b2 ip 00007f3fb90a7c39 sp 00007ffd3c011ba0 error 4 in radeonsi_dri.so[7f3fb888e000+1196000] likely on CPU 15 (core 7, socket 0)
Quoted 60 lines

Looking at the logs in the tickets, it looks like a fence reference counting error. I don't see how Jay's patch could have caused that. I made another change in that code recently that could make a difference for this issue:

commit 8f08c5b24ced1be7eb49692e4816c1916233c79b
Author: Felix Kuehling <[email protected]>
Date: Fri Oct 27 18:21:55 2023 -0400

drm/amdkfd: Run restore_workers on freezable WQs

Make restore workers freezable so we don't have to explicitly
flush them
in suspend and GPU reset code paths, and we don't accidentally
try to
restore BOs while the GPU is suspended. Not having to flush
restore_work
also helps avoid lock/fence dependencies in the GPU reset case
where we're
not allowed to wait for fences.

A side effect of this is, that we can now have multiple
concurrent threads
trying to signal the same eviction fence. Rework eviction fence
signaling
and replacement to account for that.

The GPU reset path can no longer rely on restore_process_worker
to resume
queues because evict/restore workers can run independently of
it. Instead
call a new restore_process_helper directly.

This is an RFC and request for testing.

v2:
- Reworked eviction fence signaling
- Introduced restore_process_helper

v3:
- Handle unsignaled eviction fences in restore_process_bos

FWIW, I built a plain 6.6 kernel, and was not able to reproduce the crash with some simple tests.

Regards,
Felix

Quoted 4 lines

So I agree, let's revert it.

maintainer
#7 lore raw

TBA/TMA were relocated to the upper half of the canonical address space. I don't think that qualifies as 32-bit by definition. But maybe you're using a different definition.

That said, if Mesa manages its own virtual address space in user mode, and KFD maps the TMA/TBA at an address that Mesa believes to be free, I can see how that would lead to problems.

That said, the fence refcount bug is another problem that may have been exposed by the way that a crashing Mesa application shuts down. Reverting Jay's patch certainly didn't fix that, but only hides the problem.

Regards,
  Felix

Quoted 80 lines · Marek Olšák

Hi,

I have received information that the original commit makes all 32-bit userspace VA allocations fail, so UMDs like Mesa can't even initialize and they either crash or fail to load. If TBA/TMA was relocated to the 32-bit address range, it would explain why UMDs can't allocate anything in that range.

Marek

Quoted 69 lines · Jay Cornwall
Quoted 4 lines · Felix Kuehling

A segfault in Mesa seems to be a different issue from what's mentioned in the commit message. I'd let Christian or Marek comment on compatibility with graphics UMDs. I'm not sure why this patch would affect them at all.

I was referencing this issue in OpenCL/OpenGL interop, which certainly looked related:

[   91.769002] amdgpu 0000:0a:00.0: amdgpu: bo 000000009bba4692 va 0x0800000000-0x08000001ff conflict with 0x0800000000-0x0800000002
[   91.769141] ocltst[2781]: segfault at b2 ip 00007f3fb90a7c39 sp 00007ffd3c011ba0 error 4 in radeonsi_dri.so[7f3fb888e000+1196000] likely on CPU 15 (core 7, socket 0)
Quoted 58 lines

Looking at the logs in the tickets, it looks like a fence reference counting error. I don't see how Jay's patch could have caused that. I made another change in that code recently that could make a difference for this issue:

commit 8f08c5b24ced1be7eb49692e4816c1916233c79b
Author: Felix Kuehling <[email protected]>
Date: Fri Oct 27 18:21:55 2023 -0400

drm/amdkfd: Run restore_workers on freezable WQs

Make restore workers freezable so we don't have to explicitly
flush them
in suspend and GPU reset code paths, and we don't accidentally
try to
restore BOs while the GPU is suspended. Not having to flush
restore_work
also helps avoid lock/fence dependencies in the GPU reset case
where we're
not allowed to wait for fences.

A side effect of this is, that we can now have multiple
concurrent threads
trying to signal the same eviction fence. Rework eviction fence
signaling
and replacement to account for that.

The GPU reset path can no longer rely on restore_process_worker
to resume
queues because evict/restore workers can run independently of
it. Instead
call a new restore_process_helper directly.

This is an RFC and request for testing.

v2:
- Reworked eviction fence signaling
- Introduced restore_process_helper

v3:
- Handle unsignaled eviction fences in restore_process_bos

FWIW, I built a plain 6.6 kernel, and was not able to reproduce the crash with some simple tests.

Regards,
Felix

Quoted 3 lines

So I agree, let's revert it.

#8 lore raw

The 32-bit address space means the high 32 bits are constant and predetermined and it's definitely somewhere in the upper range of the address space. If ROCm or KFD occupy that space, even accidentally, other UMDs that use libdrm for VA allocation won't be able to start. The VA range allocator is in libdrm.

Marek

Quoted 108 lines · On Fri, Jan 5, 2024, 15:20 Felix Kuehling <[email protected]> wrote:

TBA/TMA were relocated to the upper half of the canonical address space. I don't think that qualifies as 32-bit by definition. But maybe you're using a different definition.

That said, if Mesa manages its own virtual address space in user mode, and KFD maps the TMA/TBA at an address that Mesa believes to be free, I can see how that would lead to problems.

That said, the fence refcount bug is another problem that may have been exposed by the way that a crashing Mesa application shuts down. Reverting Jay's patch certainly didn't fix that, but only hides the problem.

Regards,
Felix

Quoted 11 lines · Marek Olšák

Hi,

I have received information that the original commit makes all 32-bit userspace VA allocations fail, so UMDs like Mesa can't even initialize and they either crash or fail to load. If TBA/TMA was relocated to the 32-bit address range, it would explain why UMDs can't allocate anything in that range.

Marek

On Wed, Jan 3, 2024 at 2:50 PM Jay Cornwall <[email protected]>

Quoted 7 lines · wrote:
Quoted 7 lines · Felix Kuehling
Quoted 5 lines

A segfault in Mesa seems to be a different issue from what's mentioned in the commit message. I'd let Christian or Marek comment on compatibility with graphics UMDs. I'm not sure why this patch would affect them at all.

I was referencing this issue in OpenCL/OpenGL interop, which certainly

looked related:

Quoted 2 lines
Quoted 2 lines
[   91.769002] amdgpu 0000:0a:00.0: amdgpu: bo 000000009bba4692 va

0x0800000000-0x08000001ff conflict with 0x0800000000-0x0800000002

Quoted 1 line
Quoted 1 line
[   91.769141] ocltst[2781]: segfault at b2 ip 00007f3fb90a7c39 sp

00007ffd3c011ba0 error 4 in radeonsi_dri.so[7f3fb888e000+1196000] likely on CPU 15 (core 7, socket 0)

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

Looking at the logs in the tickets, it looks like a fence reference counting error. I don't see how Jay's patch could have caused that. I made another change in that code recently that could make a difference for this issue:

     commit 8f08c5b24ced1be7eb49692e4816c1916233c79b
     Author: Felix Kuehling <[email protected]>
     Date:   Fri Oct 27 18:21:55 2023 -0400
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drm/amdkfd: Run restore_workers on freezable WQs

Make restore workers freezable so we don't have to explicitly
flush them
in suspend and GPU reset code paths, and we don't

accidentally

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

try to
restore BOs while the GPU is suspended. Not having to flush
restore_work
also helps avoid lock/fence dependencies in the GPU reset

case

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

where we're
not allowed to wait for fences.

A side effect of this is, that we can now have multiple
concurrent threads
trying to signal the same eviction fence. Rework eviction

fence

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

signaling
and replacement to account for that.

The GPU reset path can no longer rely on

restore_process_worker

Quoted 30 lines
Quoted 30 lines
Quoted 30 lines

to resume
queues because evict/restore workers can run independently of
it. Instead
call a new restore_process_helper directly.

This is an RFC and request for testing.

v2:
- Reworked eviction fence signaling
- Introduced restore_process_helper

v3:
- Handle unsignaled eviction fences in restore_process_bos

FWIW, I built a plain 6.6 kernel, and was not able to reproduce the crash with some simple tests.

Regards,
Felix

Quoted 3 lines

So I agree, let's revert it.

#9 lore raw

It looks like this would cause failures even with regular 64-bit allocations because the virtual address range allocator in libdrm asks the kernel what ranges of addresses are free, and the kernel doesn't exclude the KFD allocation from that.

Basically, no VM allocations can be done by the kernel outside the ranges reserved for the kernel.

Marek

Quoted 104 lines · Marek Olšák

The 32-bit address space means the high 32 bits are constant and predetermined and it's definitely somewhere in the upper range of the address space. If ROCm or KFD occupy that space, even accidentally, other UMDs that use libdrm for VA allocation won't be able to start. The VA range allocator is in libdrm.

Marek

Quoted 98 lines · On Fri, Jan 5, 2024, 15:20 Felix Kuehling <[email protected]> wrote:

TBA/TMA were relocated to the upper half of the canonical address space. I don't think that qualifies as 32-bit by definition. But maybe you're using a different definition.

That said, if Mesa manages its own virtual address space in user mode, and KFD maps the TMA/TBA at an address that Mesa believes to be free, I can see how that would lead to problems.

That said, the fence refcount bug is another problem that may have been exposed by the way that a crashing Mesa application shuts down. Reverting Jay's patch certainly didn't fix that, but only hides the problem.

Regards,
Felix

Quoted 80 lines · Marek Olšák

Hi,

I have received information that the original commit makes all 32-bit userspace VA allocations fail, so UMDs like Mesa can't even initialize and they either crash or fail to load. If TBA/TMA was relocated to the 32-bit address range, it would explain why UMDs can't allocate anything in that range.

Marek

Quoted 69 lines · Jay Cornwall
Quoted 4 lines · Felix Kuehling

A segfault in Mesa seems to be a different issue from what's mentioned in the commit message. I'd let Christian or Marek comment on compatibility with graphics UMDs. I'm not sure why this patch would affect them at all.

I was referencing this issue in OpenCL/OpenGL interop, which certainly looked related:

[   91.769002] amdgpu 0000:0a:00.0: amdgpu: bo 000000009bba4692 va 0x0800000000-0x08000001ff conflict with 0x0800000000-0x0800000002
[   91.769141] ocltst[2781]: segfault at b2 ip 00007f3fb90a7c39 sp 00007ffd3c011ba0 error 4 in radeonsi_dri.so[7f3fb888e000+1196000] likely on CPU 15 (core 7, socket 0)
Quoted 58 lines

Looking at the logs in the tickets, it looks like a fence reference counting error. I don't see how Jay's patch could have caused that. I made another change in that code recently that could make a difference for this issue:

commit 8f08c5b24ced1be7eb49692e4816c1916233c79b
Author: Felix Kuehling <[email protected]>
Date: Fri Oct 27 18:21:55 2023 -0400

drm/amdkfd: Run restore_workers on freezable WQs

Make restore workers freezable so we don't have to explicitly
flush them
in suspend and GPU reset code paths, and we don't accidentally
try to
restore BOs while the GPU is suspended. Not having to flush
restore_work
also helps avoid lock/fence dependencies in the GPU reset case
where we're
not allowed to wait for fences.

A side effect of this is, that we can now have multiple
concurrent threads
trying to signal the same eviction fence. Rework eviction fence
signaling
and replacement to account for that.

The GPU reset path can no longer rely on restore_process_worker
to resume
queues because evict/restore workers can run independently of
it. Instead
call a new restore_process_helper directly.

This is an RFC and request for testing.

v2:
- Reworked eviction fence signaling
- Introduced restore_process_helper

v3:
- Handle unsignaled eviction fences in restore_process_bos

FWIW, I built a plain 6.6 kernel, and was not able to reproduce the crash with some simple tests.

Regards,
Felix

Quoted 3 lines

So I agree, let's revert it.

maintainer
#10 lore raw

[+Christian]

I'm looking into virtual address reservations in amdgpu and what's reported by AMDGPU_INFO_DEV_INFO. So far I only found AMDGPU_VA_RESERVED_SIZE, which is reserved both at the start of the lower virtual address range and the end of the upper virtual address range.

The reservation size is currently 2MB. The upper reservation is used by the CSA, which is currently 128KB. That leaves plenty of room for the TMA/TBA above the CSA. I'll create a patch for that.

Regards,
 Felix

Quoted 113 lines · Marek Olšák

It looks like this would cause failures even with regular 64-bit allocations because the virtual address range allocator in libdrm asks the kernel what ranges of addresses are free, and the kernel doesn't exclude the KFD allocation from that.

Basically, no VM allocations can be done by the kernel outside the ranges reserved for the kernel.

Marek

Quoted 102 lines · Marek Olšák

The 32-bit address space means the high 32 bits are constant and predetermined and it's definitely somewhere in the upper range of the address space. If ROCm or KFD occupy that space, even accidentally, other UMDs that use libdrm for VA allocation won't be able to start. The VA range allocator is in libdrm.

Marek

Quoted 97 lines · On Fri, Jan 5, 2024, 15:20 Felix Kuehling <[email protected]> wrote:

TBA/TMA were relocated to the upper half of the canonical address space. I don't think that qualifies as 32-bit by definition. But maybe you're using a different definition.

That said, if Mesa manages its own virtual address space in user mode, and KFD maps the TMA/TBA at an address that Mesa believes to be free, I can see how that would lead to problems.

That said, the fence refcount bug is another problem that may have been exposed by the way that a crashing Mesa application shuts down. Reverting Jay's patch certainly didn't fix that, but only hides the problem.

Regards,
Felix

Quoted 80 lines · Marek Olšák

Hi,

I have received information that the original commit makes all 32-bit userspace VA allocations fail, so UMDs like Mesa can't even initialize and they either crash or fail to load. If TBA/TMA was relocated to the 32-bit address range, it would explain why UMDs can't allocate anything in that range.

Marek

Quoted 69 lines · Jay Cornwall
Quoted 4 lines · Felix Kuehling

A segfault in Mesa seems to be a different issue from what's mentioned in the commit message. I'd let Christian or Marek comment on compatibility with graphics UMDs. I'm not sure why this patch would affect them at all.

I was referencing this issue in OpenCL/OpenGL interop, which certainly looked related:

[   91.769002] amdgpu 0000:0a:00.0: amdgpu: bo 000000009bba4692 va 0x0800000000-0x08000001ff conflict with 0x0800000000-0x0800000002
[   91.769141] ocltst[2781]: segfault at b2 ip 00007f3fb90a7c39 sp 00007ffd3c011ba0 error 4 in radeonsi_dri.so[7f3fb888e000+1196000] likely on CPU 15 (core 7, socket 0)
Quoted 58 lines

Looking at the logs in the tickets, it looks like a fence reference counting error. I don't see how Jay's patch could have caused that. I made another change in that code recently that could make a difference for this issue:

commit 8f08c5b24ced1be7eb49692e4816c1916233c79b
Author: Felix Kuehling <[email protected]>
Date: Fri Oct 27 18:21:55 2023 -0400

drm/amdkfd: Run restore_workers on freezable WQs

Make restore workers freezable so we don't have to explicitly
flush them
in suspend and GPU reset code paths, and we don't accidentally
try to
restore BOs while the GPU is suspended. Not having to flush
restore_work
also helps avoid lock/fence dependencies in the GPU reset case
where we're
not allowed to wait for fences.

A side effect of this is, that we can now have multiple
concurrent threads
trying to signal the same eviction fence. Rework eviction fence
signaling
and replacement to account for that.

The GPU reset path can no longer rely on restore_process_worker
to resume
queues because evict/restore workers can run independently of
it. Instead
call a new restore_process_helper directly.

This is an RFC and request for testing.

v2:
- Reworked eviction fence signaling
- Introduced restore_process_helper

v3:
- Handle unsignaled eviction fences in restore_process_bos

FWIW, I built a plain 6.6 kernel, and was not able to reproduce the crash with some simple tests.

Regards,
Felix

Quoted 3 lines

So I agree, let's revert it.

maintainer
#11 lore raw

Arun and I have been working on a patch to split up AMDGPU_VA_RESERVED_SIZE into two separate defines.

I assumed that Arun landed this by now, but that doesn't seem to be the case.

Going to take another look, but yeah the CSA area isn't large enough to fill the upper reserved area so moving the TMA/TBA there should work.

Regards,
Christian.

Am 11.01.24 um 23:41 schrieb Felix Kuehling:

Quoted 124 lines

[+Christian]

I'm looking into virtual address reservations in amdgpu and what's reported by AMDGPU_INFO_DEV_INFO. So far I only found AMDGPU_VA_RESERVED_SIZE, which is reserved both at the start of the lower virtual address range and the end of the upper virtual address range.

The reservation size is currently 2MB. The upper reservation is used by the CSA, which is currently 128KB. That leaves plenty of room for the TMA/TBA above the CSA. I'll create a patch for that.

Regards,
 Felix

Quoted 113 lines · Marek Olšák

It looks like this would cause failures even with regular 64-bit allocations because the virtual address range allocator in libdrm asks the kernel what ranges of addresses are free, and the kernel doesn't exclude the KFD allocation from that.

Basically, no VM allocations can be done by the kernel outside the ranges reserved for the kernel.

Marek

Quoted 102 lines · Marek Olšák

The 32-bit address space means the high 32 bits are constant and predetermined and it's definitely somewhere in the upper range of the address space. If ROCm or KFD occupy that space, even accidentally, other UMDs that use libdrm for VA allocation won't be able to start. The VA range allocator is in libdrm.

Marek

Quoted 97 lines · On Fri, Jan 5, 2024, 15:20 Felix Kuehling <[email protected]> wrote:

TBA/TMA were relocated to the upper half of the canonical address space. I don't think that qualifies as 32-bit by definition. But maybe you're using a different definition.

That said, if Mesa manages its own virtual address space in user mode, and KFD maps the TMA/TBA at an address that Mesa believes to be free, I can see how that would lead to problems.

That said, the fence refcount bug is another problem that may have been exposed by the way that a crashing Mesa application shuts down. Reverting Jay's patch certainly didn't fix that, but only hides the problem.

Regards,
    Felix

Quoted 80 lines · Marek Olšák

Hi,

I have received information that the original commit makes all 32-bit userspace VA allocations fail, so UMDs like Mesa can't even initialize and they either crash or fail to load. If TBA/TMA was relocated to the 32-bit address range, it would explain why UMDs can't allocate anything in that range.

Marek

Quoted 69 lines · Jay Cornwall
Quoted 4 lines · Felix Kuehling

A segfault in Mesa seems to be a different issue from what's mentioned in the commit message. I'd let Christian or Marek comment on compatibility with graphics UMDs. I'm not sure why this patch would affect them at all.

I was referencing this issue in OpenCL/OpenGL interop, which certainly looked related:

[   91.769002] amdgpu 0000:0a:00.0: amdgpu: bo 000000009bba4692 va 0x0800000000-0x08000001ff conflict with 0x0800000000-0x0800000002
[   91.769141] ocltst[2781]: segfault at b2 ip 00007f3fb90a7c39 sp 00007ffd3c011ba0 error 4 in radeonsi_dri.so[7f3fb888e000+1196000] likely on CPU 15 (core 7, socket 0)

Quoted 58 lines

Looking at the logs in the tickets, it looks like a fence reference counting error. I don't see how Jay's patch could have caused that. I made another change in that code recently that could make a difference for this issue:

      commit 8f08c5b24ced1be7eb49692e4816c1916233c79b
      Author: Felix Kuehling <[email protected]> Date:   Fri Oct 27 18:21:55 2023 -0400

           drm/amdkfd: Run restore_workers on freezable WQs

           Make restore workers freezable so we don't have to explicitly flush them
           in suspend and GPU reset code paths, and we don't accidentally try to
           restore BOs while the GPU is suspended. Not having to flush restore_work
           also helps avoid lock/fence dependencies in the GPU reset case where we're
           not allowed to wait for fences.

           A side effect of this is, that we can now have multiple concurrent threads
           trying to signal the same eviction fence. Rework eviction fence signaling
           and replacement to account for that.

           The GPU reset path can no longer rely on restore_process_worker to resume
           queues because evict/restore workers can run independently of it. Instead
           call a new restore_process_helper directly.

           This is an RFC and request for testing.

           v2:
           - Reworked eviction fence signaling
           - Introduced restore_process_helper

           v3:
           - Handle unsignaled eviction fences in restore_process_bos

FWIW, I built a plain 6.6 kernel, and was not able to reproduce the crash with some simple tests.

Regards,
     Felix

Quoted 3 lines

So I agree, let's revert it.