dmr/amdgpu: Fix wrongly unref of BO
Subsystems (4)
PATCH 1/1 dmr/amdgpu: Fix wrongly unref of BO
According to comment of amdgpu_bo_reserve, amdgpu_bo_reserve can return with -ERESTARTSYS. When this function was interrupted by a signal, BO should not be unref. Otherwise the BO might be released while is kmapped and pinned, or BO MIGHT be deref multiple times, etc.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -1
@@ -355,8 +355,8 @@ static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev)amdgpu_bo_kunmap (adev->vram_scratch.robj);amdgpu_bo_unpin (adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +amdgpu_bo_unref (&adev->vram_scratch.robj); } -amdgpu_bo_unref (&adev->vram_scratch.robj); }/**
Signature · 6 lines
1.9.1 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 1492119298-8526-1-git-send-email-AlexBin.Xie-5C7GfCeVMHo@public.gmane.org
Quoted 5 lines · Alex Xie
According to comment of
amdgpu_bo_reserve,amdgpu_bo_reservecan return with-ERESTARTSYS. When this function was interrupted by a signal, BO should not be unref. Otherwise the BO might be released while is kmapped and pinned, or BO MIGHT be deref multiple times, etc.
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);
we have specified interruptible to false, so -ERESTARTSYS isn't possible here.
Thanks,
David Zhou
Quoted 12 lines
Change-Id If76071a768950a0d3ad9d5da7fcae04881807621Signed-off-by Alex Xie <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev) · line 362
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1@@ -355,8 +355,8 @@ static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev)amdgpu_bo_kunmap (adev->vram_scratch.robj);amdgpu_bo_unpin (adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +amdgpu_bo_unref (&adev->vram_scratch.robj); } -amdgpu_bo_unref (&adev->vram_scratch.robj); }/**
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi David,
Thanks for the comments. However, please have look at amdgpu_bo_reserve definition.
static inline int amdgpu_bo_reserve(struct amdgpu_bo *bo, bool no_intr)
When we call this function like the following:
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);
The false means interruptible.
On the other hand, when amdgpu_bo_reserve function return error, why do we unref BO without kunmap and unpin the BO? This is wrong implementation when amdgpu_bo_reserve return any error.
Thanks,
Alex Bin Xie
________________________________
From: Zhou, David(ChunMing)
Sent: Friday, April 14, 2017 12:00 AM
To: Xie, AlexBin; [email protected]
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Quoted 5 lines · Alex Xie
According to comment of
amdgpu_bo_reserve,amdgpu_bo_reservecan return with-ERESTARTSYS. When this function was interrupted by a signal, BO should not be unref. Otherwise the BO might be released while is kmapped and pinned, or BO MIGHT be deref multiple times, etc.
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);
we have specified interruptible to false, so -ERESTARTSYS isn't possible here.
Thanks,
David Zhou
Quoted 12 lines
Change-Id If76071a768950a0d3ad9d5da7fcae04881807621Signed-off-by Alex Xie <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev) · line 362
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1@@ -355,8 +355,8 @@ static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev)amdgpu_bo_kunmap (adev->vram_scratch.robj);amdgpu_bo_unpin (adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +amdgpu_bo_unref (&adev->vram_scratch.robj); } -amdgpu_bo_unref (&adev->vram_scratch.robj); }/**
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): MWHPR1201MB0045DE55E81D0FF5CE76671FF2060-3iK1xFAIwjq45V35fqe+hGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/[email protected]
Quoted 8 lines · AlexBin
Hi David,
Thanks for the comments. However, please have look at
amdgpu_bo_reservedefinition.static inline int
amdgpu_bo_reserve(struct amdgpu_bo *bo, bool no_intr)
Ah, this is a wired wrapper for ttm_bo_reserve.
Quoted 9 lines
When we call this function like the following:
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);The false means interruptible.
On the other hand, when
amdgpu_bo_reservefunction return error, why do we unref BO without kunmap and unpin the BO? This is wrong implementation whenamdgpu_bo_reservereturn any error.
Yeah, I see your mean, it's in driver un-loading, How about changing to no interruptible? Your patch will make a memleak if bo_reserve fails, but it seems not matter. I have no strong preference.
Regards,
David Zhou
Quoted 31 lines
Thanks,
Alex Bin Xie------------------------------------------------------------------------
*From:* Zhou, David(ChunMing)
*Sent:* Friday, April 14, 2017 12:00 AM
*To:* Xie, AlexBin; [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 5 lines · Alex Xie
According to comment of
amdgpu_bo_reserve,amdgpu_bo_reservecan return with-ERESTARTSYS. When this function was interrupted by a signal, BO should not be unref. Otherwise the BO might be released while is kmapped and pinned, or BO MIGHT be deref multiple times, etc.r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);we have specified interruptible to false, so
-ERESTARTSYSisn't possible here.Thanks,
David ZhouQuoted 7 lines
Change-Id If76071a768950a0d3ad9d5da7fcae04881807621Signed-off-by Alex Xie <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 17 lines
Review
Quoted 1 line
(quoted diff) +0
-0
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.cdrivers/gpu/drm/amd/amdgpu/amdgpu_device.c static void amdgpu_vram_scratch_fini(struct
Quoted 4 lines
index 53996e3..1dcc2d1 100644
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0@@ -355,8 +355,8 @@ static void amdgpu_vram_scratch_fini(struct
amdgpu_device*adev)Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1amdgpu_bo_kunmap (adev->vram_scratch.robj);amdgpu_bo_unpin (adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +amdgpu_bo_unref (&adev->vram_scratch.robj); } -amdgpu_bo_unref (&adev->vram_scratch.robj); }/**
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi David,
When amdgpu_bo_reserve return errors, we cannot release the BO. This is not a memory leak. General speaking, memory leak is unnoticed and unintentional.
The caller of function amdgpu_vram_scratch_fini ignores the return error value...
The "memory leak" is not caused by my patch. It is caused because reserving BO fails.
This patch only aim to make function amdgpu_vram_scratch_fini behave correctly.
To follow up, we can add a warning message when amdgpu_bo_reserve error happens in a different patch.
If function call amdgpu_bo_reserve is changed to uninterruptible, this changes driver behaviour. Without a substantial issue, I would be cautious for such a change.
Thanks,
Alex Bin Xie
________________________________
From: Zhou, David(ChunMing)
Sent: Monday, April 17, 2017 10:38 PM
To: Xie, AlexBin; [email protected]
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
On 2017年04月17日 22:54, Xie, AlexBin wrote:
Hi David,
Thanks for the comments. However, please have look at amdgpu_bo_reserve definition.
static inline int amdgpu_bo_reserve(struct amdgpu_bo *bo, bool no_intr)
Ah, this is a wired wrapper for ttm_bo_reserve.
When we call this function like the following:
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);
The false means interruptible.
On the other hand, when amdgpu_bo_reserve function return error, why do we unref BO without kunmap and unpin the BO? This is wrong implementation when amdgpu_bo_reserve return any error.
Yeah, I see your mean, it's in driver un-loading, How about changing to no interruptible? Your patch will make a memleak if bo_reserve fails, but it seems not matter. I have no strong preference.
Regards,
David Zhou
Thanks,
Alex Bin Xie
________________________________
From: Zhou, David(ChunMing)
Sent: Friday, April 14, 2017 12:00 AM
To: Xie, AlexBin; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Quoted 5 lines · Alex Xie
According to comment of
amdgpu_bo_reserve,amdgpu_bo_reservecan return with-ERESTARTSYS. When this function was interrupted by a signal, BO should not be unref. Otherwise the BO might be released while is kmapped and pinned, or BO MIGHT be deref multiple times, etc.
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);
we have specified interruptible to false, so -ERESTARTSYS isn't possible here.
Thanks,
David Zhou
Quoted 12 lines
Change-Id If76071a768950a0d3ad9d5da7fcae04881807621Signed-off-by Alex Xie <[email protected]><mailto:[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev) · line 362
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1@@ -355,8 +355,8 @@ static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev)amdgpu_bo_kunmap (adev->vram_scratch.robj);amdgpu_bo_unpin (adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +amdgpu_bo_unref (&adev->vram_scratch.robj); } -amdgpu_bo_unref (&adev->vram_scratch.robj); }/**
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): MWHPR1201MB00452889FF2C7FDC55630B4BF2190-3iK1xFAIwjq45V35fqe+hGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/[email protected]
Hi AlexBin,
No, David is right. This is a very common coding pattern in the kernel module.
Freeing up a BO while there still exists a kernel mapping is perfectly ok, the mapping will just be released a bit later on.
So this code is actually perfectly ok and just an optimization, but your patch breaks it and creates a memory leak.
Regards,
Christian.
Am 18.04.2017 um 17:17 schrieb Xie, AlexBin:
Quoted 109 lines
Hi David,
When
amdgpu_bo_reservereturn errors, we cannot release the BO. This is not a memory leak. General speaking, memory leak is unnoticed and unintentional.The caller of function
amdgpu_vram_scratch_finiignores the return error value...The "memory leak" is not caused by my patch. It is caused because reserving BO fails.
This patch only aim to make function
amdgpu_vram_scratch_finibehave correctly.To follow up, we can add a warning message when
amdgpu_bo_reserveerror happens in a different patch.If function call
amdgpu_bo_reserveis changed to uninterruptible, this changes driver behaviour. Without a substantial issue, I would be cautious for such a change.Thanks,
Alex Bin Xie
------------------------------------------------------------------------
*From:* Zhou, David(ChunMing)
*Sent:* Monday, April 17, 2017 10:38 PM
*To:* Xie, AlexBin; [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 8 lines · AlexBin
Hi David,
Thanks for the comments. However, please have look at
amdgpu_bo_reservedefinition.static inline int
amdgpu_bo_reserve(struct amdgpu_bo *bo, bool no_intr)Ah, this is a wired wrapper for
ttm_bo_reserve.Quoted 9 lines
When we call this function like the following:
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);The false means interruptible.
On the other hand, when
amdgpu_bo_reservefunction return error, why do we unref BO without kunmap and unpin the BO? This is wrong implementation whenamdgpu_bo_reservereturn any error.Yeah, I see your mean, it's in driver un-loading, How about changing to no interruptible? Your patch will make a memleak if
bo_reservefails, but it seems not matter. I have no strong preference.Regards,
David ZhouQuoted 31 lines
Thanks,
Alex Bin Xie------------------------------------------------------------------------
*From:* Zhou, David(ChunMing)
*Sent:* Friday, April 14, 2017 12:00 AM
*To:* Xie, AlexBin; [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 5 lines · Alex Xie
According to comment of
amdgpu_bo_reserve,amdgpu_bo_reservecan return with-ERESTARTSYS. When this function was interrupted by a signal, BO should not be unref. Otherwise the BO might be released while is kmapped and pinned, or BO MIGHT be deref multiple times, etc.r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);we have specified interruptible to false, so
-ERESTARTSYSisn't possible here.Thanks,
David ZhouQuoted 7 lines
Change-Id If76071a768950a0d3ad9d5da7fcae04881807621Signed-off-by Alex Xie <[email protected]>---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)Quoted 17 lines
Review
Quoted 1 line
(quoted diff) +0
-0
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.cdrivers/gpu/drm/amd/amdgpu/amdgpu_device.c static void amdgpu_vram_scratch_fini(struct
Quoted 4 lines
index 53996e3..1dcc2d1 100644
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0@@ -355,8 +355,8 @@ static void amdgpu_vram_scratch_fini(struct
amdgpu_device*adev)Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1amdgpu_bo_kunmap (adev->vram_scratch.robj);amdgpu_bo_unpin (adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +amdgpu_bo_unref (&adev->vram_scratch.robj); } -amdgpu_bo_unref (&adev->vram_scratch.robj); }/**
Quoted 4 lines
_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 7807726d-7318-b72f-1a14-2b37a73988bf-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org
Hi Christian,
Would you point out where/when will kunmap happen for this BO when release? It must be somewhere in some function calls.
I checked before I asked for review. But I did not see such obvious kunmap function call.
If so, there should be a comment in function amdgpu_vram_scratch_fini to avoid future confusion.
Thanks,
Alex Bin Xie
________________________________
From: Christian König <[email protected]>
Sent: Tuesday, April 18, 2017 1:46 PM
To: Xie, AlexBin; Zhou, David(ChunMing); [email protected]
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi AlexBin,
No, David is right. This is a very common coding pattern in the kernel module.
Freeing up a BO while there still exists a kernel mapping is perfectly ok, the mapping will just be released a bit later on.
So this code is actually perfectly ok and just an optimization, but your patch breaks it and creates a memory leak.
Regards,
Christian.
Am 18.04.2017 um 17:17 schrieb Xie, AlexBin:
Hi David,
When amdgpu_bo_reserve return errors, we cannot release the BO. This is not a memory leak. General speaking, memory leak is unnoticed and unintentional.
The caller of function amdgpu_vram_scratch_fini ignores the return error value...
The "memory leak" is not caused by my patch. It is caused because reserving BO fails.
This patch only aim to make function amdgpu_vram_scratch_fini behave correctly.
To follow up, we can add a warning message when amdgpu_bo_reserve error happens in a different patch.
If function call amdgpu_bo_reserve is changed to uninterruptible, this changes driver behaviour. Without a substantial issue, I would be cautious for such a change.
Thanks,
Alex Bin Xie
________________________________
From: Zhou, David(ChunMing)
Sent: Monday, April 17, 2017 10:38 PM
To: Xie, AlexBin; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
On 2017年04月17日 22:54, Xie, AlexBin wrote:
Hi David,
Thanks for the comments. However, please have look at amdgpu_bo_reserve definition.
static inline int amdgpu_bo_reserve(struct amdgpu_bo *bo, bool no_intr)
Ah, this is a wired wrapper for ttm_bo_reserve.
When we call this function like the following:
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);
The false means interruptible.
On the other hand, when amdgpu_bo_reserve function return error, why do we unref BO without kunmap and unpin the BO? This is wrong implementation when amdgpu_bo_reserve return any error.
Yeah, I see your mean, it's in driver un-loading, How about changing to no interruptible? Your patch will make a memleak if bo_reserve fails, but it seems not matter. I have no strong preference.
Regards,
David Zhou
Thanks,
Alex Bin Xie
________________________________
From: Zhou, David(ChunMing)
Sent: Friday, April 14, 2017 12:00 AM
To: Xie, AlexBin; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Quoted 5 lines · Alex Xie
According to comment of
amdgpu_bo_reserve,amdgpu_bo_reservecan return with-ERESTARTSYS. When this function was interrupted by a signal, BO should not be unref. Otherwise the BO might be released while is kmapped and pinned, or BO MIGHT be deref multiple times, etc.
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);
we have specified interruptible to false, so -ERESTARTSYS isn't possible here.
Thanks,
David Zhou
Quoted 12 lines
Change-Id If76071a768950a0d3ad9d5da7fcae04881807621Signed-off-by Alex Xie <[email protected]><mailto:[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev) · line 362
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1@@ -355,8 +355,8 @@ static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev)amdgpu_bo_kunmap (adev->vram_scratch.robj);amdgpu_bo_unpin (adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +amdgpu_bo_unref (&adev->vram_scratch.robj); } -amdgpu_bo_unref (&adev->vram_scratch.robj); }/**
_______________________________________________
amd-gfx mailing list [email protected]<mailto:[email protected]>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): MWHPR1201MB00451DFD2A06BBA3CD6F928EF2190-3iK1xFAIwjq45V35fqe+hGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/[email protected]
Hi Christian,
Have you found how/where/when? When you said "mapping will just be released a bit later on", you must know the answer.
It is difficult to prove something does not exist. Anyway, I will give it a try to prove such "later on" does not exist.
Function ttm_bo_kunmap is the only function to unmap. To prove this, search ttm_bo_map_iomap, only ttm_bo_kunmap use this enum to correctly kunmap.
Function ttm_bo_kunmap is not called by ttm itself. This is a hint that all TTM delay delete mechanism or unref mechanism will NOT kunmap BO later on.
Function ttm_bo_kunmap is called by AMDGPU function amdgpu_bo_kunmap and amdgpu_gem_prime_vunmap.
Search AMDGPU for amdgpu_bo_kunmap. All matches do not kunmap for scratch VRAM BO. amdgpu_bo_free_kernel is a suspect but the answer is still NO.
So all possibilities are searched. Did I miss anything?
Thanks,
Alex Bin Xie
________________________________
From: Xie, AlexBin
Sent: Tuesday, April 18, 2017 2:04:33 PM
To: Christian König; Zhou, David(ChunMing); [email protected]
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi Christian,
Would you point out where/when will kunmap happen for this BO when release? It must be somewhere in some function calls.
I checked before I asked for review. But I did not see such obvious kunmap function call.
If so, there should be a comment in function amdgpu_vram_scratch_fini to avoid future confusion.
Thanks,
Alex Bin Xie
________________________________
From: Christian König <[email protected]>
Sent: Tuesday, April 18, 2017 1:46 PM
To: Xie, AlexBin; Zhou, David(ChunMing); [email protected]
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi AlexBin,
No, David is right. This is a very common coding pattern in the kernel module.
Freeing up a BO while there still exists a kernel mapping is perfectly ok, the mapping will just be released a bit later on.
So this code is actually perfectly ok and just an optimization, but your patch breaks it and creates a memory leak.
Regards,
Christian.
Am 18.04.2017 um 17:17 schrieb Xie, AlexBin:
Hi David,
When amdgpu_bo_reserve return errors, we cannot release the BO. This is not a memory leak. General speaking, memory leak is unnoticed and unintentional.
The caller of function amdgpu_vram_scratch_fini ignores the return error value...
The "memory leak" is not caused by my patch. It is caused because reserving BO fails.
This patch only aim to make function amdgpu_vram_scratch_fini behave correctly.
To follow up, we can add a warning message when amdgpu_bo_reserve error happens in a different patch.
If function call amdgpu_bo_reserve is changed to uninterruptible, this changes driver behaviour. Without a substantial issue, I would be cautious for such a change.
Thanks,
Alex Bin Xie
________________________________
From: Zhou, David(ChunMing)
Sent: Monday, April 17, 2017 10:38 PM
To: Xie, AlexBin; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
On 2017年04月17日 22:54, Xie, AlexBin wrote:
Hi David,
Thanks for the comments. However, please have look at amdgpu_bo_reserve definition.
static inline int amdgpu_bo_reserve(struct amdgpu_bo *bo, bool no_intr)
Ah, this is a wired wrapper for ttm_bo_reserve.
When we call this function like the following:
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);
The false means interruptible.
On the other hand, when amdgpu_bo_reserve function return error, why do we unref BO without kunmap and unpin the BO? This is wrong implementation when amdgpu_bo_reserve return any error.
Yeah, I see your mean, it's in driver un-loading, How about changing to no interruptible? Your patch will make a memleak if bo_reserve fails, but it seems not matter. I have no strong preference.
Regards,
David Zhou
Thanks,
Alex Bin Xie
________________________________
From: Zhou, David(ChunMing)
Sent: Friday, April 14, 2017 12:00 AM
To: Xie, AlexBin; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Quoted 5 lines · Alex Xie
According to comment of
amdgpu_bo_reserve,amdgpu_bo_reservecan return with-ERESTARTSYS. When this function was interrupted by a signal, BO should not be unref. Otherwise the BO might be released while is kmapped and pinned, or BO MIGHT be deref multiple times, etc.
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);
we have specified interruptible to false, so -ERESTARTSYS isn't possible here.
Thanks,
David Zhou
Quoted 12 lines
Change-Id If76071a768950a0d3ad9d5da7fcae04881807621Signed-off-by Alex Xie <[email protected]><mailto:[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev) · line 362
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1@@ -355,8 +355,8 @@ static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev)amdgpu_bo_kunmap (adev->vram_scratch.robj);amdgpu_bo_unpin (adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +amdgpu_bo_unref (&adev->vram_scratch.robj); } -amdgpu_bo_unref (&adev->vram_scratch.robj); }/**
_______________________________________________
amd-gfx mailing list [email protected]<mailto:[email protected]>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): MWHPR1201MB004553768BCAC951D28A347AF2190-3iK1xFAIwjq45V35fqe+hGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/[email protected]
Hi AlexBin,
the answer is ttm_bo_kunmap isn't called at all and yes in the case of an iomap we leak the address space reserved.
But that is completely harmless on a 64bit system compared to leaking the memory backing the address space.
Using amdgpu_bo_free_kernel() instead of openly coding it here is probably a good idea.
Additional to that it's probably a good idea to set the no_intr flag when reserving kernel BOs. It is impossible to receive a signal during module load/unload, but it's probably better to document that in the code as well.
Regards,
Christian.
Am 18.04.2017 um 20:54 schrieb Xie, AlexBin:
Quoted 173 lines
Hi Christian,
Have you found how/where/when? When you said "mapping will just be released a bit later on", you must know the answer.
It is difficult to prove something does not exist. Anyway, I will give it a try to prove such "later on" does not exist.
Function
ttm_bo_kunmapis the only function to unmap. To prove this, searchttm_bo_map_iomap, onlyttm_bo_kunmapuse this enum to correctly kunmap.Function
ttm_bo_kunmapis not called by ttm itself. This is a hint that all TTM delay delete mechanism or unref mechanism will NOT kunmap BO later on.Function
ttm_bo_kunmapis called by AMDGPU functionamdgpu_bo_kunmapandamdgpu_gem_prime_vunmap.Search AMDGPU for
amdgpu_bo_kunmap. All matches do not kunmap for scratch VRAM BO.amdgpu_bo_free_kernelis a suspect but the answer is still NO.So all possibilities are searched. Did I miss anything?
Thanks,
Alex Bin Xie------------------------------------------------------------------------
*From:* Xie, AlexBin
*Sent:* Tuesday, April 18, 2017 2:04:33 PM
*To:* Christian König; Zhou, David(ChunMing); [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOHi Christian,
Would you point out where/when will kunmap happen for this BO when release? It must be somewhere in some function calls.
I checked before I asked for review. But I did not see such obvious kunmap function call.
If so, there should be a comment in function
amdgpu_vram_scratch_finito avoid future confusion.Thanks,
Alex Bin Xie
------------------------------------------------------------------------
*From:* Christian König <[email protected]> *Sent:* Tuesday, April 18, 2017 1:46 PM
*To:* Xie, AlexBin; Zhou, David(ChunMing); [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi AlexBin,No, David is right. This is a very common coding pattern in the kernel module.
Freeing up a BO while there still exists a kernel mapping is perfectly ok, the mapping will just be released a bit later on.
So this code is actually perfectly ok and just an optimization, but your patch breaks it and creates a memory leak.
Regards,
Christian.Am 18.04.2017 um 17:17 schrieb Xie, AlexBin:
Quoted 109 lines
Hi David,
When
amdgpu_bo_reservereturn errors, we cannot release the BO. This is not a memory leak. General speaking, memory leak is unnoticed and unintentional.The caller of function
amdgpu_vram_scratch_finiignores the return error value...The "memory leak" is not caused by my patch. It is caused because reserving BO fails.
This patch only aim to make function
amdgpu_vram_scratch_finibehave correctly.To follow up, we can add a warning message when
amdgpu_bo_reserveerror happens in a different patch.If function call
amdgpu_bo_reserveis changed to uninterruptible, this changes driver behaviour. Without a substantial issue, I would be cautious for such a change.Thanks,
Alex Bin Xie
------------------------------------------------------------------------
*From:* Zhou, David(ChunMing)
*Sent:* Monday, April 17, 2017 10:38 PM
*To:* Xie, AlexBin; [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 8 lines · AlexBin
Hi David,
Thanks for the comments. However, please have look at
amdgpu_bo_reservedefinition.static inline int
amdgpu_bo_reserve(struct amdgpu_bo *bo, bool no_intr)Ah, this is a wired wrapper for
ttm_bo_reserve.Quoted 9 lines
When we call this function like the following:
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);The false means interruptible.
On the other hand, when
amdgpu_bo_reservefunction return error, why do we unref BO without kunmap and unpin the BO? This is wrong implementation whenamdgpu_bo_reservereturn any error.Yeah, I see your mean, it's in driver un-loading, How about changing to no interruptible? Your patch will make a memleak if
bo_reservefails, but it seems not matter. I have no strong preference.Regards,
David ZhouQuoted 31 lines
Thanks,
Alex Bin Xie------------------------------------------------------------------------
*From:* Zhou, David(ChunMing)
*Sent:* Friday, April 14, 2017 12:00 AM
*To:* Xie, AlexBin; [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 5 lines · Alex Xie
According to comment of
amdgpu_bo_reserve,amdgpu_bo_reservecan return with-ERESTARTSYS. When this function was interrupted by a signal, BO should not be unref. Otherwise the BO might be released while is kmapped and pinned, or BO MIGHT be deref multiple times, etc.r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);we have specified interruptible to false, so
-ERESTARTSYSisn't possible here.Thanks,
David ZhouQuoted 7 lines
Change-Id If76071a768950a0d3ad9d5da7fcae04881807621Signed-off-by Alex Xie <[email protected]>---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)Quoted 17 lines
Review
Quoted 1 line
(quoted diff) +0
-0
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.cdrivers/gpu/drm/amd/amdgpu/amdgpu_device.c static void amdgpu_vram_scratch_fini(struct
Quoted 4 lines
index 53996e3..1dcc2d1 100644
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0@@ -355,8 +355,8 @@ static void amdgpu_vram_scratch_fini(struct
amdgpu_device*adev)Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1amdgpu_bo_kunmap (adev->vram_scratch.robj);amdgpu_bo_unpin (adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +amdgpu_bo_unref (&adev->vram_scratch.robj); } -amdgpu_bo_unref (&adev->vram_scratch.robj); }/** Quoted 4 lines
_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Quoted 4 lines
_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 4509f79a-ffd5-eb40-117c-84ea02cbf5cd-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org
Hi Christian,
Without correctly kunmap, page table is corrupted. Page entries point to wrong memory locations. You might call it completely harmless. But I think this is a severe bug. Leaking memory is better than a corrupted page table. Think security.
Would you provide any document and reference by saying" It is impossible to receive a signal during module load/unload"? For example, if the unload stuck in a lock, can CTRL+C stop the unload?
If "It is impossible to receive a signal during module load/unload", interruptible waiting is fine too, because function amdgpu_bo_reserve will return successfully.
What about there is some other return error? What about in future somebody improve amdgpu_bo_reserve to return other errors, then function amdgpu_vram_scratch_fini becomes buggy?
While I am thinking whether there is a better way for the current situation, would you give a real world example that my patch really not working? Then we can address it.
Thanks,
Alex Bin
________________________________
From: Christian König <[email protected]>
Sent: Wednesday, April 19, 2017 2:35 AM
To: Xie, AlexBin; Zhou, David(ChunMing); [email protected]
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi AlexBin,
the answer is ttm_bo_kunmap isn't called at all and yes in the case of an iomap we leak the address space reserved.
But that is completely harmless on a 64bit system compared to leaking the memory backing the address space.
Using amdgpu_bo_free_kernel() instead of openly coding it here is probably a good idea.
Additional to that it's probably a good idea to set the no_intr flag when reserving kernel BOs. It is impossible to receive a signal during module load/unload, but it's probably better to document that in the code as well.
Regards,
Christian.
Am 18.04.2017 um 20:54 schrieb Xie, AlexBin:
Hi Christian,
Have you found how/where/when? When you said "mapping will just be released a bit later on", you must know the answer.
It is difficult to prove something does not exist. Anyway, I will give it a try to prove such "later on" does not exist.
Function ttm_bo_kunmap is the only function to unmap. To prove this, search ttm_bo_map_iomap, only ttm_bo_kunmap use this enum to correctly kunmap.
Function ttm_bo_kunmap is not called by ttm itself. This is a hint that all TTM delay delete mechanism or unref mechanism will NOT kunmap BO later on.
Function ttm_bo_kunmap is called by AMDGPU function amdgpu_bo_kunmap and amdgpu_gem_prime_vunmap.
Search AMDGPU for amdgpu_bo_kunmap. All matches do not kunmap for scratch VRAM BO. amdgpu_bo_free_kernel is a suspect but the answer is still NO.
So all possibilities are searched. Did I miss anything?
Thanks,
Alex Bin Xie
________________________________
From: Xie, AlexBin
Sent: Tuesday, April 18, 2017 2:04:33 PM
To: Christian König; Zhou, David(ChunMing); [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi Christian,
Would you point out where/when will kunmap happen for this BO when release? It must be somewhere in some function calls.
I checked before I asked for review. But I did not see such obvious kunmap function call.
If so, there should be a comment in function amdgpu_vram_scratch_fini to avoid future confusion.
Thanks,
Alex Bin Xie
________________________________
From: Christian König <[email protected]><mailto:[email protected]>
Sent: Tuesday, April 18, 2017 1:46 PM
To: Xie, AlexBin; Zhou, David(ChunMing); [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi AlexBin,
No, David is right. This is a very common coding pattern in the kernel module.
Freeing up a BO while there still exists a kernel mapping is perfectly ok, the mapping will just be released a bit later on.
So this code is actually perfectly ok and just an optimization, but your patch breaks it and creates a memory leak.
Regards,
Christian.
Am 18.04.2017 um 17:17 schrieb Xie, AlexBin:
Hi David,
When amdgpu_bo_reserve return errors, we cannot release the BO. This is not a memory leak. General speaking, memory leak is unnoticed and unintentional.
The caller of function amdgpu_vram_scratch_fini ignores the return error value...
The "memory leak" is not caused by my patch. It is caused because reserving BO fails.
This patch only aim to make function amdgpu_vram_scratch_fini behave correctly.
To follow up, we can add a warning message when amdgpu_bo_reserve error happens in a different patch.
If function call amdgpu_bo_reserve is changed to uninterruptible, this changes driver behaviour. Without a substantial issue, I would be cautious for such a change.
Thanks,
Alex Bin Xie
________________________________
From: Zhou, David(ChunMing)
Sent: Monday, April 17, 2017 10:38 PM
To: Xie, AlexBin; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
On 2017年04月17日 22:54, Xie, AlexBin wrote:
Hi David,
Thanks for the comments. However, please have look at amdgpu_bo_reserve definition.
static inline int amdgpu_bo_reserve(struct amdgpu_bo *bo, bool no_intr)
Ah, this is a wired wrapper for ttm_bo_reserve.
When we call this function like the following:
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);
The false means interruptible.
On the other hand, when amdgpu_bo_reserve function return error, why do we unref BO without kunmap and unpin the BO? This is wrong implementation when amdgpu_bo_reserve return any error.
Yeah, I see your mean, it's in driver un-loading, How about changing to no interruptible? Your patch will make a memleak if bo_reserve fails, but it seems not matter. I have no strong preference.
Regards,
David Zhou
Thanks,
Alex Bin Xie
________________________________
From: Zhou, David(ChunMing)
Sent: Friday, April 14, 2017 12:00 AM
To: Xie, AlexBin; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Quoted 5 lines · Alex Xie
According to comment of
amdgpu_bo_reserve,amdgpu_bo_reservecan return with-ERESTARTSYS. When this function was interrupted by a signal, BO should not be unref. Otherwise the BO might be released while is kmapped and pinned, or BO MIGHT be deref multiple times, etc.
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);
we have specified interruptible to false, so -ERESTARTSYS isn't possible here.
Thanks,
David Zhou
Quoted 12 lines
Change-Id If76071a768950a0d3ad9d5da7fcae04881807621Signed-off-by Alex Xie <[email protected]><mailto:[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev) · line 362
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1@@ -355,8 +355,8 @@ static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev)amdgpu_bo_kunmap (adev->vram_scratch.robj);amdgpu_bo_unpin (adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +amdgpu_bo_unref (&adev->vram_scratch.robj); } -amdgpu_bo_unref (&adev->vram_scratch.robj); }/**
_______________________________________________
amd-gfx mailing list [email protected]<mailto:[email protected]>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list [email protected]<mailto:[email protected]>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): MWHPR1201MB0045EC51F6CE7C6C4283888CF2180-3iK1xFAIwjq45V35fqe+hGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/[email protected]
Quoted 1 line
Without correctly kunmap, page table is corrupted. Page entries point to wrong memory locations. You might call it completely harmless. But I think this is a severe bug. Leaking memory is better than a corrupted page table. Think security.
We are talking about the page tables for the vmalloc area in the kernel here, so no security problem. Leaking memory is much more problematic.
Quoted 2 lines
Would you provide any document and reference by saying" It is impossible to receive a signal during module load/unload"? For example, if the unload stuck in a lock, can CTRL+C stop the unload?
No, CTRL+C doesn't abort module load/unload. There have been patches to changes this a while ago, but IIRC it broke a whole bunch of driver relying on this.
Quoted 2 lines
What about there is some other return error? What about in future somebody improve
amdgpu_bo_reserveto return other errors, then functionamdgpu_vram_scratch_finibecomes buggy?
Yes, that is indeed an issue. For example -EDEADLK is possible as well. That's why I said we should use amdgpu_bo_free_kernel() instead.
Quoted 2 lines
While I am thinking whether there is a better way for the current situation, would you give a real world example that my patch really not working? Then we can address it.
I don't think there is because the driver can't receive a signal during load/unload, but the problem is rather that the patch doesn't improve the situation at all.
Regards,
Christian.
Am 19.04.2017 um 13:37 schrieb Xie, AlexBin:
Quoted 224 lines
Hi Christian,
Without correctly kunmap, page table is corrupted. Page entries point to wrong memory locations. You might call it completely harmless. But I think this is a severe bug. Leaking memory is better than a corrupted page table. Think security.
Would you provide any document and reference by saying" It is impossible to receive a signal during module load/unload"? For example, if the unload stuck in a lock, can CTRL+C stop the unload?
If "It is impossible to receive a signal during module load/unload", interruptible waiting is fine too, because function
amdgpu_bo_reservewill return successfully.What about there is some other return error? What about in future somebody improve
amdgpu_bo_reserveto return other errors, then functionamdgpu_vram_scratch_finibecomes buggy?While I am thinking whether there is a better way for the current situation, would you give a real world example that my patch really not working? Then we can address it.
Thanks,
Alex Bin
------------------------------------------------------------------------
*From:* Christian König <[email protected]> *Sent:* Wednesday, April 19, 2017 2:35 AM
*To:* Xie, AlexBin; Zhou, David(ChunMing); [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi AlexBin,the answer is
ttm_bo_kunmapisn't called at all and yes in the case of an iomap we leak the address space reserved.But that is completely harmless on a 64bit system compared to leaking the memory backing the address space.
Using
amdgpu_bo_free_kernel()instead of openly coding it here is probably a good idea.Additional to that it's probably a good idea to set the
no_intrflag when reserving kernel BOs. It is impossible to receive a signal during module load/unload, but it's probably better to document that in the code as well.Regards,
Christian.Am 18.04.2017 um 20:54 schrieb Xie, AlexBin:
Quoted 173 lines
Hi Christian,
Have you found how/where/when? When you said "mapping will just be released a bit later on", you must know the answer.
It is difficult to prove something does not exist. Anyway, I will give it a try to prove such "later on" does not exist.
Function
ttm_bo_kunmapis the only function to unmap. To prove this, searchttm_bo_map_iomap, onlyttm_bo_kunmapuse this enum to correctly kunmap.Function
ttm_bo_kunmapis not called by ttm itself. This is a hint that all TTM delay delete mechanism or unref mechanism will NOT kunmap BO later on.Function
ttm_bo_kunmapis called by AMDGPU functionamdgpu_bo_kunmapandamdgpu_gem_prime_vunmap.Search AMDGPU for
amdgpu_bo_kunmap. All matches do not kunmap for scratch VRAM BO.amdgpu_bo_free_kernelis a suspect but the answer is still NO.So all possibilities are searched. Did I miss anything?
Thanks,
Alex Bin Xie------------------------------------------------------------------------
*From:* Xie, AlexBin
*Sent:* Tuesday, April 18, 2017 2:04:33 PM
*To:* Christian König; Zhou, David(ChunMing); [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOHi Christian,
Would you point out where/when will kunmap happen for this BO when release? It must be somewhere in some function calls.
I checked before I asked for review. But I did not see such obvious kunmap function call.
If so, there should be a comment in function
amdgpu_vram_scratch_finito avoid future confusion.Thanks,
Alex Bin Xie
------------------------------------------------------------------------
*From:* Christian König <[email protected]> *Sent:* Tuesday, April 18, 2017 1:46 PM
*To:* Xie, AlexBin; Zhou, David(ChunMing); [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi AlexBin,No, David is right. This is a very common coding pattern in the kernel module.
Freeing up a BO while there still exists a kernel mapping is perfectly ok, the mapping will just be released a bit later on.
So this code is actually perfectly ok and just an optimization, but your patch breaks it and creates a memory leak.
Regards,
Christian.Am 18.04.2017 um 17:17 schrieb Xie, AlexBin:
Quoted 109 lines
Hi David,
When
amdgpu_bo_reservereturn errors, we cannot release the BO. This is not a memory leak. General speaking, memory leak is unnoticed and unintentional.The caller of function
amdgpu_vram_scratch_finiignores the return error value...The "memory leak" is not caused by my patch. It is caused because reserving BO fails.
This patch only aim to make function
amdgpu_vram_scratch_finibehave correctly.To follow up, we can add a warning message when
amdgpu_bo_reserveerror happens in a different patch.If function call
amdgpu_bo_reserveis changed to uninterruptible, this changes driver behaviour. Without a substantial issue, I would be cautious for such a change.Thanks,
Alex Bin Xie
------------------------------------------------------------------------
*From:* Zhou, David(ChunMing)
*Sent:* Monday, April 17, 2017 10:38 PM
*To:* Xie, AlexBin; [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 8 lines · AlexBin
Hi David,
Thanks for the comments. However, please have look at
amdgpu_bo_reservedefinition.static inline int
amdgpu_bo_reserve(struct amdgpu_bo *bo, bool no_intr)Ah, this is a wired wrapper for
ttm_bo_reserve.Quoted 9 lines
When we call this function like the following:
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);The false means interruptible.
On the other hand, when
amdgpu_bo_reservefunction return error, why do we unref BO without kunmap and unpin the BO? This is wrong implementation whenamdgpu_bo_reservereturn any error.Yeah, I see your mean, it's in driver un-loading, How about changing to no interruptible? Your patch will make a memleak if
bo_reservefails, but it seems not matter. I have no strong preference.Regards,
David ZhouQuoted 31 lines
Thanks,
Alex Bin Xie------------------------------------------------------------------------
*From:* Zhou, David(ChunMing)
*Sent:* Friday, April 14, 2017 12:00 AM
*To:* Xie, AlexBin; [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 5 lines · Alex Xie
According to comment of
amdgpu_bo_reserve,amdgpu_bo_reservecan return with-ERESTARTSYS. When this function was interrupted by a signal, BO should not be unref. Otherwise the BO might be released while is kmapped and pinned, or BO MIGHT be deref multiple times, etc.r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);we have specified interruptible to false, so
-ERESTARTSYSisn't possible here.Thanks,
David ZhouQuoted 7 lines
Change-Id If76071a768950a0d3ad9d5da7fcae04881807621Signed-off-by Alex Xie <[email protected]>---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)Quoted 17 lines
Review
Quoted 1 line
(quoted diff) +0
-0
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.cdrivers/gpu/drm/amd/amdgpu/amdgpu_device.c static void amdgpu_vram_scratch_fini(struct
Quoted 4 lines
index 53996e3..1dcc2d1 100644
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0@@ -355,8 +355,8 @@ static void amdgpu_vram_scratch_fini(struct
amdgpu_device*adev)Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1amdgpu_bo_kunmap (adev->vram_scratch.robj);amdgpu_bo_unpin (adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +amdgpu_bo_unref (&adev->vram_scratch.robj); } -amdgpu_bo_unref (&adev->vram_scratch.robj); }/** Quoted 4 lines
_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfxQuoted 4 lines
_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Quoted 4 lines
_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 2b43e2a9-0c79-f945-c835-0cfc486304c8-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org
Hi Christian,
Missing kunmap mapping in vmalloc will make kernel master page table incorrect. I would not call such issue as completely harmless. Please note that AMD graphic driver can run in 32 bit system. In 32 bit system, vmalloc address space is much smaller than size of most GPU VRAM.
amdgpu_bo_free_kernel has same issue as amdgpu_vram_scratch_fini. 1. It calls amdgpu_bo_reserve interruptible too. 2. It misses kunmap when amdgpu_bo_reserve returns error too. As result, kernel master page table can become incorrect, or as you call it "completely harmless vmalloc space leaking".
Because amdgpu_bo_free_kernel is used in more places, such as psp command submission, there will be bigger chance to have other usage where signal is not blocked. This will become a real bug.
I am thinking that we may fix the issue completely when TTM releases BO. But that is a bigger change.
It is good that you agree that there is no real world bad example caused by my patch. I will not discuss whether it is an improvement or not now to save time for both of us.
Thanks,
Alex Bin Xie
________________________________
From: Christian König <[email protected]>
Sent: Wednesday, April 19, 2017 7:50 AM
To: Xie, AlexBin; Zhou, David(ChunMing); [email protected]
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Without correctly kunmap, page table is corrupted. Page entries point to wrong memory locations. You might call it completely harmless. But I think this is a severe bug. Leaking memory is better than a corrupted page table. Think security. We are talking about the page tables for the vmalloc area in the kernel here, so no security problem. Leaking memory is much more problematic.
Would you provide any document and reference by saying" It is impossible to receive a signal during module load/unload"? For example, if the unload stuck in a lock, can CTRL+C stop the unload?
No, CTRL+C doesn't abort module load/unload. There have been patches to changes this a while ago, but IIRC it broke a whole bunch of driver relying on this.
What about there is some other return error? What about in future somebody improve amdgpu_bo_reserve to return other errors, then function amdgpu_vram_scratch_fini becomes buggy?
Yes, that is indeed an issue. For example -EDEADLK is possible as well. That's why I said we should use amdgpu_bo_free_kernel() instead.
While I am thinking whether there is a better way for the current situation, would you give a real world example that my patch really not working? Then we can address it.
I don't think there is because the driver can't receive a signal during load/unload, but the problem is rather that the patch doesn't improve the situation at all.
Regards,
Christian.
Am 19.04.2017 um 13:37 schrieb Xie, AlexBin:
Hi Christian,
Without correctly kunmap, page table is corrupted. Page entries point to wrong memory locations. You might call it completely harmless. But I think this is a severe bug. Leaking memory is better than a corrupted page table. Think security.
Would you provide any document and reference by saying" It is impossible to receive a signal during module load/unload"? For example, if the unload stuck in a lock, can CTRL+C stop the unload?
If "It is impossible to receive a signal during module load/unload", interruptible waiting is fine too, because function amdgpu_bo_reserve will return successfully.
What about there is some other return error? What about in future somebody improve amdgpu_bo_reserve to return other errors, then function amdgpu_vram_scratch_fini becomes buggy?
While I am thinking whether there is a better way for the current situation, would you give a real world example that my patch really not working? Then we can address it.
Thanks,
Alex Bin
________________________________
From: Christian König <[email protected]><mailto:[email protected]>
Sent: Wednesday, April 19, 2017 2:35 AM
To: Xie, AlexBin; Zhou, David(ChunMing); [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi AlexBin,
the answer is ttm_bo_kunmap isn't called at all and yes in the case of an iomap we leak the address space reserved.
But that is completely harmless on a 64bit system compared to leaking the memory backing the address space.
Using amdgpu_bo_free_kernel() instead of openly coding it here is probably a good idea.
Additional to that it's probably a good idea to set the no_intr flag when reserving kernel BOs. It is impossible to receive a signal during module load/unload, but it's probably better to document that in the code as well.
Regards,
Christian.
Am 18.04.2017 um 20:54 schrieb Xie, AlexBin:
Hi Christian,
Have you found how/where/when? When you said "mapping will just be released a bit later on", you must know the answer.
It is difficult to prove something does not exist. Anyway, I will give it a try to prove such "later on" does not exist.
Function ttm_bo_kunmap is the only function to unmap. To prove this, search ttm_bo_map_iomap, only ttm_bo_kunmap use this enum to correctly kunmap.
Function ttm_bo_kunmap is not called by ttm itself. This is a hint that all TTM delay delete mechanism or unref mechanism will NOT kunmap BO later on.
Function ttm_bo_kunmap is called by AMDGPU function amdgpu_bo_kunmap and amdgpu_gem_prime_vunmap.
Search AMDGPU for amdgpu_bo_kunmap. All matches do not kunmap for scratch VRAM BO. amdgpu_bo_free_kernel is a suspect but the answer is still NO.
So all possibilities are searched. Did I miss anything?
Thanks,
Alex Bin Xie
________________________________
From: Xie, AlexBin
Sent: Tuesday, April 18, 2017 2:04:33 PM
To: Christian König; Zhou, David(ChunMing); [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi Christian,
Would you point out where/when will kunmap happen for this BO when release? It must be somewhere in some function calls.
I checked before I asked for review. But I did not see such obvious kunmap function call.
If so, there should be a comment in function amdgpu_vram_scratch_fini to avoid future confusion.
Thanks,
Alex Bin Xie
________________________________
From: Christian König <[email protected]><mailto:[email protected]>
Sent: Tuesday, April 18, 2017 1:46 PM
To: Xie, AlexBin; Zhou, David(ChunMing); [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi AlexBin,
No, David is right. This is a very common coding pattern in the kernel module.
Freeing up a BO while there still exists a kernel mapping is perfectly ok, the mapping will just be released a bit later on.
So this code is actually perfectly ok and just an optimization, but your patch breaks it and creates a memory leak.
Regards,
Christian.
Am 18.04.2017 um 17:17 schrieb Xie, AlexBin:
Hi David,
When amdgpu_bo_reserve return errors, we cannot release the BO. This is not a memory leak. General speaking, memory leak is unnoticed and unintentional.
The caller of function amdgpu_vram_scratch_fini ignores the return error value...
The "memory leak" is not caused by my patch. It is caused because reserving BO fails.
This patch only aim to make function amdgpu_vram_scratch_fini behave correctly.
To follow up, we can add a warning message when amdgpu_bo_reserve error happens in a different patch.
If function call amdgpu_bo_reserve is changed to uninterruptible, this changes driver behaviour. Without a substantial issue, I would be cautious for such a change.
Thanks,
Alex Bin Xie
________________________________
From: Zhou, David(ChunMing)
Sent: Monday, April 17, 2017 10:38 PM
To: Xie, AlexBin; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
On 2017年04月17日 22:54, Xie, AlexBin wrote:
Hi David,
Thanks for the comments. However, please have look at amdgpu_bo_reserve definition.
static inline int amdgpu_bo_reserve(struct amdgpu_bo *bo, bool no_intr)
Ah, this is a wired wrapper for ttm_bo_reserve.
When we call this function like the following:
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);
The false means interruptible.
On the other hand, when amdgpu_bo_reserve function return error, why do we unref BO without kunmap and unpin the BO? This is wrong implementation when amdgpu_bo_reserve return any error.
Yeah, I see your mean, it's in driver un-loading, How about changing to no interruptible? Your patch will make a memleak if bo_reserve fails, but it seems not matter. I have no strong preference.
Regards,
David Zhou
Thanks,
Alex Bin Xie
________________________________
From: Zhou, David(ChunMing)
Sent: Friday, April 14, 2017 12:00 AM
To: Xie, AlexBin; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Quoted 5 lines · Alex Xie
According to comment of
amdgpu_bo_reserve,amdgpu_bo_reservecan return with-ERESTARTSYS. When this function was interrupted by a signal, BO should not be unref. Otherwise the BO might be released while is kmapped and pinned, or BO MIGHT be deref multiple times, etc.
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);
we have specified interruptible to false, so -ERESTARTSYS isn't possible here.
Thanks,
David Zhou
Quoted 12 lines
Change-Id If76071a768950a0d3ad9d5da7fcae04881807621Signed-off-by Alex Xie <[email protected]><mailto:[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev) · line 362
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1@@ -355,8 +355,8 @@ static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev)amdgpu_bo_kunmap (adev->vram_scratch.robj);amdgpu_bo_unpin (adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +amdgpu_bo_unref (&adev->vram_scratch.robj); } -amdgpu_bo_unref (&adev->vram_scratch.robj); }/**
_______________________________________________
amd-gfx mailing list [email protected]<mailto:[email protected]>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list [email protected]<mailto:[email protected]>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list [email protected]<mailto:[email protected]>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): MWHPR1201MB0045B04F77E1F1C6D37D24FBF2180-3iK1xFAIwjq45V35fqe+hGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/[email protected]
Hi AlexBin,
Quoted 1 line
Missing kunmap mapping in vmalloc will make kernel master page table incorrect.
That's what I tried to explain yesterday, but unfortunately didn't had time to do so. There is not corruption of the kernel master page table in this case!
The call of ttm_bo_kunmap is completely optional, take a look at amdgpu_ttm_io_mem_reserve() and amdgpu_ttm_io_mem_free().
The aperture is kept mapped into the page tables for the whole time the driver is loaded. So this is a complete no-op and only done for consistency.
Quoted 2 lines
It is good that you agree that there is no real world bad example caused by my patch. I will not discuss whether it is an improvement or not now to save time for both of us.
Great at least we can now agree to completely drop this patch.
Thanks,
Christian.
Am 19.04.2017 um 21:30 schrieb Xie, AlexBin:
Quoted 40 lines
Hi Christian,
Missing kunmap mapping in vmalloc will make kernel master page table incorrect. I would not call such issue as completely harmless. Please note that AMD graphic driver can run in 32 bit system. In 32 bit system, vmalloc address space is much smaller than size of most GPU VRAM.
amdgpu_bo_free_kernelhas same issue asamdgpu_vram_scratch_fini. 1. It callsamdgpu_bo_reserveinterruptible too. 2. It misses kunmap whenamdgpu_bo_reservereturns error too. As result, kernel master page table can become incorrect, or as you call it "completely harmless vmalloc space leaking".Because
amdgpu_bo_free_kernelis used in more places, such as psp command submission, there will be bigger chance to have other usage where signal is not blocked. This will become a real bug.I am thinking that we may fix the issue completely when TTM releases BO. But that is a bigger change.
It is good that you agree that there is no real world bad example caused by my patch. I will not discuss whether it is an improvement or not now to save time for both of us.
Thanks,
Alex Bin Xie
------------------------------------------------------------------------
*From:* Christian König <[email protected]> *Sent:* Wednesday, April 19, 2017 7:50 AM
*To:* Xie, AlexBin; Zhou, David(ChunMing); [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 1 line
Without correctly kunmap, page table is corrupted. Page entries point to wrong memory locations. You might call it completely harmless. But I think this is a severe bug. Leaking memory is better than a corrupted page table. Think security.
We are talking about the page tables for the vmalloc area in the kernel here, so no security problem. Leaking memory is much more problematic.
Quoted 2 lines
Would you provide any document and reference by saying" It is impossible to receive a signal during module load/unload"? For example, if the unload stuck in a lock, can CTRL+C stop the unload?
No, CTRL+C doesn't abort module load/unload. There have been patches to changes this a while ago, but IIRC it broke a whole bunch of driver relying on this.
Quoted 2 lines
What about there is some other return error? What about in future somebody improve
amdgpu_bo_reserveto return other errors, then functionamdgpu_vram_scratch_finibecomes buggy?Yes, that is indeed an issue. For example
-EDEADLKis possible as well. That's why I said we should useamdgpu_bo_free_kernel()instead.
Quoted 238 lines
Quoted 2 lines
While I am thinking whether there is a better way for the current situation, would you give a real world example that my patch really not working? Then we can address it.
I don't think there is because the driver can't receive a signal during load/unload, but the problem is rather that the patch doesn't improve the situation at all.
Regards,
Christian.Am 19.04.2017 um 13:37 schrieb Xie, AlexBin:
Quoted 228 lines
Hi Christian,
Without correctly kunmap, page table is corrupted. Page entries point to wrong memory locations. You might call it completely harmless. But I think this is a severe bug. Leaking memory is better than a corrupted page table. Think security.
Would you provide any document and reference by saying" It is impossible to receive a signal during module load/unload"? For example, if the unload stuck in a lock, can CTRL+C stop the unload?
If "It is impossible to receive a signal during module load/unload", interruptible waiting is fine too, because function
amdgpu_bo_reservewill return successfully.What about there is some other return error? What about in future somebody improve
amdgpu_bo_reserveto return other errors, then functionamdgpu_vram_scratch_finibecomes buggy?While I am thinking whether there is a better way for the current situation, would you give a real world example that my patch really not working? Then we can address it.
Thanks,
Alex Bin
------------------------------------------------------------------------
*From:* Christian König <[email protected]> *Sent:* Wednesday, April 19, 2017 2:35 AM
*To:* Xie, AlexBin; Zhou, David(ChunMing); [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi AlexBin,the answer is
ttm_bo_kunmapisn't called at all and yes in the case of an iomap we leak the address space reserved.But that is completely harmless on a 64bit system compared to leaking the memory backing the address space.
Using
amdgpu_bo_free_kernel()instead of openly coding it here is probably a good idea.Additional to that it's probably a good idea to set the
no_intrflag when reserving kernel BOs. It is impossible to receive a signal during module load/unload, but it's probably better to document that in the code as well.Regards,
Christian.Am 18.04.2017 um 20:54 schrieb Xie, AlexBin:
Quoted 173 lines
Hi Christian,
Have you found how/where/when? When you said "mapping will just be released a bit later on", you must know the answer.
It is difficult to prove something does not exist. Anyway, I will give it a try to prove such "later on" does not exist.
Function
ttm_bo_kunmapis the only function to unmap. To prove this, searchttm_bo_map_iomap, onlyttm_bo_kunmapuse this enum to correctly kunmap.Function
ttm_bo_kunmapis not called by ttm itself. This is a hint that all TTM delay delete mechanism or unref mechanism will NOT kunmap BO later on.Function
ttm_bo_kunmapis called by AMDGPU functionamdgpu_bo_kunmapandamdgpu_gem_prime_vunmap.Search AMDGPU for
amdgpu_bo_kunmap. All matches do not kunmap for scratch VRAM BO.amdgpu_bo_free_kernelis a suspect but the answer is still NO.So all possibilities are searched. Did I miss anything?
Thanks,
Alex Bin Xie------------------------------------------------------------------------
*From:* Xie, AlexBin
*Sent:* Tuesday, April 18, 2017 2:04:33 PM
*To:* Christian König; Zhou, David(ChunMing); [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOHi Christian,
Would you point out where/when will kunmap happen for this BO when release? It must be somewhere in some function calls.
I checked before I asked for review. But I did not see such obvious kunmap function call.
If so, there should be a comment in function
amdgpu_vram_scratch_finito avoid future confusion.Thanks,
Alex Bin Xie
------------------------------------------------------------------------
*From:* Christian König <[email protected]> *Sent:* Tuesday, April 18, 2017 1:46 PM
*To:* Xie, AlexBin; Zhou, David(ChunMing); [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi AlexBin,No, David is right. This is a very common coding pattern in the kernel module.
Freeing up a BO while there still exists a kernel mapping is perfectly ok, the mapping will just be released a bit later on.
So this code is actually perfectly ok and just an optimization, but your patch breaks it and creates a memory leak.
Regards,
Christian.Am 18.04.2017 um 17:17 schrieb Xie, AlexBin:
Quoted 109 lines
Hi David,
When
amdgpu_bo_reservereturn errors, we cannot release the BO. This is not a memory leak. General speaking, memory leak is unnoticed and unintentional.The caller of function
amdgpu_vram_scratch_finiignores the return error value...The "memory leak" is not caused by my patch. It is caused because reserving BO fails.
This patch only aim to make function
amdgpu_vram_scratch_finibehave correctly.To follow up, we can add a warning message when
amdgpu_bo_reserveerror happens in a different patch.If function call
amdgpu_bo_reserveis changed to uninterruptible, this changes driver behaviour. Without a substantial issue, I would be cautious for such a change.Thanks,
Alex Bin Xie
------------------------------------------------------------------------
*From:* Zhou, David(ChunMing)
*Sent:* Monday, April 17, 2017 10:38 PM
*To:* Xie, AlexBin; [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 8 lines · AlexBin
Hi David,
Thanks for the comments. However, please have look at
amdgpu_bo_reservedefinition.static inline int
amdgpu_bo_reserve(struct amdgpu_bo *bo, bool no_intr)Ah, this is a wired wrapper for
ttm_bo_reserve.Quoted 9 lines
When we call this function like the following:
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);The false means interruptible.
On the other hand, when
amdgpu_bo_reservefunction return error, why do we unref BO without kunmap and unpin the BO? This is wrong implementation whenamdgpu_bo_reservereturn any error.Yeah, I see your mean, it's in driver un-loading, How about changing to no interruptible? Your patch will make a memleak if
bo_reservefails, but it seems not matter. I have no strong preference.Regards,
David ZhouQuoted 31 lines
Thanks,
Alex Bin Xie------------------------------------------------------------------------
*From:* Zhou, David(ChunMing)
*Sent:* Friday, April 14, 2017 12:00 AM
*To:* Xie, AlexBin; [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 5 lines · Alex Xie
According to comment of
amdgpu_bo_reserve,amdgpu_bo_reservecan return with-ERESTARTSYS. When this function was interrupted by a signal, BO should not be unref. Otherwise the BO might be released while is kmapped and pinned, or BO MIGHT be deref multiple times, etc.r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);we have specified interruptible to false, so
-ERESTARTSYSisn't possible here.Thanks,
David ZhouQuoted 7 lines
Change-Id If76071a768950a0d3ad9d5da7fcae04881807621Signed-off-by Alex Xie <[email protected]>---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)Quoted 17 lines
Review
Quoted 1 line
(quoted diff) +0
-0
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.cdrivers/gpu/drm/amd/amdgpu/amdgpu_device.c static void amdgpu_vram_scratch_fini(struct
Quoted 4 lines
index 53996e3..1dcc2d1 100644
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0@@ -355,8 +355,8 @@ static void amdgpu_vram_scratch_fini(struct
amdgpu_device*adev)Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1amdgpu_bo_kunmap (adev->vram_scratch.robj);amdgpu_bo_unpin (adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +amdgpu_bo_unref (&adev->vram_scratch.robj); } -amdgpu_bo_unref (&adev->vram_scratch.robj); }/** Quoted 4 lines
_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfxQuoted 4 lines
_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 96809715-3c78-3784-fbeb-57a00359e3f3-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org
Hi Christian,
I read amdgpu_ttm_io_mem_reserve() and amdgpu_ttm_io_mem_free() and relevant codes from amdgpu_vram_scratch_init and amdgpu_vram_scratch_fini.
No. For the current source code, I think the premap and no-op is not working.
I add printk to prove. You can see bo_kmap_type is an invalid value. ioremap_wc is really called to map the IO range into vmalloc space.
...
Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759623] entering amdgpu_vram_scratch_init. Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759631] scratch ioremap_wc Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759631] bo_kmap_type = 129 Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759632] bus address = (null) Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759632] is_iomem = 1 Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759633] leaving amdgpu_vram_scratch_init.
...
I don't have log on rmmod AMDGPU yet. There are quite some settings to make that happen in my computer. I think they are symemtric. Both mapping and unmapping are not no-op.
Here is the printk patch for your reference.
From 25f95239c23225008e4b59f30b9b5363fb321f94 Mon Sep 17 00:00:00 2001
From: Alex Xie <[email protected]>
Date: Thu, 20 Apr 2017 17:48:49 -0400
Subject: [PATCH] A hack to trace premap and noop.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 22 ++++++++++++++++++++++ drivers/gpu/drm/ttm/ttm_bo.c | 1 + drivers/gpu/drm/ttm/ttm_bo_util.c | 29 ++++++++++++++++++++++++++--- include/drm/ttm/ttm_bo_api.h | 1 + 4 files changed, 50 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +22 -0
@@ -313,6 +313,7 @@ static void amdgpu_block_invalid_wreg(struct amdgpu_device *adev,static int amdgpu_vram_scratch_init (struct amdgpu_device *adev) {int r; +printk ("entering amdgpu_vram_scratch_init." );if (adev->vram_scratch.robj ==NULL ) { r = amdgpu_bo_create(adev, AMDGPU_GPU_PAGE_SIZE,
@@ -340,16 +341,36 @@ static int amdgpu_vram_scratch_init(struct amdgpu_device *adev)amdgpu_bo_unpin (adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +/* Would like a printk to see if map / unmap is noop*/ + adev->vram_scratch.robj->tbo.mem.bus.printk = true; + +if (adev->vram_scratch.robj->kmap.bo_kmap_type == ttm_bo_map_premapped) +printk ("amdgpu_vram_scratch premapped!" ); + +printk ("bo_kmap_type = %d" , adev->vram_scratch.robj->kmap.bo_kmap_type); +printk ("bus address = %p" , adev->vram_scratch.robj->tbo.mem.bus.addr); +printk ("is_iomem = %d" , adev->vram_scratch.robj->tbo.mem.bus.is_iomem); +printk ("leaving amdgpu_vram_scratch_init." ); +return r; }static void amdgpu_vram_scratch_fini (struct amdgpu_device *adev) {int r; +printk ("entering amdgpu_vram_scratch_fini." );if (adev->vram_scratch.robj ==NULL ) {return ; } + +if (adev->vram_scratch.robj->kmap.bo_kmap_type == ttm_bo_map_premapped) +printk ("amdgpu_vram_scratch premapped!" ); + +printk ("bo_kmap_type = %d" , adev->vram_scratch.robj->kmap.bo_kmap_type); +printk ("bus address = %p" , adev->vram_scratch.robj->tbo.mem.bus.addr); +printk ("is_iomem = %d" , adev->vram_scratch.robj->tbo.mem.bus.is_iomem); + r =amdgpu_bo_reserve (adev->vram_scratch.robj, false);if (likely (r ==0 )) {amdgpu_bo_kunmap (adev->vram_scratch.robj);
@@ -357,6 +378,7 @@ static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev)amdgpu_bo_unreserve (adev->vram_scratch.robj); }amdgpu_bo_unref (&adev->vram_scratch.robj); +printk ("leaving amdgpu_vram_scratch_fini." ); }/**
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
@@ -1178,6 +1178,7 @@ int ttm_bo_init_reserved(struct ttm_bo_device *bdev, bo->mem.page_alignment = page_alignment; bo->mem.bus.io_reserved_vm = false; bo->mem.bus.io_reserved_count =0 ; + bo->mem.bus.printk = false; bo->moving =NULL ; bo->mem.placement = (TTM_PL_FLAG_SYSTEM |TTM_PL_FLAG_CACHED ); bo->persistent_swap_storage = persistent_swap_storage;
drivers/gpu/drm/ttm/ttm_bo_util.c +26 -3
@@ -526,14 +526,24 @@ static int ttm_bo_ioremap(struct ttm_buffer_object *bo,if (bo->mem.bus.addr) { map->bo_kmap_type = ttm_bo_map_premapped; map->virtual = (void *)(((u8 *)bo->mem.bus.addr) + offset); +if (bo->mem.bus.printk) +printk ("scratch premapping" ); + }else { map->bo_kmap_type = ttm_bo_map_iomap; -if (mem->placement &TTM_PL_FLAG_WC ) +if (mem->placement &TTM_PL_FLAG_WC ) { map->virtual =ioremap_wc (bo->mem.bus.base + bo->mem.bus.offset + offset, size); -else +if (bo->mem.bus.printk) +printk ("scratch ioremap_wc" ); + + } +else { map->virtual =ioremap_nocache (bo->mem.bus.base + bo->mem.bus.offset + offset, size); +if (bo->mem.bus.printk) +printk ("scratch ioremap_nocache" ); + } }return (!map->virtual) ? -ENOMEM :0 ; }
@@ -618,21 +628,34 @@ void ttm_bo_kunmap(struct ttm_bo_kmap_obj *map)struct ttm_mem_type_manager *man = &bo->bdev->man[bo->mem.mem_type]; -if (!map->virtual) +if (!map->virtual) { +if (bo->mem.bus.printk) +printk ("scratch unmap return earlier" );return ; + }switch (map->bo_kmap_type) {case ttm_bo_map_iomap: +if (bo->mem.bus.printk) +printk ("scratch iounmap" );iounmap (map->virtual);break ;case ttm_bo_map_vmap: +if (bo->mem.bus.printk) +printk ("scratch vunmap" );vunmap (map->virtual);break ;case ttm_bo_map_kmap: +if (bo->mem.bus.printk) +printk ("scratch kunmap" );kunmap (map->page);break ;case ttm_bo_map_premapped: +if (bo->mem.bus.printk) +printk ("scratch unmap ttm_bo_map_premapped" );break ;default : +if (bo->mem.bus.printk) +printk ("scratch unmap bug" );BUG (); } (void )ttm_mem_io_lock (man, false);
include/drm/ttm/ttm_bo_api.h +1 -0
@@ -70,6 +70,7 @@ struct ttm_bus_placement {bool is_iomem;bool io_reserved_vm;uint64_t io_reserved_count; +bool printk; };
--
2.7.4
Thanks,
Alex Bin
________________________________
From: Christian König <[email protected]>
Sent: Thursday, April 20, 2017 4:43 AM
To: Xie, AlexBin; Zhou, David(ChunMing); [email protected]
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi AlexBin,
Missing kunmap mapping in vmalloc will make kernel master page table incorrect.
That's what I tried to explain yesterday, but unfortunately didn't had time to do so. There is not corruption of the kernel master page table in this case!
The call of ttm_bo_kunmap is completely optional, take a look at amdgpu_ttm_io_mem_reserve() and amdgpu_ttm_io_mem_free().
The aperture is kept mapped into the page tables for the whole time the driver is loaded. So this is a complete no-op and only done for consistency.
It is good that you agree that there is no real world bad example caused by my patch. I will not discuss whether it is an improvement or not now to save time for both of us.
Great at least we can now agree to completely drop this patch.
Thanks,
Christian.
Am 19.04.2017 um 21:30 schrieb Xie, AlexBin:
Hi Christian,
Missing kunmap mapping in vmalloc will make kernel master page table incorrect. I would not call such issue as completely harmless. Please note that AMD graphic driver can run in 32 bit system. In 32 bit system, vmalloc address space is much smaller than size of most GPU VRAM.
amdgpu_bo_free_kernel has same issue as amdgpu_vram_scratch_fini. 1. It calls amdgpu_bo_reserve interruptible too. 2. It misses kunmap when amdgpu_bo_reserve returns error too. As result, kernel master page table can become incorrect, or as you call it "completely harmless vmalloc space leaking".
Because amdgpu_bo_free_kernel is used in more places, such as psp command submission, there will be bigger chance to have other usage where signal is not blocked. This will become a real bug.
I am thinking that we may fix the issue completely when TTM releases BO. But that is a bigger change.
It is good that you agree that there is no real world bad example caused by my patch. I will not discuss whether it is an improvement or not now to save time for both of us.
Thanks,
Alex Bin Xie
________________________________
From: Christian König <[email protected]><mailto:[email protected]>
Sent: Wednesday, April 19, 2017 7:50 AM
To: Xie, AlexBin; Zhou, David(ChunMing); [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Without correctly kunmap, page table is corrupted. Page entries point to wrong memory locations. You might call it completely harmless. But I think this is a severe bug. Leaking memory is better than a corrupted page table. Think security. We are talking about the page tables for the vmalloc area in the kernel here, so no security problem. Leaking memory is much more problematic.
Would you provide any document and reference by saying" It is impossible to receive a signal during module load/unload"? For example, if the unload stuck in a lock, can CTRL+C stop the unload?
No, CTRL+C doesn't abort module load/unload. There have been patches to changes this a while ago, but IIRC it broke a whole bunch of driver relying on this.
What about there is some other return error? What about in future somebody improve amdgpu_bo_reserve to return other errors, then function amdgpu_vram_scratch_fini becomes buggy?
Yes, that is indeed an issue. For example -EDEADLK is possible as well. That's why I said we should use amdgpu_bo_free_kernel() instead.
While I am thinking whether there is a better way for the current situation, would you give a real world example that my patch really not working? Then we can address it.
I don't think there is because the driver can't receive a signal during load/unload, but the problem is rather that the patch doesn't improve the situation at all.
Regards,
Christian.
Am 19.04.2017 um 13:37 schrieb Xie, AlexBin:
Hi Christian,
Without correctly kunmap, page table is corrupted. Page entries point to wrong memory locations. You might call it completely harmless. But I think this is a severe bug. Leaking memory is better than a corrupted page table. Think security.
Would you provide any document and reference by saying" It is impossible to receive a signal during module load/unload"? For example, if the unload stuck in a lock, can CTRL+C stop the unload?
If "It is impossible to receive a signal during module load/unload", interruptible waiting is fine too, because function amdgpu_bo_reserve will return successfully.
What about there is some other return error? What about in future somebody improve amdgpu_bo_reserve to return other errors, then function amdgpu_vram_scratch_fini becomes buggy?
While I am thinking whether there is a better way for the current situation, would you give a real world example that my patch really not working? Then we can address it.
Thanks,
Alex Bin
________________________________
From: Christian König <[email protected]><mailto:[email protected]>
Sent: Wednesday, April 19, 2017 2:35 AM
To: Xie, AlexBin; Zhou, David(ChunMing); [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi AlexBin,
the answer is ttm_bo_kunmap isn't called at all and yes in the case of an iomap we leak the address space reserved.
But that is completely harmless on a 64bit system compared to leaking the memory backing the address space.
Using amdgpu_bo_free_kernel() instead of openly coding it here is probably a good idea.
Additional to that it's probably a good idea to set the no_intr flag when reserving kernel BOs. It is impossible to receive a signal during module load/unload, but it's probably better to document that in the code as well.
Regards,
Christian.
Am 18.04.2017 um 20:54 schrieb Xie, AlexBin:
Hi Christian,
Have you found how/where/when? When you said "mapping will just be released a bit later on", you must know the answer.
It is difficult to prove something does not exist. Anyway, I will give it a try to prove such "later on" does not exist.
Function ttm_bo_kunmap is the only function to unmap. To prove this, search ttm_bo_map_iomap, only ttm_bo_kunmap use this enum to correctly kunmap.
Function ttm_bo_kunmap is not called by ttm itself. This is a hint that all TTM delay delete mechanism or unref mechanism will NOT kunmap BO later on.
Function ttm_bo_kunmap is called by AMDGPU function amdgpu_bo_kunmap and amdgpu_gem_prime_vunmap.
Search AMDGPU for amdgpu_bo_kunmap. All matches do not kunmap for scratch VRAM BO. amdgpu_bo_free_kernel is a suspect but the answer is still NO.
So all possibilities are searched. Did I miss anything?
Thanks,
Alex Bin Xie
________________________________
From: Xie, AlexBin
Sent: Tuesday, April 18, 2017 2:04:33 PM
To: Christian König; Zhou, David(ChunMing); [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi Christian,
Would you point out where/when will kunmap happen for this BO when release? It must be somewhere in some function calls.
I checked before I asked for review. But I did not see such obvious kunmap function call.
If so, there should be a comment in function amdgpu_vram_scratch_fini to avoid future confusion.
Thanks,
Alex Bin Xie
________________________________
From: Christian König <[email protected]><mailto:[email protected]>
Sent: Tuesday, April 18, 2017 1:46 PM
To: Xie, AlexBin; Zhou, David(ChunMing); [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi AlexBin,
No, David is right. This is a very common coding pattern in the kernel module.
Freeing up a BO while there still exists a kernel mapping is perfectly ok, the mapping will just be released a bit later on.
So this code is actually perfectly ok and just an optimization, but your patch breaks it and creates a memory leak.
Regards,
Christian.
Am 18.04.2017 um 17:17 schrieb Xie, AlexBin:
Hi David,
When amdgpu_bo_reserve return errors, we cannot release the BO. This is not a memory leak. General speaking, memory leak is unnoticed and unintentional.
The caller of function amdgpu_vram_scratch_fini ignores the return error value...
The "memory leak" is not caused by my patch. It is caused because reserving BO fails.
This patch only aim to make function amdgpu_vram_scratch_fini behave correctly.
To follow up, we can add a warning message when amdgpu_bo_reserve error happens in a different patch.
If function call amdgpu_bo_reserve is changed to uninterruptible, this changes driver behaviour. Without a substantial issue, I would be cautious for such a change.
Thanks,
Alex Bin Xie
________________________________
From: Zhou, David(ChunMing)
Sent: Monday, April 17, 2017 10:38 PM
To: Xie, AlexBin; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
On 2017年04月17日 22:54, Xie, AlexBin wrote:
Hi David,
Thanks for the comments. However, please have look at amdgpu_bo_reserve definition.
static inline int amdgpu_bo_reserve(struct amdgpu_bo *bo, bool no_intr)
Ah, this is a wired wrapper for ttm_bo_reserve.
When we call this function like the following:
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);
The false means interruptible.
On the other hand, when amdgpu_bo_reserve function return error, why do we unref BO without kunmap and unpin the BO? This is wrong implementation when amdgpu_bo_reserve return any error.
Yeah, I see your mean, it's in driver un-loading, How about changing to no interruptible? Your patch will make a memleak if bo_reserve fails, but it seems not matter. I have no strong preference.
Regards,
David Zhou
Thanks,
Alex Bin Xie
________________________________
From: Zhou, David(ChunMing)
Sent: Friday, April 14, 2017 12:00 AM
To: Xie, AlexBin; [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Quoted 5 lines · Alex Xie
According to comment of
amdgpu_bo_reserve,amdgpu_bo_reservecan return with-ERESTARTSYS. When this function was interrupted by a signal, BO should not be unref. Otherwise the BO might be released while is kmapped and pinned, or BO MIGHT be deref multiple times, etc.
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);
we have specified interruptible to false, so -ERESTARTSYS isn't possible here.
Thanks,
David Zhou
Quoted 12 lines
Change-Id If76071a768950a0d3ad9d5da7fcae04881807621Signed-off-by Alex Xie <[email protected]><mailto:[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev) · line 362
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1@@ -355,8 +355,8 @@ static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev)amdgpu_bo_kunmap (adev->vram_scratch.robj);amdgpu_bo_unpin (adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +amdgpu_bo_unref (&adev->vram_scratch.robj); } -amdgpu_bo_unref (&adev->vram_scratch.robj); }/**
_______________________________________________
amd-gfx mailing list [email protected]<mailto:[email protected]>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list [email protected]<mailto:[email protected]>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list [email protected]<mailto:[email protected]>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): MWHPR1201MB004568A33CBF287710759301F21B0-3iK1xFAIwjq45V35fqe+hGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/[email protected]
Hi Alex,
Quoted 2 lines
No. For the current source code, I think the premap and no-op is not working.
Indeed, we don't set bo->mem.bus.addr in amdgpu_ttm_io_mem_reserve() any more. Felix will probably want to fix that for the KFD branch.
Anyway, as I said not unmapping the page tables is harmless compared to not releasing the memory backing it.
So please just do as I told you and change the interruptible reservation to a non-interruptible.
Regards,
Christian.
Am 20.04.2017 um 23:56 schrieb Xie, AlexBin:
Quoted 504 lines
Hi Christian,
I read
amdgpu_ttm_io_mem_reserve()andamdgpu_ttm_io_mem_free()and relevant codes fromamdgpu_vram_scratch_initandamdgpu_vram_scratch_fini.No. For the current source code, I think the premap and no-op is not working.
I add printk to prove. You can see
bo_kmap_typeis an invalid value.ioremap_wcis really called to map the IO range into vmalloc space....
Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759623] entering amdgpu_vram_scratch_init. Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759631] scratch ioremap_wc Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759631] bo_kmap_type = 129 Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759632] bus address = (null) Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759632] is_iomem = 1 Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759633] leaving amdgpu_vram_scratch_init....
I don't have log on rmmod AMDGPU yet. There are quite some settings to make that happen in my computer. I think they are symemtric. Both mapping and unmapping are not no-op.
Here is the printk patch for your reference.
From
25f95239c23225008e4b59f30b9b5363fb321f94Mon Sep 17 00:00:00 2001
From: Alex Xie <[email protected]>
Date: Thu, 20 Apr 2017 17:48:49 -0400
Subject: [PATCH] A hack to trace premap and noop.Change-Id I61fbbdbd82f62433e229b2e7e97be7a780ea8afaSigned-off-by Alex Xie <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 22 ++++++++++++++++++++++ drivers/gpu/drm/ttm/ttm_bo.c | 1 + drivers/gpu/drm/ttm/ttm_bo_util.c | 29 ++++++++++++++++++++++++++--- include/drm/ttm/ttm_bo_api.h | 1 + 4 files changed, 50 insertions(+), 3 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +22
-0@@ -313,6 +313,7 @@ static void amdgpu_block_invalid_wreg(structamdgpu_device *adev, static int amdgpu_vram_scratch_init(struct amdgpu_device *adev) { int r; +printk ("entering amdgpu_vram_scratch_init." ); if (adev->vram_scratch.robj == NULL) { r = amdgpu_bo_create(adev, AMDGPU_GPU_PAGE_SIZE,@@ -340,16 +341,36 @@ static int amdgpu_vram_scratch_init(struct amdgpu_device *adev) amdgpu_bo_unpin(adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +/* Would like a printk to see if map / unmap is noop*/ +adev->vram_scratch.robj->tbo.mem.bus.printk = true; + +if (adev->vram_scratch.robj->kmap.bo_kmap_type == ttm_bo_map_premapped) +printk ("amdgpu_vram_scratch premapped!" ); + +printk ("bo_kmap_type = %d" , adev->vram_scratch.robj->kmap.bo_kmap_type); +printk ("bus address = %p" , adev->vram_scratch.robj->tbo.mem.bus.addr); +printk ("is_iomem = %d" , adev->vram_scratch.robj->tbo.mem.bus.is_iomem); +printk ("leaving amdgpu_vram_scratch_init." ); +return r; }static void amdgpu_vram_scratch_fini (struct amdgpu_device *adev) {int r; +printk ("entering amdgpu_vram_scratch_fini." );if (adev->vram_scratch.robj ==NULL ) {return ; } + +if (adev->vram_scratch.robj->kmap.bo_kmap_type == ttm_bo_map_premapped) +printk ("amdgpu_vram_scratch premapped!" ); + +printk ("bo_kmap_type = %d" , adev->vram_scratch.robj->kmap.bo_kmap_type); +printk ("bus address = %p" , adev->vram_scratch.robj->tbo.mem.bus.addr); +printk ("is_iomem = %d" , adev->vram_scratch.robj->tbo.mem.bus.is_iomem); + r =amdgpu_bo_reserve (adev->vram_scratch.robj, false);if (likely (r ==0 )) {amdgpu_bo_kunmap (adev->vram_scratch.robj);@@ -357,6 +378,7 @@ static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev) amdgpu_bo_unreserve(adev->vram_scratch.robj); }amdgpu_bo_unref (&adev->vram_scratch.robj); +printk ("leaving amdgpu_vram_scratch_fini." ); }/** drivers/gpu/drm/ttm/ttm_bo.c +1
-0@@ -1178,6 +1178,7 @@ int ttm_bo_init_reserved(struct ttm_bo_device *bdev, bo->mem.page_alignment = page_alignment; bo->mem.bus.io_reserved_vm = false; bo->mem.bus.io_reserved_count =0 ; +bo->mem.bus.printk = false; bo->moving =NULL ; bo->mem.placement = (TTM_PL_FLAG_SYSTEM |TTM_PL_FLAG_CACHED ); bo->persistent_swap_storage = persistent_swap_storage;drivers/gpu/drm/ttm/ttm_bo_util.c +12
-2@@ -526,14 +526,24 @@ static int ttm_bo_ioremap(struct ttm_buffer_object *bo, if (bo->mem.bus.addr) { map->bo_kmap_type = ttm_bo_map_premapped; map->virtual = (void *)(((u8 *)bo->mem.bus.addr) + offset); +if (bo->mem.bus.printk) +printk ("scratch premapping" ); + } else { map->bo_kmap_type = ttm_bo_map_iomap; -if (mem->placement &TTM_PL_FLAG_WC ) +if (mem->placement &TTM_PL_FLAG_WC ) { map->virtual =ioremap_wc (bo->mem.bus.base + bo->mem.bus.offset + offset, size); -else +if (bo->mem.bus.printk) +printk ("scratch ioremap_wc" ); + +} +else { map->virtual =ioremap_nocache (bo->mem.bus.base + bo->mem.bus.offset + offset, size); +if (bo->mem.bus.printk) +printk ("scratch ioremap_nocache" ); +} }return (!map->virtual) ? -ENOMEM :0 ;}
drivers/gpu/drm/ttm/ttm_bo_util.c +0
-0@@ -618,21 +628,34 @@ void ttm_bo_kunmap(struct ttm_bo_kmap_obj *map)struct
ttm_mem_type_manager*man =
&bo->bdev->man[bo->mem.mem_type];drivers/gpu/drm/ttm/ttm_bo_util.c +3
-1-if (!map->virtual) +if (!map->virtual) { +if (bo->mem.bus.printk) +printk ("scratch unmap return earlier" );return;
+}
switch (map->bo_kmap_type) {
casettm_bo_map_iomap:drivers/gpu/drm/ttm/ttm_bo_util.c +2
-0+if (bo->mem.bus.printk) +printk ("scratch iounmap" );
iounmap(map->virtual);
break;
casettm_bo_map_vmap:drivers/gpu/drm/ttm/ttm_bo_util.c +2
-0+if (bo->mem.bus.printk) +printk ("scratch vunmap" );
vunmap(map->virtual);
break;
casettm_bo_map_kmap:drivers/gpu/drm/ttm/ttm_bo_util.c +2
-0+if (bo->mem.bus.printk) +printk ("scratch kunmap" );
kunmap(map->page);
break;
casettm_bo_map_premapped:drivers/gpu/drm/ttm/ttm_bo_util.c +2
-0+if (bo->mem.bus.printk) +printk ("scratch unmap ttm_bo_map_premapped" );break;
default:drivers/gpu/drm/ttm/ttm_bo_util.c +2
-0+if (bo->mem.bus.printk) +printk ("scratch unmap bug" );
BUG();
}
(void)ttm_mem_io_lock(man, false);include/drm/ttm/ttm_bo_api.h +1
-0@@ -70,6 +70,7 @@ struct ttm_bus_placement { boolis_iomem; boolio_reserved_vm;uint64_t io_reserved_count; +bool printk; };Signature · 15 lines
2.7.4 Thanks, Alex Bin ------------------------------------------------------------------------ *From:* Christian König <[email protected]> *Sent:* Thursday, April 20, 2017 4:43 AM *To:* Xie, AlexBin; Zhou, David(ChunMing); [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO Hi AlexBin,Quoted 1 line
Missing kunmap mapping in vmalloc will make kernel master page table incorrect.
That's what I tried to explain yesterday, but unfortunately didn't had time to do so. There is not corruption of the kernel master page table in this case!
The call of
ttm_bo_kunmapis completely optional, take a look atamdgpu_ttm_io_mem_reserve()andamdgpu_ttm_io_mem_free().The aperture is kept mapped into the page tables for the whole time the driver is loaded. So this is a complete no-op and only done for consistency.
Quoted 2 lines
It is good that you agree that there is no real world bad example caused by my patch. I will not discuss whether it is an improvement or not now to save time for both of us.
Great at least we can now agree to completely drop this patch.
Thanks,
Christian.Am 19.04.2017 um 21:30 schrieb Xie, AlexBin:
Quoted 40 lines
Hi Christian,
Missing kunmap mapping in vmalloc will make kernel master page table incorrect. I would not call such issue as completely harmless. Please note that AMD graphic driver can run in 32 bit system. In 32 bit system, vmalloc address space is much smaller than size of most GPU VRAM.
amdgpu_bo_free_kernelhas same issue asamdgpu_vram_scratch_fini. 1. It callsamdgpu_bo_reserveinterruptible too. 2. It misses kunmap whenamdgpu_bo_reservereturns error too. As result, kernel master page table can become incorrect, or as you call it "completely harmless vmalloc space leaking".Because
amdgpu_bo_free_kernelis used in more places, such as psp command submission, there will be bigger chance to have other usage where signal is not blocked. This will become a real bug.I am thinking that we may fix the issue completely when TTM releases BO. But that is a bigger change.
It is good that you agree that there is no real world bad example caused by my patch. I will not discuss whether it is an improvement or not now to save time for both of us.
Thanks,
Alex Bin Xie
------------------------------------------------------------------------
*From:* Christian König <[email protected]> *Sent:* Wednesday, April 19, 2017 7:50 AM
*To:* Xie, AlexBin; Zhou, David(ChunMing); [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 1 line
Without correctly kunmap, page table is corrupted. Page entries point to wrong memory locations. You might call it completely harmless. But I think this is a severe bug. Leaking memory is better than a corrupted page table. Think security.
We are talking about the page tables for the vmalloc area in the kernel here, so no security problem. Leaking memory is much more problematic.
Quoted 2 lines
Would you provide any document and reference by saying" It is impossible to receive a signal during module load/unload"? For example, if the unload stuck in a lock, can CTRL+C stop the unload?
No, CTRL+C doesn't abort module load/unload. There have been patches to changes this a while ago, but IIRC it broke a whole bunch of driver relying on this.
Quoted 2 lines
What about there is some other return error? What about in future somebody improve
amdgpu_bo_reserveto return other errors, then functionamdgpu_vram_scratch_finibecomes buggy?Yes, that is indeed an issue. For example
-EDEADLKis possible as well. That's why I said we should useamdgpu_bo_free_kernel()instead.Quoted 238 lines
Quoted 2 lines
While I am thinking whether there is a better way for the current situation, would you give a real world example that my patch really not working? Then we can address it.
I don't think there is because the driver can't receive a signal during load/unload, but the problem is rather that the patch doesn't improve the situation at all.
Regards,
Christian.Am 19.04.2017 um 13:37 schrieb Xie, AlexBin:
Quoted 228 lines
Hi Christian,
Without correctly kunmap, page table is corrupted. Page entries point to wrong memory locations. You might call it completely harmless. But I think this is a severe bug. Leaking memory is better than a corrupted page table. Think security.
Would you provide any document and reference by saying" It is impossible to receive a signal during module load/unload"? For example, if the unload stuck in a lock, can CTRL+C stop the unload?
If "It is impossible to receive a signal during module load/unload", interruptible waiting is fine too, because function
amdgpu_bo_reservewill return successfully.What about there is some other return error? What about in future somebody improve
amdgpu_bo_reserveto return other errors, then functionamdgpu_vram_scratch_finibecomes buggy?While I am thinking whether there is a better way for the current situation, would you give a real world example that my patch really not working? Then we can address it.
Thanks,
Alex Bin
------------------------------------------------------------------------
*From:* Christian König <[email protected]> *Sent:* Wednesday, April 19, 2017 2:35 AM
*To:* Xie, AlexBin; Zhou, David(ChunMing); [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi AlexBin,the answer is
ttm_bo_kunmapisn't called at all and yes in the case of an iomap we leak the address space reserved.But that is completely harmless on a 64bit system compared to leaking the memory backing the address space.
Using
amdgpu_bo_free_kernel()instead of openly coding it here is probably a good idea.Additional to that it's probably a good idea to set the
no_intrflag when reserving kernel BOs. It is impossible to receive a signal during module load/unload, but it's probably better to document that in the code as well.Regards,
Christian.Am 18.04.2017 um 20:54 schrieb Xie, AlexBin:
Quoted 173 lines
Hi Christian,
Have you found how/where/when? When you said "mapping will just be released a bit later on", you must know the answer.
It is difficult to prove something does not exist. Anyway, I will give it a try to prove such "later on" does not exist.
Function
ttm_bo_kunmapis the only function to unmap. To prove this, searchttm_bo_map_iomap, onlyttm_bo_kunmapuse this enum to correctly kunmap.Function
ttm_bo_kunmapis not called by ttm itself. This is a hint that all TTM delay delete mechanism or unref mechanism will NOT kunmap BO later on.Function
ttm_bo_kunmapis called by AMDGPU functionamdgpu_bo_kunmapandamdgpu_gem_prime_vunmap.Search AMDGPU for
amdgpu_bo_kunmap. All matches do not kunmap for scratch VRAM BO.amdgpu_bo_free_kernelis a suspect but the answer is still NO.So all possibilities are searched. Did I miss anything?
Thanks,
Alex Bin Xie------------------------------------------------------------------------
*From:* Xie, AlexBin
*Sent:* Tuesday, April 18, 2017 2:04:33 PM
*To:* Christian König; Zhou, David(ChunMing); [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOHi Christian,
Would you point out where/when will kunmap happen for this BO when release? It must be somewhere in some function calls.
I checked before I asked for review. But I did not see such obvious kunmap function call.
If so, there should be a comment in function
amdgpu_vram_scratch_finito avoid future confusion.Thanks,
Alex Bin Xie
------------------------------------------------------------------------
*From:* Christian König <[email protected]> *Sent:* Tuesday, April 18, 2017 1:46 PM
*To:* Xie, AlexBin; Zhou, David(ChunMing); [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO
Hi AlexBin,No, David is right. This is a very common coding pattern in the kernel module.
Freeing up a BO while there still exists a kernel mapping is perfectly ok, the mapping will just be released a bit later on.
So this code is actually perfectly ok and just an optimization, but your patch breaks it and creates a memory leak.
Regards,
Christian.Am 18.04.2017 um 17:17 schrieb Xie, AlexBin:
Quoted 109 lines
Hi David,
When
amdgpu_bo_reservereturn errors, we cannot release the BO. This is not a memory leak. General speaking, memory leak is unnoticed and unintentional.The caller of function
amdgpu_vram_scratch_finiignores the return error value...The "memory leak" is not caused by my patch. It is caused because reserving BO fails.
This patch only aim to make function
amdgpu_vram_scratch_finibehave correctly.To follow up, we can add a warning message when
amdgpu_bo_reserveerror happens in a different patch.If function call
amdgpu_bo_reserveis changed to uninterruptible, this changes driver behaviour. Without a substantial issue, I would be cautious for such a change.Thanks,
Alex Bin Xie
------------------------------------------------------------------------
*From:* Zhou, David(ChunMing)
*Sent:* Monday, April 17, 2017 10:38 PM
*To:* Xie, AlexBin; [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 8 lines · AlexBin
Hi David,
Thanks for the comments. However, please have look at
amdgpu_bo_reservedefinition.static inline int
amdgpu_bo_reserve(struct amdgpu_bo *bo, bool no_intr)Ah, this is a wired wrapper for
ttm_bo_reserve.Quoted 9 lines
When we call this function like the following:
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);The false means interruptible.
On the other hand, when
amdgpu_bo_reservefunction return error, why do we unref BO without kunmap and unpin the BO? This is wrong implementation whenamdgpu_bo_reservereturn any error.Yeah, I see your mean, it's in driver un-loading, How about changing to no interruptible? Your patch will make a memleak if
bo_reservefails, but it seems not matter. I have no strong preference.Regards,
David ZhouQuoted 31 lines
Thanks,
Alex Bin Xie------------------------------------------------------------------------
*From:* Zhou, David(ChunMing)
*Sent:* Friday, April 14, 2017 12:00 AM
*To:* Xie, AlexBin; [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 5 lines · Alex Xie
According to comment of
amdgpu_bo_reserve,amdgpu_bo_reservecan return with-ERESTARTSYS. When this function was interrupted by a signal, BO should not be unref. Otherwise the BO might be released while is kmapped and pinned, or BO MIGHT be deref multiple times, etc.r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);we have specified interruptible to false, so
-ERESTARTSYSisn't possible here.Thanks,
David ZhouQuoted 7 lines
Change-Id If76071a768950a0d3ad9d5da7fcae04881807621Signed-off-by Alex Xie <[email protected]>---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)Quoted 17 lines
Review
Quoted 1 line
(quoted diff) +0
-0
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.cdrivers/gpu/drm/amd/amdgpu/amdgpu_device.c static void amdgpu_vram_scratch_fini(struct
Quoted 4 lines
index 53996e3..1dcc2d1 100644
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0@@ -355,8 +355,8 @@ static void amdgpu_vram_scratch_fini(struct
amdgpu_device*adev)Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1amdgpu_bo_kunmap (adev->vram_scratch.robj);amdgpu_bo_unpin (adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +amdgpu_bo_unref (&adev->vram_scratch.robj); } -amdgpu_bo_unref (&adev->vram_scratch.robj); }/** Quoted 4 lines
_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfxQuoted 4 lines
_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): af065740-e219-fccb-4dac-d46f3472f4aa-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org
Quoted 7 lines · Christian König
Hi Alex,
Quoted 3 lines
No. For the current source code, I think the premap and no-op is not working.
Indeed, we don't set
bo->mem.bus.addrinamdgpu_ttm_io_mem_reserve()any more. Felix will probably want to fix that for the KFD branch.
I vaguely remember discussing this in the past: using mem->bus.addr to keep memory permanently CPU-mapped and avoid redundant ioremap calls. As I remember it, we weren't actually using this. It's only something we considered at one point.
Regards,
Felix
Quoted 639 lines
Anyway, as I said not unmapping the page tables is harmless compared to not releasing the memory backing it.
So please just do as I told you and change the interruptible reservation to a non-interruptible.
Regards,
Christian.Am 20.04.2017 um 23:56 schrieb Xie, AlexBin:
Quoted 624 lines
Hi Christian,
I read
amdgpu_ttm_io_mem_reserve()andamdgpu_ttm_io_mem_free()and relevant codes fromamdgpu_vram_scratch_init
andamdgpu_vram_scratch_fini.No. For the current source code, I think the premap and no-op is not working.
I add printk to prove. You can see
bo_kmap_typeis an invalid value.ioremap_wcis really called to map the IO range into vmalloc space....
Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759623] entering amdgpu_vram_scratch_init. Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759631] scratch ioremap_wc Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759631] bo_kmap_type = 129 Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759632] bus address = (null) Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759632] is_iomem = 1 Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759633]leaving
amdgpu_vram_scratch_init.
...I don't have log on rmmod AMDGPU yet. There are quite some settings to make that happen in my computer.
I think they are symemtric. Both mapping and unmapping are not no-op.Here is the printk patch for your reference.
From
25f95239c23225008e4b59f30b9b5363fb321f94Mon Sep 17 00:00:00 2001
From: Alex Xie <[email protected]>
Date: Thu, 20 Apr 2017 17:48:49 -0400
Subject: [PATCH] A hack to trace premap and noop.Change-Id I61fbbdbd82f62433e229b2e7e97be7a780ea8afaSigned-off-by Alex Xie <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 22 ++++++++++++++++++++++ drivers/gpu/drm/ttm/ttm_bo.c | 1 + drivers/gpu/drm/ttm/ttm_bo_util.c | 29 ++++++++++++++++++++++++++--- include/drm/ttm/ttm_bo_api.h | 1 + 4 files changed, 50 insertions(+), 3 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-0@@ -313,6 +313,7 @@ static void amdgpu_block_invalid_wreg(structamdgpu_device *adev, static int amdgpu_vram_scratch_init(struct amdgpu_device *adev) { int r; +printk ("entering amdgpu_vram_scratch_init." );if (adev->vram_scratch.robj ==NULL ) {r = amdgpu_bo_create(adev,
AMDGPU_GPU_PAGE_SIZE,drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +20
-0@@ -340,16 +341,36 @@ static int amdgpu_vram_scratch_init(struct amdgpu_device *adev) amdgpu_bo_unpin(adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +/* Would like a printk to see if map / unmap is noop*/ +adev->vram_scratch.robj->tbo.mem.bus.printk = true; + +if (adev->vram_scratch.robj->kmap.bo_kmap_type == ttm_bo_map_premapped) +printk ("amdgpu_vram_scratch premapped!" ); + +printk ("bo_kmap_type = %d" , adev->vram_scratch.robj->kmap.bo_kmap_type); +printk ("bus address = %p" , adev->vram_scratch.robj->tbo.mem.bus.addr); +printk ("is_iomem = %d" , adev->vram_scratch.robj->tbo.mem.bus.is_iomem); +printk ("leaving amdgpu_vram_scratch_init." ); +return r; }static void amdgpu_vram_scratch_fini (struct amdgpu_device *adev) {int r; +printk ("entering amdgpu_vram_scratch_fini." );if (adev->vram_scratch.robj ==NULL ) {return ; } + +if (adev->vram_scratch.robj->kmap.bo_kmap_type == ttm_bo_map_premapped) +printk ("amdgpu_vram_scratch premapped!" ); + +printk ("bo_kmap_type = %d" , adev->vram_scratch.robj->kmap.bo_kmap_type); +printk ("bus address = %p" , adev->vram_scratch.robj->tbo.mem.bus.addr); +printk ("is_iomem = %d" , adev->vram_scratch.robj->tbo.mem.bus.is_iomem); + r =amdgpu_bo_reserve (adev->vram_scratch.robj, false);if (likely (r ==0 )) {
amdgpu_bo_kunmap(adev->vram_scratch.robj);drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-0@@ -357,6 +378,7 @@ static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev) amdgpu_bo_unreserve(adev->vram_scratch.robj); }amdgpu_bo_unref (&adev->vram_scratch.robj); +printk ("leaving amdgpu_vram_scratch_fini." ); }/**
drivers/gpu/drm/ttm/ttm_bo.c +1
-0@@ -1178,6 +1178,7 @@ int ttm_bo_init_reserved(struct ttm_bo_device *bdev, bo->mem.page_alignment = page_alignment; bo->mem.bus.io_reserved_vm = false; bo->mem.bus.io_reserved_count =0 ; +bo->mem.bus.printk = false; bo->moving =NULL ; bo->mem.placement = (TTM_PL_FLAG_SYSTEM |TTM_PL_FLAG_CACHED );
bo->persistent_swap_storage=persistent_swap_storage;drivers/gpu/drm/ttm/ttm_bo_util.c +27
-3@@ -526,14 +526,24 @@ static int ttm_bo_ioremap(struct ttm_buffer_object *bo, if (bo->mem.bus.addr) { map->bo_kmap_type = ttm_bo_map_premapped; map->virtual = (void *)(((u8 *)bo->mem.bus.addr) + offset); +if (bo->mem.bus.printk) +printk ("scratch premapping" ); + } else { map->bo_kmap_type = ttm_bo_map_iomap; -if (mem->placement &TTM_PL_FLAG_WC ) +if (mem->placement &TTM_PL_FLAG_WC ) { map->virtual =ioremap_wc (bo->mem.bus.base + bo->mem.bus.offset + offset, size); -else +if (bo->mem.bus.printk) +printk ("scratch ioremap_wc" ); + +} +else { map->virtual =ioremap_nocache (bo->mem.bus.base + bo->mem.bus.offset + offset, size); +if (bo->mem.bus.printk) +printk ("scratch ioremap_nocache" ); +} }return (!map->virtual) ? -ENOMEM :0 ; }@@ -618,21 +628,34 @@ void ttm_bo_kunmap(struct ttm_bo_kmap_obj *map)struct ttm_mem_type_manager *man = &bo->bdev->man[bo->mem.mem_type]; -if (!map->virtual) +if (!map->virtual) { +if (bo->mem.bus.printk) +printk ("scratch unmap return earlier" );return ; +}switch (map->bo_kmap_type) {case ttm_bo_map_iomap: +if (bo->mem.bus.printk) +printk ("scratch iounmap" );iounmap (map->virtual);break ;case ttm_bo_map_vmap: +if (bo->mem.bus.printk) +printk ("scratch vunmap" );vunmap (map->virtual);break ;case ttm_bo_map_kmap: +if (bo->mem.bus.printk) +printk ("scratch kunmap" );kunmap (map->page);break ;case ttm_bo_map_premapped: +if (bo->mem.bus.printk) +printk ("scratch unmap ttm_bo_map_premapped" );break ;default : +if (bo->mem.bus.printk) +printk ("scratch unmap bug" );BUG (); } (void )ttm_mem_io_lock (man, false);include/drm/ttm/ttm_bo_api.h +1
-0@@ -70,6 +70,7 @@ struct ttm_bus_placement { boolis_iomem; boolio_reserved_vm;uint64_t io_reserved_count; +bool printk; };Signature · 16 lines
2.7.4 Thanks, Alex Bin ------------------------------------------------------------------------ *From:* Christian König <[email protected]> *Sent:* Thursday, April 20, 2017 4:43 AM *To:* Xie, AlexBin; Zhou, David(ChunMing); [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO Hi AlexBin,Quoted 2 lines
Missing kunmap mapping in vmalloc will make kernel master page table incorrect.
That's what I tried to explain yesterday, but unfortunately didn't had time to do so. There is not corruption of the kernel master page table in this case!
The call of
ttm_bo_kunmapis completely optional, take a look atamdgpu_ttm_io_mem_reserve()andamdgpu_ttm_io_mem_free().The aperture is kept mapped into the page tables for the whole time the driver is loaded. So this is a complete no-op and only done for consistency.
Quoted 4 lines
It is good that you agree that there is no real world bad example caused by my patch. I will not discuss whether it is an improvement or not now to save time for both of us.
Great at least we can now agree to completely drop this patch.
Thanks,
Christian.Am 19.04.2017 um 21:30 schrieb Xie, AlexBin:
Quoted 70 lines
Hi Christian,
Missing kunmap mapping in vmalloc will make kernel master page table incorrect. I would not call such issue as completely harmless. Please note that AMD graphic driver can run in 32 bit system. In 32 bit system, vmalloc address space is much smaller than size of most GPU VRAM.
amdgpu_bo_free_kernelhas same issue asamdgpu_vram_scratch_fini. 1. It callsamdgpu_bo_reserveinterruptible too. 2. It misses kunmap whenamdgpu_bo_reservereturns error too. As result, kernel master page table can become incorrect, or as you call it "completely harmless vmalloc space leaking".Because
amdgpu_bo_free_kernelis used in more places, such as psp command submission, there will be bigger chance to have other usage where signal is not blocked. This will become a real bug.I am thinking that we may fix the issue completely when TTM releases BO. But that is a bigger change.
It is good that you agree that there is no real world bad example caused by my patch. I will not discuss whether it is an improvement or not now to save time for both of us.
Thanks,
Alex Bin Xie
------------------------------------------------------------------------
*From:* Christian König <[email protected]>
*Sent:* Wednesday, April 19, 2017 7:50 AM
*To:* Xie, AlexBin; Zhou, David(ChunMing); [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 4 lines
Without correctly kunmap, page table is corrupted. Page entries point to wrong memory locations. You might call it completely harmless. But I think this is a severe bug. Leaking memory is better than a corrupted page table. Think security.
We are talking about the page tables for the vmalloc area in the kernel here, so no security problem. Leaking memory is much more problematic.
Quoted 4 lines
Would you provide any document and reference by saying" It is impossible to receive a signal during module load/unload"? For example, if the unload stuck in a lock, can CTRL+C stop the unload?
No, CTRL+C doesn't abort module load/unload. There have been patches to changes this a while ago, but IIRC it broke a whole bunch of driver relying on this.
Quoted 4 lines
What about there is some other return error? What about in future somebody improve
amdgpu_bo_reserveto return other errors, then functionamdgpu_vram_scratch_finibecomes buggy?Yes, that is indeed an issue. For example
-EDEADLKis possible as well. That's why I said we should useamdgpu_bo_free_kernel()instead.Quoted 4 lines
While I am thinking whether there is a better way for the current situation, would you give a real world example that my patch really not working? Then we can address it.
Quoted 294 lines
I don't think there is because the driver can't receive a signal during load/unload, but the problem is rather that the patch doesn't improve the situation at all.
Regards,
Christian.Am 19.04.2017 um 13:37 schrieb Xie, AlexBin:
Quoted 284 lines
Hi Christian,
Without correctly kunmap, page table is corrupted. Page entries point to wrong memory locations. You might call it completely harmless. But I think this is a severe bug. Leaking memory is better than a corrupted page table. Think security.
Would you provide any document and reference by saying" It is impossible to receive a signal during module load/unload"? For example, if the unload stuck in a lock, can CTRL+C stop the unload?
If "It is impossible to receive a signal during module load/unload", interruptible waiting is fine too, because function
amdgpu_bo_reservewill return successfully.What about there is some other return error? What about in future somebody improve
amdgpu_bo_reserveto return other errors, then functionamdgpu_vram_scratch_finibecomes buggy?While I am thinking whether there is a better way for the current situation, would you give a real world example that my patch really not working? Then we can address it.
Thanks,
Alex Bin
------------------------------------------------------------------------
*From:* Christian König <[email protected]> *Sent:* Wednesday, April 19, 2017 2:35 AM
*To:* Xie, AlexBin; Zhou, David(ChunMing); [email protected]
*Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOHi AlexBin,
the answer is
ttm_bo_kunmapisn't called at all and yes in the case of an iomap we leak the address space reserved.But that is completely harmless on a 64bit system compared to leaking the memory backing the address space.
Using
amdgpu_bo_free_kernel()instead of openly coding it here is probably a good idea.Additional to that it's probably a good idea to set the
no_intrflag when reserving kernel BOs. It is impossible to receive a signal during module load/unload, but it's probably better to document that in the code as well.Regards,
Christian.Am 18.04.2017 um 20:54 schrieb Xie, AlexBin:
Quoted 210 lines
Hi Christian,
Have you found how/where/when? When you said "mapping will just be released a bit later on", you must know the answer.
It is difficult to prove something does not exist. Anyway, I will give it a try to prove such "later on" does not exist.
Function
ttm_bo_kunmapis the only function to unmap. To prove this, searchttm_bo_map_iomap, onlyttm_bo_kunmapuse this enum to correctly kunmap.Function
ttm_bo_kunmapis not called by ttm itself. This is a hint that all TTM delay delete mechanism or unref mechanism will NOT kunmap BO later on.Function
ttm_bo_kunmapis called by AMDGPU functionamdgpu_bo_kunmapandamdgpu_gem_prime_vunmap.Search AMDGPU for
amdgpu_bo_kunmap. All matches do not kunmap for scratch VRAM BO.amdgpu_bo_free_kernelis a suspect but the answer is still NO.So all possibilities are searched. Did I miss anything?
Thanks,
Alex Bin Xie------------------------------------------------------------------------
*From:* Xie, AlexBin
*Sent:* Tuesday, April 18, 2017 2:04:33 PM
*To:* Christian König; Zhou, David(ChunMing); [email protected]
*Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOHi Christian,
Would you point out where/when will kunmap happen for this BO when release? It must be somewhere in some function calls.
I checked before I asked for review. But I did not see such obvious kunmap function call.
If so, there should be a comment in
functionamdgpu_vram_scratch_finito avoid future confusion.Thanks,
Alex Bin Xie
------------------------------------------------------------------------
*From:* Christian König <[email protected]> *Sent:* Tuesday, April 18, 2017 1:46 PM
*To:* Xie, AlexBin; Zhou, David(ChunMing); [email protected]
*Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOHi AlexBin,
No, David is right. This is a very common coding pattern in the kernel module.
Freeing up a BO while there still exists a kernel mapping is perfectly ok, the mapping will just be released a bit later on.
So this code is actually perfectly ok and just an optimization, but your patch breaks it and creates a memory leak.
Regards,
Christian.Am 18.04.2017 um 17:17 schrieb Xie, AlexBin:
Quoted 127 lines
Hi David,
When
amdgpu_bo_reservereturn errors, we cannot release the BO. This is not a memory leak. General speaking, memory leak is unnoticed and unintentional.The caller of function
amdgpu_vram_scratch_finiignores the return error value...The "memory leak" is not caused by my patch. It is caused because reserving BO fails.
This patch only aim to make function
amdgpu_vram_scratch_finibehave correctly.To follow up, we can add a warning message when
amdgpu_bo_reserveerror happens in a different patch.If function call
amdgpu_bo_reserveis changed to uninterruptible, this changes driver behaviour. Without a substantial issue, I would be cautious for such a change.Thanks,
Alex Bin Xie
------------------------------------------------------------------------
*From:* Zhou, David(ChunMing)
*Sent:* Monday, April 17, 2017 10:38 PM
*To:* Xie, AlexBin; [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 10 lines · AlexBin
Hi David,
Thanks for the comments. However, please have look at
amdgpu_bo_reservedefinition.static inline int
amdgpu_bo_reserve(struct amdgpu_bo *bo, bool no_intr)Ah, this is a wired wrapper for
ttm_bo_reserve.Quoted 11 lines
When we call this function like the following:
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);The false means interruptible.
On the other hand, when
amdgpu_bo_reservefunction return error, why do we unref BO without kunmap and unpin the BO? This is wrong implementation whenamdgpu_bo_reservereturn any error.Yeah, I see your mean, it's in driver un-loading, How about changing to no interruptible? Your patch will make a memleak if
bo_reservefails, but it seems not matter. I have no strong preference.Regards,
David ZhouQuoted 33 lines
Thanks,
Alex Bin Xie------------------------------------------------------------------------
*From:* Zhou, David(ChunMing)
*Sent:* Friday, April 14, 2017 12:00 AM
*To:* Xie, AlexBin; [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 5 lines · Alex Xie
According to comment of
amdgpu_bo_reserve,amdgpu_bo_reservecan return with-ERESTARTSYS. When this function was interrupted by a signal, BO should not be unref. Otherwise the BO might be released while is kmapped and pinned, or BO MIGHT be deref multiple times, etc.r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);we have specified interruptible to false, so
-ERESTARTSYSisn't possible
here.Thanks,
David ZhouQuoted 7 lines
Change-Id If76071a768950a0d3ad9d5da7fcae04881807621Signed-off-by Alex Xie <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Quoted 17 lines
Review
Quoted 1 line
(quoted diff) +0
-0
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.cdrivers/gpu/drm/amd/amdgpu/amdgpu_device.c static void
Quoted 4 lines
index 53996e3..1dcc2d1 100644
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0@@ -355,8 +355,8 @@ static void
amdgpu_vram_scratch_fini(struct amdgpu_device *adev)Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1amdgpu_bo_kunmap (adev->vram_scratch.robj);amdgpu_bo_unpin (adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +amdgpu_bo_unref (&adev->vram_scratch.robj); } -amdgpu_bo_unref (&adev->vram_scratch.robj); }/** Quoted 4 lines
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfxQuoted 4 lines
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Quoted 4 lines
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 003d5e5f-ea99-4517-fd4a-de4df3ce1b89-5C7GfCeVMHo@public.gmane.org
Am 21.04.2017 um 17:43 schrieb Felix Kuehling:
Quoted 6 lines · Christian König
Quoted 5 lines
Hi Alex,
Quoted 3 lines
No. For the current source code, I think the premap and no-op is not working.
Quoted 6 lines
Quoted 2 lines
Indeed, we don't set
bo->mem.bus.addrinamdgpu_ttm_io_mem_reserve()any more. Felix will probably want to fix that for the KFD branch.I vaguely remember discussing this in the past: using
mem->bus.addrto keep memory permanently CPU-mapped and avoid redundant ioremap calls. As I remember it, we weren't actually using this. It's only something we considered at one point.
Well that explains why my memory fooled me.
My last status was that we implemented this together with CPU based page table updates.
Otherwise mapping them for CPU access is rather heavy if we indeed update the page tables every time.
Christian.
Quoted 627 lines
Regards,
FelixQuoted 623 lines
Anyway, as I said not unmapping the page tables is harmless compared to not releasing the memory backing it.
So please just do as I told you and change the interruptible reservation to a non-interruptible.
Regards,
Christian.Am 20.04.2017 um 23:56 schrieb Xie, AlexBin:
Quoted 610 lines
Hi Christian,
I read
amdgpu_ttm_io_mem_reserve()andamdgpu_ttm_io_mem_free()and relevant codes fromamdgpu_vram_scratch_init
andamdgpu_vram_scratch_fini.No. For the current source code, I think the premap and no-op is not working.
I add printk to prove. You can see
bo_kmap_typeis an invalid value.ioremap_wcis really called to map the IO range into vmalloc space....
Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759623] entering amdgpu_vram_scratch_init. Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759631] scratch ioremap_wc Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759631] bo_kmap_type = 129 Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759632] bus address = (null) Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759632] is_iomem = 1 Apr 20 16:31:18 axie-System-Product-Name kernel: [ 106.759633]leaving
amdgpu_vram_scratch_init.
...I don't have log on rmmod AMDGPU yet. There are quite some settings to make that happen in my computer.
I think they are symemtric. Both mapping and unmapping are not no-op.Here is the printk patch for your reference.
From
25f95239c23225008e4b59f30b9b5363fb321f94Mon Sep 17 00:00:00 2001
From: Alex Xie <[email protected]>
Date: Thu, 20 Apr 2017 17:48:49 -0400
Subject: [PATCH] A hack to trace premap and noop.Change-Id I61fbbdbd82f62433e229b2e7e97be7a780ea8afaSigned-off-by Alex Xie <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 22 ++++++++++++++++++++++ drivers/gpu/drm/ttm/ttm_bo.c | 1 + drivers/gpu/drm/ttm/ttm_bo_util.c | 29 ++++++++++++++++++++++++++--- include/drm/ttm/ttm_bo_api.h | 1 + 4 files changed, 50 insertions(+), 3 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-0@@ -313,6 +313,7 @@ static void amdgpu_block_invalid_wreg(structamdgpu_device *adev, static int amdgpu_vram_scratch_init(struct amdgpu_device *adev) { int r; +printk ("entering amdgpu_vram_scratch_init." );if (adev->vram_scratch.robj ==NULL ) {r = amdgpu_bo_create(adev,
AMDGPU_GPU_PAGE_SIZE,drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +20
-0@@ -340,16 +341,36 @@ static int amdgpu_vram_scratch_init(struct amdgpu_device *adev) amdgpu_bo_unpin(adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +/* Would like a printk to see if map / unmap is noop*/ +adev->vram_scratch.robj->tbo.mem.bus.printk = true; + +if (adev->vram_scratch.robj->kmap.bo_kmap_type == ttm_bo_map_premapped) +printk ("amdgpu_vram_scratch premapped!" ); + +printk ("bo_kmap_type = %d" , adev->vram_scratch.robj->kmap.bo_kmap_type); +printk ("bus address = %p" , adev->vram_scratch.robj->tbo.mem.bus.addr); +printk ("is_iomem = %d" , adev->vram_scratch.robj->tbo.mem.bus.is_iomem); +printk ("leaving amdgpu_vram_scratch_init." ); +return r; }static void amdgpu_vram_scratch_fini (struct amdgpu_device *adev) {int r; +printk ("entering amdgpu_vram_scratch_fini." );if (adev->vram_scratch.robj ==NULL ) {return ; } + +if (adev->vram_scratch.robj->kmap.bo_kmap_type == ttm_bo_map_premapped) +printk ("amdgpu_vram_scratch premapped!" ); + +printk ("bo_kmap_type = %d" , adev->vram_scratch.robj->kmap.bo_kmap_type); +printk ("bus address = %p" , adev->vram_scratch.robj->tbo.mem.bus.addr); +printk ("is_iomem = %d" , adev->vram_scratch.robj->tbo.mem.bus.is_iomem); + r =amdgpu_bo_reserve (adev->vram_scratch.robj, false);if (likely (r ==0 )) {amdgpu_bo_kunmap(adev->vram_scratch.robj);drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-0@@ -357,6 +378,7 @@ static void amdgpu_vram_scratch_fini(struct amdgpu_device *adev) amdgpu_bo_unreserve(adev->vram_scratch.robj); }amdgpu_bo_unref (&adev->vram_scratch.robj); +printk ("leaving amdgpu_vram_scratch_fini." ); }/**
drivers/gpu/drm/ttm/ttm_bo.c +1
-0@@ -1178,6 +1178,7 @@ int ttm_bo_init_reserved(struct ttm_bo_device *bdev, bo->mem.page_alignment = page_alignment; bo->mem.bus.io_reserved_vm = false; bo->mem.bus.io_reserved_count =0 ; +bo->mem.bus.printk = false; bo->moving =NULL ; bo->mem.placement = (TTM_PL_FLAG_SYSTEM |TTM_PL_FLAG_CACHED );bo->persistent_swap_storage = persistent_swap_storage;drivers/gpu/drm/ttm/ttm_bo_util.c +27
-3@@ -526,14 +526,24 @@ static int ttm_bo_ioremap(struct ttm_buffer_object *bo, if (bo->mem.bus.addr) { map->bo_kmap_type = ttm_bo_map_premapped; map->virtual = (void *)(((u8 *)bo->mem.bus.addr) + offset); +if (bo->mem.bus.printk) +printk ("scratch premapping" ); + } else { map->bo_kmap_type = ttm_bo_map_iomap; -if (mem->placement &TTM_PL_FLAG_WC ) +if (mem->placement &TTM_PL_FLAG_WC ) { map->virtual =ioremap_wc (bo->mem.bus.base + bo->mem.bus.offset + offset, size); -else +if (bo->mem.bus.printk) +printk ("scratch ioremap_wc" ); + +} +else { map->virtual =ioremap_nocache (bo->mem.bus.base + bo->mem.bus.offset + offset, size); +if (bo->mem.bus.printk) +printk ("scratch ioremap_nocache" ); +} }return (!map->virtual) ? -ENOMEM :0 ; }@@ -618,21 +628,34 @@ void ttm_bo_kunmap(struct ttm_bo_kmap_obj *map)struct ttm_mem_type_manager *man = &bo->bdev->man[bo->mem.mem_type]; -if (!map->virtual) +if (!map->virtual) { +if (bo->mem.bus.printk) +printk ("scratch unmap return earlier" );return ; +}switch (map->bo_kmap_type) {case ttm_bo_map_iomap: +if (bo->mem.bus.printk) +printk ("scratch iounmap" );iounmap (map->virtual);break ;case ttm_bo_map_vmap: +if (bo->mem.bus.printk) +printk ("scratch vunmap" );vunmap (map->virtual);break ;case ttm_bo_map_kmap: +if (bo->mem.bus.printk) +printk ("scratch kunmap" );kunmap (map->page);break ;case ttm_bo_map_premapped: +if (bo->mem.bus.printk) +printk ("scratch unmap ttm_bo_map_premapped" );break ;default : +if (bo->mem.bus.printk) +printk ("scratch unmap bug" );BUG (); } (void )ttm_mem_io_lock (man, false);include/drm/ttm/ttm_bo_api.h +1
-0@@ -70,6 +70,7 @@ struct ttm_bus_placement { boolis_iomem; boolio_reserved_vm;uint64_t io_reserved_count; +bool printk; };Signature · 16 lines
2.7.4 Thanks, Alex Bin ------------------------------------------------------------------------ *From:* Christian König <[email protected]> *Sent:* Thursday, April 20, 2017 4:43 AM *To:* Xie, AlexBin; Zhou, David(ChunMing); [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BO Hi AlexBin,Quoted 2 lines
Missing kunmap mapping in vmalloc will make kernel master page table incorrect.
That's what I tried to explain yesterday, but unfortunately didn't had time to do so. There is not corruption of the kernel master page table in this case!
The call of
ttm_bo_kunmapis completely optional, take a look atamdgpu_ttm_io_mem_reserve()andamdgpu_ttm_io_mem_free().The aperture is kept mapped into the page tables for the whole time the driver is loaded. So this is a complete no-op and only done for consistency.
Quoted 4 lines
It is good that you agree that there is no real world bad example caused by my patch. I will not discuss whether it is an improvement or not now to save time for both of us.
Great at least we can now agree to completely drop this patch.
Thanks,
Christian.Am 19.04.2017 um 21:30 schrieb Xie, AlexBin:
Quoted 69 lines
Hi Christian,
Missing kunmap mapping in vmalloc will make kernel master page table incorrect. I would not call such issue as completely harmless. Please note that AMD graphic driver can run in 32 bit system. In 32 bit system, vmalloc address space is much smaller than size of most GPU VRAM.
amdgpu_bo_free_kernelhas same issue asamdgpu_vram_scratch_fini. 1. It callsamdgpu_bo_reserveinterruptible too. 2. It misses kunmap whenamdgpu_bo_reservereturns error too. As result, kernel master page table can become incorrect, or as you call it "completely harmless vmalloc space leaking".Because
amdgpu_bo_free_kernelis used in more places, such as psp command submission, there will be bigger chance to have other usage where signal is not blocked. This will become a real bug.I am thinking that we may fix the issue completely when TTM releases BO. But that is a bigger change.
It is good that you agree that there is no real world bad example caused by my patch. I will not discuss whether it is an improvement or not now to save time for both of us.
Thanks,
Alex Bin Xie
------------------------------------------------------------------------
*From:* Christian König <[email protected]>
*Sent:* Wednesday, April 19, 2017 7:50 AM
*To:* Xie, AlexBin; Zhou, David(ChunMing); [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 4 lines
Without correctly kunmap, page table is corrupted. Page entries point to wrong memory locations. You might call it completely harmless. But I think this is a severe bug. Leaking memory is better than a corrupted page table. Think security.
We are talking about the page tables for the vmalloc area in the kernel here, so no security problem. Leaking memory is much more problematic.
Quoted 4 lines
Would you provide any document and reference by saying" It is impossible to receive a signal during module load/unload"? For example, if the unload stuck in a lock, can CTRL+C stop the unload?
No, CTRL+C doesn't abort module load/unload. There have been patches to changes this a while ago, but IIRC it broke a whole bunch of driver relying on this.
Quoted 4 lines
What about there is some other return error? What about in future somebody improve
amdgpu_bo_reserveto return other errors, then functionamdgpu_vram_scratch_finibecomes buggy?Yes, that is indeed an issue. For example
-EDEADLKis possible as well. That's why I said we should useamdgpu_bo_free_kernel()instead.Quoted 4 lines
While I am thinking whether there is a better way for the current situation, would you give a real world example that my patch really not working? Then we can address it.
Quoted 283 lines
I don't think there is because the driver can't receive a signal during load/unload, but the problem is rather that the patch doesn't improve the situation at all.
Regards,
Christian.Am 19.04.2017 um 13:37 schrieb Xie, AlexBin:
Quoted 274 lines
Hi Christian,
Without correctly kunmap, page table is corrupted. Page entries point to wrong memory locations. You might call it completely harmless. But I think this is a severe bug. Leaking memory is better than a corrupted page table. Think security.
Would you provide any document and reference by saying" It is impossible to receive a signal during module load/unload"? For example, if the unload stuck in a lock, can CTRL+C stop the unload?
If "It is impossible to receive a signal during module load/unload", interruptible waiting is fine too, because function
amdgpu_bo_reservewill return successfully.What about there is some other return error? What about in future somebody improve
amdgpu_bo_reserveto return other errors, then functionamdgpu_vram_scratch_finibecomes buggy?While I am thinking whether there is a better way for the current situation, would you give a real world example that my patch really not working? Then we can address it.
Thanks,
Alex Bin
------------------------------------------------------------------------
*From:* Christian König <[email protected]> *Sent:* Wednesday, April 19, 2017 2:35 AM
*To:* Xie, AlexBin; Zhou, David(ChunMing); [email protected]
*Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOHi AlexBin,
the answer is
ttm_bo_kunmapisn't called at all and yes in the case of an iomap we leak the address space reserved.But that is completely harmless on a 64bit system compared to leaking the memory backing the address space.
Using
amdgpu_bo_free_kernel()instead of openly coding it here is probably a good idea.Additional to that it's probably a good idea to set the
no_intrflag when reserving kernel BOs. It is impossible to receive a signal during module load/unload, but it's probably better to document that in the code as well.Regards,
Christian.Am 18.04.2017 um 20:54 schrieb Xie, AlexBin:
Quoted 202 lines
Hi Christian,
Have you found how/where/when? When you said "mapping will just be released a bit later on", you must know the answer.
It is difficult to prove something does not exist. Anyway, I will give it a try to prove such "later on" does not exist.
Function
ttm_bo_kunmapis the only function to unmap. To prove this, searchttm_bo_map_iomap, onlyttm_bo_kunmapuse this enum to correctly kunmap.Function
ttm_bo_kunmapis not called by ttm itself. This is a hint that all TTM delay delete mechanism or unref mechanism will NOT kunmap BO later on.Function
ttm_bo_kunmapis called by AMDGPU functionamdgpu_bo_kunmapandamdgpu_gem_prime_vunmap.Search AMDGPU for
amdgpu_bo_kunmap. All matches do not kunmap for scratch VRAM BO.amdgpu_bo_free_kernelis a suspect but the answer is still NO.So all possibilities are searched. Did I miss anything?
Thanks,
Alex Bin Xie------------------------------------------------------------------------
*From:* Xie, AlexBin
*Sent:* Tuesday, April 18, 2017 2:04:33 PM
*To:* Christian König; Zhou, David(ChunMing); [email protected]
*Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOHi Christian,
Would you point out where/when will kunmap happen for this BO when release? It must be somewhere in some function calls.
I checked before I asked for review. But I did not see such obvious kunmap function call.
If so, there should be a comment in
functionamdgpu_vram_scratch_finito avoid future confusion.Thanks,
Alex Bin Xie
------------------------------------------------------------------------
*From:* Christian König <[email protected]> *Sent:* Tuesday, April 18, 2017 1:46 PM
*To:* Xie, AlexBin; Zhou, David(ChunMing); [email protected]
*Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOHi AlexBin,
No, David is right. This is a very common coding pattern in the kernel module.
Freeing up a BO while there still exists a kernel mapping is perfectly ok, the mapping will just be released a bit later on.
So this code is actually perfectly ok and just an optimization, but your patch breaks it and creates a memory leak.
Regards,
Christian.Am 18.04.2017 um 17:17 schrieb Xie, AlexBin:
Quoted 117 lines
Hi David,
When
amdgpu_bo_reservereturn errors, we cannot release the BO. This is not a memory leak. General speaking, memory leak is unnoticed and unintentional.The caller of function
amdgpu_vram_scratch_finiignores the return error value...The "memory leak" is not caused by my patch. It is caused because reserving BO fails.
This patch only aim to make function
amdgpu_vram_scratch_finibehave correctly.To follow up, we can add a warning message when
amdgpu_bo_reserveerror happens in a different patch.If function call
amdgpu_bo_reserveis changed to uninterruptible, this changes driver behaviour. Without a substantial issue, I would be cautious for such a change.Thanks,
Alex Bin Xie
------------------------------------------------------------------------
*From:* Zhou, David(ChunMing)
*Sent:* Monday, April 17, 2017 10:38 PM
*To:* Xie, AlexBin; [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 9 lines · AlexBin
Hi David,
Thanks for the comments. However, please have look at
amdgpu_bo_reservedefinition.static inline int
amdgpu_bo_reserve(struct amdgpu_bo *bo, bool no_intr)Ah, this is a wired wrapper for
ttm_bo_reserve.Quoted 10 lines
When we call this function like the following:
r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);The false means interruptible.
On the other hand, when
amdgpu_bo_reservefunction return error, why do we unref BO without kunmap and unpin the BO? This is wrong implementation whenamdgpu_bo_reservereturn any error.Yeah, I see your mean, it's in driver un-loading, How about changing to no interruptible? Your patch will make a memleak if
bo_reservefails, but it seems not matter. I have no strong preference.Regards,
David ZhouQuoted 46 lines
Thanks,
Alex Bin Xie------------------------------------------------------------------------
*From:* Zhou, David(ChunMing)
*Sent:* Friday, April 14, 2017 12:00 AM
*To:* Xie, AlexBin; [email protected] *Subject:* Re: [PATCH] dmr/amdgpu: Fix wrongly unref of BOQuoted 5 lines · Alex Xie
According to comment of
amdgpu_bo_reserve,amdgpu_bo_reservecan return with-ERESTARTSYS. When this function was interrupted by a signal, BO should not be unref. Otherwise the BO might be released while is kmapped and pinned, or BO MIGHT be deref multiple times, etc.r = amdgpu_bo_reserve(adev->vram_scratch.robj, false);we have specified interruptible to false, so
-ERESTARTSYSisn't possible
here.Thanks,
David ZhouReview
Quoted 7 lines
Change-Id If76071a768950a0d3ad9d5da7fcae04881807621Signed-off-by Alex Xie <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)(quoted diff) +0
-0
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.cdrivers/gpu/drm/amd/amdgpu/amdgpu_device.c static void
Quoted 4 lines
index 53996e3..1dcc2d1 100644
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0@@ -355,8 +355,8 @@ static void
amdgpu_vram_scratch_fini(struct amdgpu_device *adev)Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1amdgpu_bo_kunmap (adev->vram_scratch.robj);amdgpu_bo_unpin (adev->vram_scratch.robj);amdgpu_bo_unreserve (adev->vram_scratch.robj); +amdgpu_bo_unref (&adev->vram_scratch.robj); } -amdgpu_bo_unref (&adev->vram_scratch.robj); }Quoted 7 lines
Quoted 1 line
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static void · line 362
Quoted 1 line
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0/** _______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfxQuoted 4 lines
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Quoted 8 lines
Quoted 4 lines
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): a8715aca-05ff-1262-e2f5-31279052e606-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org
Quoted 14 lines · Christian König
Am 21.04.2017 um 17:43 schrieb Felix Kuehling:
Quoted 6 lines · Christian König
Quoted 5 lines
Hi Alex,
Quoted 3 lines
No. For the current source code, I think the premap and no-op is not working.
Quoted 6 lines
Quoted 2 lines
Indeed, we don't set
bo->mem.bus.addrinamdgpu_ttm_io_mem_reserve()any more. Felix will probably want to fix that for the KFD branch.I vaguely remember discussing this in the past: using
mem->bus.addrto keep memory permanently CPU-mapped and avoid redundant ioremap calls. As I remember it, we weren't actually using this. It's only something we considered at one point.
Quoted 7 lines
Well that explains why my memory fooled me.
My last status was that we implemented this together with CPU based page table updates.
Otherwise mapping them for CPU access is rather heavy if we indeed update the page tables every time.
We also don't currently have CPU page table updates for KFD. When we rebased on 4.9, the VM code was restructured so much that we essentially had to go back and reimplement it. Harish is working on this right now.
Regards,
Felix
Quoted 3 lines
Christian.
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx