Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3B6EBC433EF for ; Thu, 21 Apr 2022 06:26:33 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id BD2FD10F452; Thu, 21 Apr 2022 06:26:32 +0000 (UTC) Received: from mail-ej1-x634.google.com (mail-ej1-x634.google.com [IPv6:2a00:1450:4864:20::634]) by gabe.freedesktop.org (Postfix) with ESMTPS id F1F2110F44E; Thu, 21 Apr 2022 06:26:30 +0000 (UTC) Received: by mail-ej1-x634.google.com with SMTP id u15so7854985ejf.11; Wed, 20 Apr 2022 23:26:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:date:mime-version:user-agent:subject:content-language:to :references:from:in-reply-to; bh=cd1egrVMwW6VynWrNSFyUHI/QPQ/T4PEowZfARpEMYo=; b=OcHukgust1pXfGtFgT4ZSrPYVWPPReFZv1OKeBBvlTAbJMG+zbyO1XGHFsWmispyx+ L4ANORg4QZ50gMscSf2rAJlDhZmyM0JG+xv5BCdvVNfUuCalsbpa+93u3YaviyD7Ko3Z Y3KC0FOARL1TazkV8fzHleniLoOal9sbgtUkaDhqC3d7ApEtmFoYSdcNYtIs0eIkL6bY 009NksmndEKHSE9CNDqfFPKptH4XHu0vtcHL6C0iH3xduqOUS2ZcnqTYM+kDw3Dxo+Bi /O7VdT6vZ9bEKu7B1hxbAk27jCiL4b2RIMt9QmChdvvJIAoNAfzUJQIWW2aZIx36YhYB 5ieQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:references:from:in-reply-to; bh=cd1egrVMwW6VynWrNSFyUHI/QPQ/T4PEowZfARpEMYo=; b=6GuAhFAUboyR1JbbZKC0/k/NjWSNZaqX/t3PgU3b3KmMbwEy6v1WUBChjFm2XgB705 N1zHG4vzL/5SGiRmbvbgTluBPJt/rwKsA6yMG1F4lZbJlAzCLR4ctITYybPI+SiEOdhv i0dVSEZGEYiIV2LkjRWZ8vD9hvMhqCAs6gMIfLjoP9hAH9LDRpd2NtaiwNoqiX1YDdrO VLkzfaTJDCiXkelMEGAap0WUNZU2M3lL9Yv9E49M23yxyXsNIbo498mwxrMplvHvJwUO L/POlK4pEnqUoIdDiv00ItNZ4nFQSmoxMnnVw4gxPhLQ2WxYZVfpfmQVug1ctKr6lRNJ /ZZQ== X-Gm-Message-State: AOAM530gqYT/zojiCKK0NisBAQWso1uK2P6kk5txruUXOsqVn5NyFowW bSPS0Zl+W5Gs+RcYEnZ5fN4= X-Google-Smtp-Source: ABdhPJxU7lLXxpZENOm11+GPWGbpLlKpzt/pl7vrTjNHecZMJg29IpbJl7RzSZz7ECwc0eMfANF6+A== X-Received: by 2002:a17:907:2ce5:b0:6f0:1d2c:118d with SMTP id hz5-20020a1709072ce500b006f01d2c118dmr1439715ejc.715.1650522389220; Wed, 20 Apr 2022 23:26:29 -0700 (PDT) Received: from ?IPV6:2a02:908:1252:fb60:74ec:39fd:9a7c:7ff3? ([2a02:908:1252:fb60:74ec:39fd:9a7c:7ff3]) by smtp.gmail.com with ESMTPSA id wc23-20020a170907125700b006efa5c8ba7esm4240546ejb.116.2022.04.20.23.26.28 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 20 Apr 2022 23:26:28 -0700 (PDT) Content-Type: multipart/alternative; boundary="------------vdznHasFmNx4e2RAlAoQXvZx" Message-ID: <8ce90f44-d318-65ba-045a-18cbecf1734a@gmail.com> Date: Thu, 21 Apr 2022 08:26:27 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.7.0 Subject: Re: [PATCH] drm/ttm: fix ttm tt init fail when size exceeds kmalloc limit Content-Language: en-US To: "Wang, Yang(Kevin)" , "Kuehling, Felix" , "Lazar, Lijo" , "Koenig, Christian" , "dri-devel@lists.freedesktop.org" , "amd-gfx@lists.freedesktop.org" References: <20220420085612.1664787-1-KevinYang.Wang@amd.com> <00dfec71-cf38-d6d9-8775-0ae88f59224e@amd.com> <65ef352b-59c8-e802-5285-afc59e56fa9b@amd.com> <572ed15c-b47e-e7b3-850a-cb0d5259af04@gmail.com> <71f3ac77-b6ed-e503-8e05-dcf9d2c2811a@amd.com> <075258d7-16af-a664-0406-2b2862efe05f@amd.com> <6d1ec935-3b41-fef6-efe3-577f8316edd2@amd.com> From: =?UTF-8?Q?Christian_K=c3=b6nig?= In-Reply-To: X-BeenThere: amd-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion list for AMD gfx List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: amd-gfx-bounces@lists.freedesktop.org Sender: "amd-gfx" This is a multi-part message in MIME format. --------------vdznHasFmNx4e2RAlAoQXvZx Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Am 21.04.22 um 04:15 schrieb Wang, Yang(Kevin): > > [AMD Official Use Only] > > > > > ------------------------------------------------------------------------ > *From:* Kuehling, Felix > *Sent:* Thursday, April 21, 2022 5:21 AM > *To:* Lazar, Lijo ; Koenig, Christian > ; Wang, Yang(Kevin) > ; Christian König > ; dri-devel@lists.freedesktop.org > ; amd-gfx@lists.freedesktop.org > > *Subject:* Re: [PATCH] drm/ttm: fix ttm tt init fail when size exceeds > kmalloc limit > > On 2022-04-20 09:23, Lazar, Lijo wrote: > > > > > > On 4/20/2022 6:26 PM, Christian König wrote: > >> Am 20.04.22 um 14:54 schrieb Wang, Yang(Kevin): > >>> > >>> [AMD Official Use Only] > >>> > >>> > >>> Hi Chris, > >>> > >>> 1) Change the test case to use something larger than 1TiB. > >>> sure, we can increase the size of BO and make test pass, > >>> but if user really want to allocate 1TB GTT BO, we have no reason to > >>> let it fail? right? > >> > >> No, the reason is the underlying core kernel doesn't allow kvmalloc > >> allocations with GFP_ZERO which are large enough to hold the array of > >> allocated pages for this. > >> > >> We are working on top of the core Linux kernel and should *NEVER* > >> ever add workarounds like what was suggested here. > > > > > AFAIU, for the purpose of ttm use, fallback to vmalloc is fine. > > > >  * Please note that any use of gfp flags outside of GFP_KERNEL is > > careful to not > >  * fall back to vmalloc. > >  * > > That's weird, because kvcalloc does the same thing. If that were not > able to fall back to vmalloc, it would be pretty useless. > >     static inline __alloc_size(1, 2) void *kvcalloc(size_t n, size_t > size, gfp_t flags) >     { >              return kvmalloc_array(n, size, flags | __GFP_ZERO); >     } > > Maybe kvcalloc is the function we TTM should be using here anyway, > instead of open-coding the kvmalloc_array call with an extra GFP flag. > > Regards, >    Felix > > Yes, I agree with your point, and in amdkfd driver code, we have the > same risk in svm_range_dma_map_dev(). Yes, sounds like a good idea to me as well to change that. Regards, Christian. > > Best Regards, > Kevin > > > > > Actually the current implementation documents the behavior, but it is > > deep inside the implementation to be noticeable - at least not obvious > > while using kvmalloc_array. > > > > Thanks, > > Lijo > > > >> Regards, > >> Christian. > >> > >>> the system availed memory about 2T, but it will still fail. > >>> > >>> 2) Change kvmalloc to allow GFP_ZERO allocations even in the vmalloc > >>> fallback path. > >>>     the 5.18 kernel will add this patch to fix this issue . > >>> > >>> Best Regards, > >>> Kevin > >>> > ------------------------------------------------------------------------ > >>> > >>> *From:* Koenig, Christian > >>> *Sent:* Wednesday, April 20, 2022 8:42 PM > >>> *To:* Wang, Yang(Kevin) ; Christian König > >>> ; dri-devel@lists.freedesktop.org > >>> ; amd-gfx@lists.freedesktop.org > >>> > >>> *Subject:* Re: [PATCH] drm/ttm: fix ttm tt init fail when size > >>> exceeds kmalloc limit > >>> Hi Kevin, > >>> > >>> yes and that is perfectly valid and expected behavior. There is > >>> absolutely no need to change anything in TTM here. > >>> > >>> What we could do is: > >>> 1) Change the test case to use something larger than 1TiB. > >>> 2) Change kvmalloc to allow GFP_ZERO allocations even in the vmalloc > >>> fallback path. > >>> > >>> Regards, > >>> Christian. > >>> > >>> Am 20.04.22 um 14:39 schrieb Wang, Yang(Kevin): > >>>> > >>>> [AMD Official Use Only] > >>>> > >>>> > >>>> Hi Chirs, > >>>> > >>>> yes, right, the amdgpu drive rwill use amdgpu_bo_validate_size() > >>>> function to verify bo size, > >>>> but when driver try to allocate VRAM domain bo fail, the amdgpu > >>>> driver will fall back to allocate domain = (GTT | VRAM)  bo. > >>>> please check following code, it will cause the 2nd time to allocate > >>>> bo fail during allocate 256Mb buffer to store dma address (via > >>>> kvmalloc()). > >>>> > >>>> initial_domain = (u32)(0xffffffff & args->in.domains); > >>>> retry: > >>>>         r = amdgpu_gem_object_create(adev, size, args->in.alignment, > >>>>                    initial_domain, > >>>>                    flags, ttm_bo_type_device, resv, &gobj); > >>>>         if (r && r != -ERESTARTSYS) { > >>>>                 if (flags & AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED) { > >>>>       flags &= ~AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED; > >>>>       goto retry; > >>>>                 } > >>>> > >>>>                 if (initial_domain == AMDGPU_GEM_DOMAIN_VRAM) { > >>>>       initial_domain |= AMDGPU_GEM_DOMAIN_GTT; > >>>>       goto retry; > >>>>                 } > >>>> DRM_DEBUG("Failed to allocate GEM object (%llu, %d, %llu, %d)\n", > >>>>               size, initial_domain, args->in.alignment, r); > >>>>         } > >>>> > >>>> Best Regards, > >>>> Kevin > >>>> > >>>> > ------------------------------------------------------------------------ > >>>> > >>>> *From:* Christian König > >>>> > > >>>> *Sent:* Wednesday, April 20, 2022 7:55 PM > >>>> *To:* Wang, Yang(Kevin) > >>>> >; > Koenig, Christian > >>>> >; > >>>> dri-devel@lists.freedesktop.org > >>>> > > >>>> > >>>> >; > >>>> amd-gfx@lists.freedesktop.org > >>>> > > >>>> > > > >>>> *Subject:* Re: [PATCH] drm/ttm: fix ttm tt init fail when size > >>>> exceeds kmalloc limit > >>>> Hi Kevin, > >>>> > >>>> no, the test case should already fail in amdgpu_bo_validate_size(). > >>>> > >>>> If we have a system with 2TiB of memory where the test case could > >>>> succeed then we should increase the requested size to something > >>>> larger. > >>>> > >>>> And if the underlying core Linux kernel functions don't allow > >>>> allocations as large as the requested one we should *NEVER* ever > >>>> add workarounds like this. > >>>> > >>>> It is perfectly expected that this test case is not able to fulfill > >>>> the desired allocation. That it fails during kvmalloc is > >>>> unfortunate, but not a show stopper. > >>>> > >>>> Regards, > >>>> Christian. > >>>> > >>>> > >>>> Am 20.04.22 um 13:32 schrieb Wang, Yang(Kevin): > >>>>> > >>>>> [AMD Official Use Only] > >>>>> > >>>>> > >>>>> Hi Chris, > >>>>> > >>>>> you misunderstood background about this case. > >>>>> > >>>>> although we expect this test case to fail, it should fail at the > >>>>> location where the Bo actual memory is actually allocated. now the > >>>>> code logic will cause the failure to allocate memory to store DMA > >>>>> address. > >>>>> > >>>>> e.g: the case is failed in 2TB system ram machine, it should be > >>>>> allocated successful, but it is failed. > >>>>> > >>>>> allocate 1TB BO, the ttm should allocate 1TB/4k * 8 buffer to > >>>>> store allocate result (page address), this should not be failed > >>>>> usually. > >>>>> > >>>>> There is a similar fix in upstream kernel 5.18, before this fix > >>>>> entered the TTM code, this problem existed in TTM. > >>>>> > >>>>> kernel/git/torvalds/linux.git - Linux kernel source tree > >>>>> > >>>>> mm: allow !GFP_KERNEL allocations for kvmalloc > >>>>> > >>>>> Best Regards, > >>>>> Kevin > >>>>> > >>>>> > ------------------------------------------------------------------------ > >>>>> > >>>>> *From:* Koenig, Christian > >>>>> > > >>>>> *Sent:* Wednesday, April 20, 2022 6:53 PM > >>>>> *To:* Wang, Yang(Kevin) > >>>>> >; > dri-devel@lists.freedesktop.org > >>>>> > > >>>>> > >>>>> >; > >>>>> amd-gfx@lists.freedesktop.org > >>>>> > > >>>>> > >>>>> > > >>>>> *Subject:* Re: [PATCH] drm/ttm: fix ttm tt init fail when size > >>>>> exceeds kmalloc limit > >>>>> Am 20.04.22 um 11:07 schrieb Wang, Yang(Kevin): > >>>>>> > >>>>>> [AMD Official Use Only] > >>>>>> > >>>>>> > >>>>>> > >>>>>> > ------------------------------------------------------------------------ > >>>>>> > >>>>>> *From:* Koenig, Christian > >>>>>> > > >>>>>> *Sent:* Wednesday, April 20, 2022 5:00 PM > >>>>>> *To:* Wang, Yang(Kevin) > >>>>>> >; dri-devel@lists.freedesktop.org > >>>>>> > > >>>>>> > >>>>>> >; > >>>>>> amd-gfx@lists.freedesktop.org > >>>>>> > > >>>>>> > >>>>>> > > >>>>>> *Subject:* Re: [PATCH] drm/ttm: fix ttm tt init fail when size > >>>>>> exceeds kmalloc limit > >>>>>> Am 20.04.22 um 10:56 schrieb Yang Wang: > >>>>>> > if the __GFP_ZERO is set, the kvmalloc() can't fallback to use > >>>>>> vmalloc() > >>>>>> > >>>>>> Hui what? Why should kvmalloc() not be able to fallback to > vmalloc() > >>>>>> when __GFP_ZERO is set? > >>>>>> > >>>>>> And even that is really the case then that sounds like a bug in > >>>>>> kvmalloc(). > >>>>>> > >>>>>> Regards, > >>>>>> Christian. > >>>>>> > >>>>>> [kevin]: > >>>>>> it is really test case from libdrm amdgpu test, which try to > >>>>>> allocate a big BO which will cause ttm tt init fail. > >>>>> > >>>>> > >>>>> LOL! Guys, this test case is intended to fail! > >>>>> * > >>>>> *The test consists of allocating a buffer so ridiculous large that > >>>>> it should never succeed and be rejected by the kernel driver. > >>>>> > >>>>> This patch here is a really clear NAK. > >>>>> > >>>>> Regards, > >>>>> Christian. > >>>>> > >>>>>> it may be a kvmalloc() bug, and this patch can as a workaround > >>>>>> in ttm before this issue fix. > >>>>>> > >>>>>> void *kvmalloc_node(size_t size, gfp_t flags, int node) > >>>>>> { > >>>>>> ... > >>>>>>       if ((flags & GFP_KERNEL) != GFP_KERNEL) > >>>>>>               return kmalloc_node(size, flags, node); > >>>>>> ... > >>>>>> } > >>>>>> > >>>>>> Best Regards, > >>>>>> Kevin > >>>>>> > >>>>>> > to allocate memory, when request size is exceeds kmalloc limit, > >>>>>> it will > >>>>>> > cause allocate memory fail. > >>>>>> > > >>>>>> > e.g: when ttm want to create a BO with 1TB size, it maybe fail. > >>>>>> > > >>>>>> > Signed-off-by: Yang Wang > >>>>>> > > >>>>>> > --- > >>>>>> > drivers/gpu/drm/ttm/ttm_tt.c | 14 +++++++++++--- > >>>>>> >   1 file changed, 11 insertions(+), 3 deletions(-) > >>>>>> > > >>>>>> > diff --git a/drivers/gpu/drm/ttm/ttm_tt.c > >>>>>> b/drivers/gpu/drm/ttm/ttm_tt.c > >>>>>> > index 79c870a3bef8..9f2f3e576b8d 100644 > >>>>>> > --- a/drivers/gpu/drm/ttm/ttm_tt.c > >>>>>> > +++ b/drivers/gpu/drm/ttm/ttm_tt.c > >>>>>> > @@ -97,9 +97,12 @@ int ttm_tt_create(struct ttm_buffer_object > >>>>>> *bo, bool zero_alloc) > >>>>>> >   static int ttm_tt_alloc_page_directory(struct ttm_tt *ttm) > >>>>>> >   { > >>>>>> >        ttm->pages = kvmalloc_array(ttm->num_pages, sizeof(void*), > >>>>>> > - GFP_KERNEL | __GFP_ZERO); > >>>>>> > + GFP_KERNEL); > >>>>>> >        if (!ttm->pages) > >>>>>> >                return -ENOMEM; > >>>>>> > + > >>>>>> > + memset(ttm->pages, 0, ttm->num_pages * sizeof(void *)); > >>>>>> > + > >>>>>> >        return 0; > >>>>>> >   } > >>>>>> > > >>>>>> > @@ -108,10 +111,12 @@ static int > >>>>>> ttm_dma_tt_alloc_page_directory(struct ttm_tt *ttm) > >>>>>> >        ttm->pages = kvmalloc_array(ttm->num_pages, > >>>>>> > sizeof(*ttm->pages) + > >>>>>> > sizeof(*ttm->dma_address), > >>>>>> > - GFP_KERNEL | __GFP_ZERO); > >>>>>> > + GFP_KERNEL); > >>>>>> >        if (!ttm->pages) > >>>>>> >                return -ENOMEM; > >>>>>> > > >>>>>> > + memset(ttm->pages, 0, ttm->num_pages * > >>>>>> (sizeof(*ttm->pages) + sizeof(*ttm->dma_address))); > >>>>>> > + > >>>>>> >        ttm->dma_address = (void *)(ttm->pages + ttm->num_pages); > >>>>>> >        return 0; > >>>>>> >   } > >>>>>> > @@ -120,9 +125,12 @@ static int > >>>>>> ttm_sg_tt_alloc_page_directory(struct ttm_tt *ttm) > >>>>>> >   { > >>>>>> >        ttm->dma_address = kvmalloc_array(ttm->num_pages, > >>>>>> > sizeof(*ttm->dma_address), > >>>>>> > - GFP_KERNEL | __GFP_ZERO); > >>>>>> > + GFP_KERNEL); > >>>>>> >        if (!ttm->dma_address) > >>>>>> >                return -ENOMEM; > >>>>>> > + > >>>>>> > + memset(ttm->dma_address, 0, ttm->num_pages * > >>>>>> sizeof(*ttm->dma_address)); > >>>>>> > + > >>>>>> >        return 0; > >>>>>> >   } > >>>>>> > > >>>>>> > >>>>> > >>>> > >>> > >> --------------vdznHasFmNx4e2RAlAoQXvZx Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit Am 21.04.22 um 04:15 schrieb Wang, Yang(Kevin):

[AMD Official Use Only]





From: Kuehling, Felix <Felix.Kuehling@amd.com>
Sent: Thursday, April 21, 2022 5:21 AM
To: Lazar, Lijo <Lijo.Lazar@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>; Wang, Yang(Kevin) <KevinYang.Wang@amd.com>; Christian König <ckoenig.leichtzumerken@gmail.com>; dri-devel@lists.freedesktop.org <dri-devel@lists.freedesktop.org>; amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>
Subject: Re: [PATCH] drm/ttm: fix ttm tt init fail when size exceeds kmalloc limit
 

On 2022-04-20 09:23, Lazar, Lijo wrote:
>
>
> On 4/20/2022 6:26 PM, Christian König wrote:
>> Am 20.04.22 um 14:54 schrieb Wang, Yang(Kevin):
>>>
>>> [AMD Official Use Only]
>>>
>>>
>>> Hi Chris,
>>>
>>> 1) Change the test case to use something larger than 1TiB.
>>> sure, we can increase the size of BO and make test pass,
>>> but if user really want to allocate 1TB GTT BO, we have no reason to
>>> let it fail? right?
>>
>> No, the reason is the underlying core kernel doesn't allow kvmalloc
>> allocations with GFP_ZERO which are large enough to hold the array of
>> allocated pages for this.
>>
>> We are working on top of the core Linux kernel and should *NEVER*
>> ever add workarounds like what was suggested here. >
>
> AFAIU, for the purpose of ttm use, fallback to vmalloc is fine.
>
>  * Please note that any use of gfp flags outside of GFP_KERNEL is
> careful to not
>  * fall back to vmalloc.
>  *

That's weird, because kvcalloc does the same thing. If that were not
able to fall back to vmalloc, it would be pretty useless.

    static inline __alloc_size(1, 2) void *kvcalloc(size_t n, size_t size, gfp_t flags)
    {
             return kvmalloc_array(n, size, flags | __GFP_ZERO);
    }

Maybe kvcalloc is the function we TTM should be using here anyway,
instead of open-coding the kvmalloc_array call with an extra GFP flag.

Regards,
   Felix

Yes, I agree with your point, and in amdkfd driver code, we have the same risk in svm_range_dma_map_dev().

Yes, sounds like a good idea to me as well to change that.

Regards,
Christian.


Best Regards,
Kevin

>
> Actually the current implementation documents the behavior, but it is
> deep inside the implementation to be noticeable - at least not obvious
> while using kvmalloc_array.
>
> Thanks,
> Lijo
>
>> Regards,
>> Christian.
>>
>>> the system availed memory about 2T, but it will still fail.
>>>
>>> 2) Change kvmalloc to allow GFP_ZERO allocations even in the vmalloc
>>> fallback path.
>>>     the 5.18 kernel will add this patch to fix this issue .
>>>
>>> Best Regards,
>>> Kevin
>>> ------------------------------------------------------------------------
>>>
>>> *From:* Koenig, Christian <Christian.Koenig@amd.com>
>>> *Sent:* Wednesday, April 20, 2022 8:42 PM
>>> *To:* Wang, Yang(Kevin) <KevinYang.Wang@amd.com>; Christian König
>>> <ckoenig.leichtzumerken@gmail.com>; dri-devel@lists.freedesktop.org
>>> <dri-devel@lists.freedesktop.org>; amd-gfx@lists.freedesktop.org
>>> <amd-gfx@lists.freedesktop.org>
>>> *Subject:* Re: [PATCH] drm/ttm: fix ttm tt init fail when size
>>> exceeds kmalloc limit
>>> Hi Kevin,
>>>
>>> yes and that is perfectly valid and expected behavior. There is
>>> absolutely no need to change anything in TTM here.
>>>
>>> What we could do is:
>>> 1) Change the test case to use something larger than 1TiB.
>>> 2) Change kvmalloc to allow GFP_ZERO allocations even in the vmalloc
>>> fallback path.
>>>
>>> Regards,
>>> Christian.
>>>
>>> Am 20.04.22 um 14:39 schrieb Wang, Yang(Kevin):
>>>>
>>>> [AMD Official Use Only]
>>>>
>>>>
>>>> Hi Chirs,
>>>>
>>>> yes, right, the amdgpu drive rwill use amdgpu_bo_validate_size()
>>>> function to verify bo size,
>>>> but when driver try to allocate VRAM domain bo fail, the amdgpu
>>>> driver will fall back to allocate domain = (GTT | VRAM)  bo.
>>>> please check following code, it will cause the 2nd time to allocate
>>>> bo fail during allocate 256Mb buffer to store dma address (via
>>>> kvmalloc()).
>>>>
>>>> initial_domain = (u32)(0xffffffff & args->in.domains);
>>>> retry:
>>>>         r = amdgpu_gem_object_create(adev, size, args->in.alignment,
>>>>                    initial_domain,
>>>>                    flags, ttm_bo_type_device, resv, &gobj);
>>>>         if (r && r != -ERESTARTSYS) {
>>>>                 if (flags & AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED) {
>>>>       flags &= ~AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED;
>>>>       goto retry;
>>>>                 }
>>>>
>>>>                 if (initial_domain == AMDGPU_GEM_DOMAIN_VRAM) {
>>>>       initial_domain |= AMDGPU_GEM_DOMAIN_GTT;
>>>>       goto retry;
>>>>                 }
>>>> DRM_DEBUG("Failed to allocate GEM object (%llu, %d, %llu, %d)\n",
>>>>               size, initial_domain, args->in.alignment, r);
>>>>         }
>>>>
>>>> Best Regards,
>>>> Kevin
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> *From:* Christian König <ckoenig.leichtzumerken@gmail.com>
>>>> <mailto:ckoenig.leichtzumerken@gmail.com>
>>>> *Sent:* Wednesday, April 20, 2022 7:55 PM
>>>> *To:* Wang, Yang(Kevin) <KevinYang.Wang@amd.com>
>>>> <mailto:KevinYang.Wang@amd.com>; Koenig, Christian
>>>> <Christian.Koenig@amd.com> <mailto:Christian.Koenig@amd.com>;
>>>> dri-devel@lists.freedesktop.org
>>>> <mailto:dri-devel@lists.freedesktop.org>
>>>> <dri-devel@lists.freedesktop.org>
>>>> <mailto:dri-devel@lists.freedesktop.org>;
>>>> amd-gfx@lists.freedesktop.org
>>>> <mailto:amd-gfx@lists.freedesktop.org>
>>>> <amd-gfx@lists.freedesktop.org> <mailto:amd-gfx@lists.freedesktop.org>
>>>> *Subject:* Re: [PATCH] drm/ttm: fix ttm tt init fail when size
>>>> exceeds kmalloc limit
>>>> Hi Kevin,
>>>>
>>>> no, the test case should already fail in amdgpu_bo_validate_size().
>>>>
>>>> If we have a system with 2TiB of memory where the test case could
>>>> succeed then we should increase the requested size to something
>>>> larger.
>>>>
>>>> And if the underlying core Linux kernel functions don't allow
>>>> allocations as large as the requested one we should *NEVER* ever
>>>> add workarounds like this.
>>>>
>>>> It is perfectly expected that this test case is not able to fulfill
>>>> the desired allocation. That it fails during kvmalloc is
>>>> unfortunate, but not a show stopper.
>>>>
>>>> Regards,
>>>> Christian.
>>>>
>>>>
>>>> Am 20.04.22 um 13:32 schrieb Wang, Yang(Kevin):
>>>>>
>>>>> [AMD Official Use Only]
>>>>>
>>>>>
>>>>> Hi Chris,
>>>>>
>>>>> you misunderstood background about this case.
>>>>>
>>>>> although we expect this test case to fail, it should fail at the
>>>>> location where the Bo actual memory is actually allocated. now the
>>>>> code logic will cause the failure to allocate memory to store DMA
>>>>> address.
>>>>>
>>>>> e.g: the case is failed in 2TB system ram machine, it should be
>>>>> allocated successful, but it is failed.
>>>>>
>>>>> allocate 1TB BO, the ttm should allocate 1TB/4k * 8 buffer to
>>>>> store allocate result (page address), this should not be failed
>>>>> usually.
>>>>>
>>>>> There is a similar fix in upstream kernel 5.18, before this fix
>>>>> entered the TTM code, this problem existed in TTM.
>>>>>
>>>>> kernel/git/torvalds/linux.git - Linux kernel source tree
>>>>> <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.18-rc3&id=a421ef303008b0ceee2cfc625c3246fa7654b0ca
>>>>> mm: allow !GFP_KERNEL allocations for kvmalloc
>>>>>
>>>>> Best Regards,
>>>>> Kevin
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> *From:* Koenig, Christian <Christian.Koenig@amd.com>
>>>>> <mailto:Christian.Koenig@amd.com>
>>>>> *Sent:* Wednesday, April 20, 2022 6:53 PM
>>>>> *To:* Wang, Yang(Kevin) <KevinYang.Wang@amd.com>
>>>>> <mailto:KevinYang.Wang@amd.com>; dri-devel@lists.freedesktop.org
>>>>> <mailto:dri-devel@lists.freedesktop.org>
>>>>> <dri-devel@lists.freedesktop.org>
>>>>> <mailto:dri-devel@lists.freedesktop.org>;
>>>>> amd-gfx@lists.freedesktop.org
>>>>> <mailto:amd-gfx@lists.freedesktop.org>
>>>>> <amd-gfx@lists.freedesktop.org>
>>>>> <mailto:amd-gfx@lists.freedesktop.org>
>>>>> *Subject:* Re: [PATCH] drm/ttm: fix ttm tt init fail when size
>>>>> exceeds kmalloc limit
>>>>> Am 20.04.22 um 11:07 schrieb Wang, Yang(Kevin):
>>>>>>
>>>>>> [AMD Official Use Only]
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>> *From:* Koenig, Christian <Christian.Koenig@amd.com>
>>>>>> <mailto:Christian.Koenig@amd.com>
>>>>>> *Sent:* Wednesday, April 20, 2022 5:00 PM
>>>>>> *To:* Wang, Yang(Kevin) <KevinYang.Wang@amd.com>
>>>>>> <mailto:KevinYang.Wang@amd.com>; dri-devel@lists.freedesktop.org
>>>>>> <mailto:dri-devel@lists.freedesktop.org>
>>>>>> <dri-devel@lists.freedesktop.org>
>>>>>> <mailto:dri-devel@lists.freedesktop.org>;
>>>>>> amd-gfx@lists.freedesktop.org
>>>>>> <mailto:amd-gfx@lists.freedesktop.org>
>>>>>> <amd-gfx@lists.freedesktop.org>
>>>>>> <mailto:amd-gfx@lists.freedesktop.org>
>>>>>> *Subject:* Re: [PATCH] drm/ttm: fix ttm tt init fail when size
>>>>>> exceeds kmalloc limit
>>>>>> Am 20.04.22 um 10:56 schrieb Yang Wang:
>>>>>> > if the __GFP_ZERO is set, the kvmalloc() can't fallback to use
>>>>>> vmalloc()
>>>>>>
>>>>>> Hui what? Why should kvmalloc() not be able to fallback to vmalloc()
>>>>>> when __GFP_ZERO is set?
>>>>>>
>>>>>> And even that is really the case then that sounds like a bug in
>>>>>> kvmalloc().
>>>>>>
>>>>>> Regards,
>>>>>> Christian.
>>>>>>
>>>>>> [kevin]:
>>>>>> it is really test case from libdrm amdgpu test, which try to
>>>>>> allocate a big BO which will cause ttm tt init fail.
>>>>>
>>>>>
>>>>> LOL! Guys, this test case is intended to fail!
>>>>> *
>>>>> *The test consists of allocating a buffer so ridiculous large that
>>>>> it should never succeed and be rejected by the kernel driver.
>>>>>
>>>>> This patch here is a really clear NAK.
>>>>>
>>>>> Regards,
>>>>> Christian.
>>>>>
>>>>>> it may be a kvmalloc() bug, and this patch can as a workaround
>>>>>> in ttm before this issue fix.
>>>>>>
>>>>>> void *kvmalloc_node(size_t size, gfp_t flags, int node)
>>>>>> {
>>>>>> ...
>>>>>>       if ((flags & GFP_KERNEL) != GFP_KERNEL)
>>>>>>               return kmalloc_node(size, flags, node);
>>>>>> ...
>>>>>> }
>>>>>>
>>>>>> Best Regards,
>>>>>> Kevin
>>>>>>
>>>>>> > to allocate memory, when request size is exceeds kmalloc limit,
>>>>>> it will
>>>>>> > cause allocate memory fail.
>>>>>> >
>>>>>> > e.g: when ttm want to create a BO with 1TB size, it maybe fail.
>>>>>> >
>>>>>> > Signed-off-by: Yang Wang <KevinYang.Wang@amd.com>
>>>>>> <mailto:KevinYang.Wang@amd.com>
>>>>>> > ---
>>>>>> >   drivers/gpu/drm/ttm/ttm_tt.c | 14 +++++++++++---
>>>>>> >   1 file changed, 11 insertions(+), 3 deletions(-)
>>>>>> >
>>>>>> > diff --git a/drivers/gpu/drm/ttm/ttm_tt.c
>>>>>> b/drivers/gpu/drm/ttm/ttm_tt.c
>>>>>> > index 79c870a3bef8..9f2f3e576b8d 100644
>>>>>> > --- a/drivers/gpu/drm/ttm/ttm_tt.c
>>>>>> > +++ b/drivers/gpu/drm/ttm/ttm_tt.c
>>>>>> > @@ -97,9 +97,12 @@ int ttm_tt_create(struct ttm_buffer_object
>>>>>> *bo, bool zero_alloc)
>>>>>> >   static int ttm_tt_alloc_page_directory(struct ttm_tt *ttm)
>>>>>> >   {
>>>>>> >        ttm->pages = kvmalloc_array(ttm->num_pages, sizeof(void*),
>>>>>> > -                     GFP_KERNEL | __GFP_ZERO);
>>>>>> > + GFP_KERNEL);
>>>>>> >        if (!ttm->pages)
>>>>>> >                return -ENOMEM;
>>>>>> > +
>>>>>> > +     memset(ttm->pages, 0, ttm->num_pages * sizeof(void *));
>>>>>> > +
>>>>>> >        return 0;
>>>>>> >   }
>>>>>> >
>>>>>> > @@ -108,10 +111,12 @@ static int
>>>>>> ttm_dma_tt_alloc_page_directory(struct ttm_tt *ttm)
>>>>>> >        ttm->pages = kvmalloc_array(ttm->num_pages,
>>>>>> > sizeof(*ttm->pages) +
>>>>>> > sizeof(*ttm->dma_address),
>>>>>> > - GFP_KERNEL | __GFP_ZERO);
>>>>>> > + GFP_KERNEL);
>>>>>> >        if (!ttm->pages)
>>>>>> >                return -ENOMEM;
>>>>>> >
>>>>>> > +     memset(ttm->pages, 0, ttm->num_pages *
>>>>>> (sizeof(*ttm->pages) + sizeof(*ttm->dma_address)));
>>>>>> > +
>>>>>> >        ttm->dma_address = (void *)(ttm->pages + ttm->num_pages);
>>>>>> >        return 0;
>>>>>> >   }
>>>>>> > @@ -120,9 +125,12 @@ static int
>>>>>> ttm_sg_tt_alloc_page_directory(struct ttm_tt *ttm)
>>>>>> >   {
>>>>>> >        ttm->dma_address = kvmalloc_array(ttm->num_pages,
>>>>>> > sizeof(*ttm->dma_address),
>>>>>> > - GFP_KERNEL | __GFP_ZERO);
>>>>>> > + GFP_KERNEL);
>>>>>> >        if (!ttm->dma_address)
>>>>>> >                return -ENOMEM;
>>>>>> > +
>>>>>> > +     memset(ttm->dma_address, 0, ttm->num_pages *
>>>>>> sizeof(*ttm->dma_address));
>>>>>> > +
>>>>>> >        return 0;
>>>>>> >   }
>>>>>> >
>>>>>>
>>>>>
>>>>
>>>
>>

--------------vdznHasFmNx4e2RAlAoQXvZx--