Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,INCLUDES_PATCH,MAILING_LIST_MULTI, MIME_HTML_MOSTLY,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F637C2BA19 for ; Tue, 14 Apr 2020 14:46:53 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1FBC42085B for ; Tue, 14 Apr 2020 14:46:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="3+Gc+4Ju" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1FBC42085B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=amd-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C8AF06E143; Tue, 14 Apr 2020 14:46:52 +0000 (UTC) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2061e.outbound.protection.outlook.com [IPv6:2a01:111:f400:7eaa::61e]) by gabe.freedesktop.org (Postfix) with ESMTPS id DE9116E143 for ; Tue, 14 Apr 2020 14:46:50 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=m4Mzk7xomOM6av3mmrsx5t9YHQ4XvedyB//+jt6hI6DhCZAQKkAm2yl7XuuNnj2sjUFC9esdKG2SubbFoHkWh1xUTk7lIF4PDEjdQXWxWJWMvz8NVMSCrkxUujQo+6wvkWUPu+n39ujyFLHvo/9mKREWaML+hzHd+fNntva1Mz/15xEghmrTQO7g6rk8MbexFpqHsHSe/JuWNyb+kk9+2drMKbwFYHXmrizRsMXuAjzTscBYbkd7D0sO/n3PZLgKOByMGQRz6c+uZU1VWgAUvhh/JgKTZtX6tAJwSxGz29fbzaEIQbrfktDBGXhtuyEzm+iO2PxSzLcAE8+KZjlhAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JAkgVB5uzxANfcHpr+FwlPEoETqt1f857nI7JeCkYI8=; b=OYiE9DLdUmp56x6GGW1NeLuNHjfyZfqO7av0hZsv1FuXEWDsM3wB9yQEakOBaqHFpFqPErZfpW7O7nfE42XK5Dl4SRNL4RQYJofQJVZALyXpHvtSexWBM0qT4Zz2bQe7LsE05HTWb9urcumtIXFwDzwQPRuqglduWgmZPLb2aogzuMNW5wVKdHj2o1jqRrc9MdLHAKQZSLPk+skM+eF2Bi2UpYo3V9qfcNr/A0w3WDBkU4qoBf5b7c3mcwqJwgyDuFREXPhNkvD7/JP/WjbrZ8lwaFifyC3uoPfflqwD/le3P9uKXtPlMwhd9LPyaj8Ywl0iPcQI56fZh09fpSD+4g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JAkgVB5uzxANfcHpr+FwlPEoETqt1f857nI7JeCkYI8=; b=3+Gc+4JuWMtUMLRA40BWYsBiL1FiEe8XM/szaOBySbPf1qC85r+J6GKCfkTUhZVuCrF/DdYkg9sTRcB5CIS5qZEkEV+IjiNosDqVqi3G3VnuiA6ucmhWt9ZJJS+c6SzT2ATxmebhwR7kNauXqjZY6kZHO6r/dcv2mgJu09RUkwY= Received: from DM6PR12MB4401.namprd12.prod.outlook.com (2603:10b6:5:2a9::15) by DM6PR12MB4185.namprd12.prod.outlook.com (2603:10b6:5:216::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2900.26; Tue, 14 Apr 2020 14:46:49 +0000 Received: from DM6PR12MB4401.namprd12.prod.outlook.com ([fe80::f164:85c4:1b51:14d2]) by DM6PR12MB4401.namprd12.prod.outlook.com ([fe80::f164:85c4:1b51:14d2%4]) with mapi id 15.20.2900.028; Tue, 14 Apr 2020 14:46:49 +0000 From: "Koenig, Christian" To: "Deucher, Alexander" Subject: Re: [PATCH] Revert "drm/amdgpu: use the BAR if possible in amdgpu_device_vram_access v2" Thread-Topic: [PATCH] Revert "drm/amdgpu: use the BAR if possible in amdgpu_device_vram_access v2" Thread-Index: AQHWEcA5+D7UIeTa4E6FAfQP6w9qZah4pBqAgAAHWCCAAAUBgIAAAxkX Date: Tue, 14 Apr 2020 14:46:48 +0000 Message-ID: <146d9570-724e-423d-931e-24c96821aaae@email.android.com> References: <20200413182026.2561-1-kent.russell@amd.com> <85fcb568-b0d8-b6c9-4e62-3866aa2da0c9@gmail.com>, , In-Reply-To: Accept-Language: de-DE, en-US Content-Language: de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Christian.Koenig@amd.com; x-originating-ip: [2a02:908:1252:fb60:a4a4:79aa:f763:24b7] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 912ef90b-1bed-4225-a94d-08d7e082a981 x-ms-traffictypediagnostic: DM6PR12MB4185:|DM6PR12MB4185: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2657; x-forefront-prvs: 0373D94D15 x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR12MB4401.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(10009020)(4636009)(346002)(136003)(39860400002)(376002)(396003)(366004)(6486002)(2906002)(478600001)(66946007)(45080400002)(76116006)(91956017)(6512007)(66446008)(64756008)(66556008)(66476007)(9686003)(316002)(54906003)(53546011)(966005)(6506007)(8936002)(186003)(31696002)(19627405001)(81156014)(8676002)(4326008)(30864003)(71200400001)(5660300002)(66574012)(6862004)(6636002)(86362001)(31686004)(579004); DIR:OUT; SFP:1101; received-spf: None (protection.outlook.com: amd.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: fjIgQAQ2mZ/fc31uwKM5GdgengIOebU2Yy9HkbWmRWtsl0yWc9nL4dpvzhEKoPwabX8WxVBvl66SjPrzHy5n74hvsRz2+X88wlCbHuA8GyJ0y6Pc2otnWhb/cGacQVoqBN7PaheXVEF+zucsOI1fDPLGgRgvbqkbJD4N8vPP4+PyFLOiLZu9qk0kH+rSbbcaSj8zIPJg6Sa/sBL+3kty5GVgYwLg1geAhfKIaXgiAayG57xS8oxVVSp2uepJXRsMe2ynwr30h1meiB17XafOZi3F5QipvjFds8oA0hKuX5YvjI0Pv9XUjKp55zrx/RqOERB+d3/ms3XLlrZXqVTVw+wiVdxnmnHpKZVN17Wk1Z4JrgkxHpNkaDip5G65UyfNNiemCVhID4M71pFkjNi+ENMmQi++7t5a9EUXepZmw9SnAyi+80VP3C5GHKA5nUfXiIcFDuH9edjp4/1gHNtqtlSyqq5dx1fHM9g5lfpgFoo= x-ms-exchange-antispam-messagedata: X872YYUZfKzUmCzunVktHLuC9J+CVSJn86zlingPSU90/L4Wc2ObH1c6ECkEARCIFq6jqhoXJ5RwFaxkZQXDrTF66VNQJ0FsYNh+RO+zDUGEJjBWGUCoSAvj8CFZ/lBFufZ51+eK99BMHZrnnFXNBRnDmfnmNRUR+dU92M3V8kzc/UvvfWeQd0wopDyzV6o06cuPLvcYL62k2cbKC113Dw== MIME-Version: 1.0 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 912ef90b-1bed-4225-a94d-08d7e082a981 X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Apr 2020 14:46:48.9585 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: weJhyVy4kbARAO5KBL55Gna/E/BhBM0El/R7E9qQkSnYKt9q196sRfuS02unksRh X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4185 X-BeenThere: amd-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion list for AMD gfx List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Kuehling, Felix" , "Kim, Jonathan" , "Russell, Kent" , "amd-gfx@lists.freedesktop.org" Content-Type: multipart/mixed; boundary="===============0738367152==" Errors-To: amd-gfx-bounces@lists.freedesktop.org Sender: "amd-gfx" --===============0738367152== Content-Language: de-DE Content-Type: multipart/alternative; boundary="_000_146d9570724e423d931e24c96821aaaeemailandroidcom_" --_000_146d9570724e423d931e24c96821aaaeemailandroidcom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable That's exactly my concern as well. This looks a bit like the test creates erroneous data somehow, but there do= esn't seems to be a RAS check in the MM data path. And now that we use the BAR path it goes up in flames. I just don't see how we can create erroneous data in a test case? Christian. Am 14.04.2020 16:35 schrieb "Deucher, Alexander" : [AMD Public Use] If this causes an issue, any access to vram via the BAR could cause an issu= e. Alex ________________________________ From: amd-gfx on behalf of Russell,= Kent Sent: Tuesday, April 14, 2020 10:19 AM To: Koenig, Christian ; amd-gfx@lists.freedesktop= .org Cc: Kuehling, Felix ; Kim, Jonathan Subject: RE: [PATCH] Revert "drm/amdgpu: use the BAR if possible in amdgpu_= device_vram_access v2" [AMD Official Use Only - Internal Distribution Only] On VG20 or MI100, as soon as we run the subtest, we get the dmesg output be= low, and then the kernel ends up hanging. I don't know enough about the tes= t itself to know why this is occurring, but Jon Kim and Felix were discussi= ng it on a separate thread when the issue was first reported, so they can h= opefully provide some additional information. Kent > -----Original Message----- > From: Christian K=F6nig > Sent: Tuesday, April 14, 2020 9:52 AM > To: Russell, Kent ; amd-gfx@lists.freedesktop.org > Subject: Re: [PATCH] Revert "drm/amdgpu: use the BAR if possible in > amdgpu_device_vram_access v2" > > Am 13.04.20 um 20:20 schrieb Kent Russell: > > This reverts commit c12b84d6e0d70f1185e6daddfd12afb671791b6e. > > The original patch causes a RAS event and subsequent kernel hard-hang > > when running the KFDMemoryTest.PtraceAccessInvisibleVram on VG20 and > > Arcturus > > > > dmesg output at hang time: > > [drm] RAS event of type ERREVENT_ATHUB_INTERRUPT detected! > > amdgpu 0000:67:00.0: GPU reset begin! > > Evicting PASID 0x8000 queues > > Started evicting pasid 0x8000 > > qcm fence wait loop timeout expired > > The cp might be in an unrecoverable state due to an unsuccessful > > queues preemption Failed to evict process queues Failed to suspend > > process 0x8000 Finished evicting pasid 0x8000 Started restoring pasid > > 0x8000 Finished restoring pasid 0x8000 [drm] UVD VCPU state may lost > > due to RAS ERREVENT_ATHUB_INTERRUPT > > amdgpu: [powerplay] Failed to send message 0x26, response 0x0 > > amdgpu: [powerplay] Failed to set soft min gfxclk ! > > amdgpu: [powerplay] Failed to upload DPM Bootup Levels! > > amdgpu: [powerplay] Failed to send message 0x7, response 0x0 > > amdgpu: [powerplay] [DisableAllSMUFeatures] Failed to disable all smu > features! > > amdgpu: [powerplay] [DisableDpmTasks] Failed to disable all smu feature= s! > > amdgpu: [powerplay] [PowerOffAsic] Failed to disable DPM! > > [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of IP > > block failed -5 > > Do you have more information on what's going wrong here since this is a r= eally > important patch for KFD debugging. > > > > > Signed-off-by: Kent Russell > > Reviewed-by: Christian K=F6nig > > > --- > > drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 26 ---------------------= - > > 1 file changed, 26 deletions(-) > > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > index cf5d6e585634..a3f997f84020 100644 > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > @@ -254,32 +254,6 @@ void amdgpu_device_vram_access(struct > amdgpu_device *adev, loff_t pos, > > uint32_t hi =3D ~0; > > uint64_t last; > > > > - > > -#ifdef CONFIG_64BIT > > - last =3D min(pos + size, adev->gmc.visible_vram_size); > > - if (last > pos) { > > - void __iomem *addr =3D adev->mman.aper_base_kaddr + pos; > > - size_t count =3D last - pos; > > - > > - if (write) { > > - memcpy_toio(addr, buf, count); > > - mb(); > > - amdgpu_asic_flush_hdp(adev, NULL); > > - } else { > > - amdgpu_asic_invalidate_hdp(adev, NULL); > > - mb(); > > - memcpy_fromio(buf, addr, count); > > - } > > - > > - if (count =3D=3D size) > > - return; > > - > > - pos +=3D count; > > - buf +=3D count / 4; > > - size -=3D count; > > - } > > -#endif > > - > > spin_lock_irqsave(&adev->mmio_idx_lock, flags); > > for (last =3D pos + size; pos < last; pos +=3D 4) { > > uint32_t tmp =3D pos >> 31; _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flists.f= reedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=3D02%7C01%7Calexande= r.deucher%40amd.com%7C68e0bfea2a5f4a909ab108d7e07ed164%7C3dd8961fe4884e608e= 11a82d994e183d%7C0%7C0%7C637224707637289768&sdata=3DttNOHJt0IwywpOIWahK= jjuC6OkT1jxduc6iMzYzndpg%3D&reserved=3D0 Am 14.04.2020 16:35 schrieb "Deucher, Alexander" : [AMD Public Use] If this causes an issue, any access to vram via the BAR could cause an issu= e. Alex ________________________________ From: amd-gfx on behalf of Russell,= Kent Sent: Tuesday, April 14, 2020 10:19 AM To: Koenig, Christian ; amd-gfx@lists.freedesktop= .org Cc: Kuehling, Felix ; Kim, Jonathan Subject: RE: [PATCH] Revert "drm/amdgpu: use the BAR if possible in amdgpu_= device_vram_access v2" [AMD Official Use Only - Internal Distribution Only] On VG20 or MI100, as soon as we run the subtest, we get the dmesg output be= low, and then the kernel ends up hanging. I don't know enough about the tes= t itself to know why this is occurring, but Jon Kim and Felix were discussi= ng it on a separate thread when the issue was first reported, so they can h= opefully provide some additional information. Kent > -----Original Message----- > From: Christian K=F6nig > Sent: Tuesday, April 14, 2020 9:52 AM > To: Russell, Kent ; amd-gfx@lists.freedesktop.org > Subject: Re: [PATCH] Revert "drm/amdgpu: use the BAR if possible in > amdgpu_device_vram_access v2" > > Am 13.04.20 um 20:20 schrieb Kent Russell: > > This reverts commit c12b84d6e0d70f1185e6daddfd12afb671791b6e. > > The original patch causes a RAS event and subsequent kernel hard-hang > > when running the KFDMemoryTest.PtraceAccessInvisibleVram on VG20 and > > Arcturus > > > > dmesg output at hang time: > > [drm] RAS event of type ERREVENT_ATHUB_INTERRUPT detected! > > amdgpu 0000:67:00.0: GPU reset begin! > > Evicting PASID 0x8000 queues > > Started evicting pasid 0x8000 > > qcm fence wait loop timeout expired > > The cp might be in an unrecoverable state due to an unsuccessful > > queues preemption Failed to evict process queues Failed to suspend > > process 0x8000 Finished evicting pasid 0x8000 Started restoring pasid > > 0x8000 Finished restoring pasid 0x8000 [drm] UVD VCPU state may lost > > due to RAS ERREVENT_ATHUB_INTERRUPT > > amdgpu: [powerplay] Failed to send message 0x26, response 0x0 > > amdgpu: [powerplay] Failed to set soft min gfxclk ! > > amdgpu: [powerplay] Failed to upload DPM Bootup Levels! > > amdgpu: [powerplay] Failed to send message 0x7, response 0x0 > > amdgpu: [powerplay] [DisableAllSMUFeatures] Failed to disable all smu > features! > > amdgpu: [powerplay] [DisableDpmTasks] Failed to disable all smu feature= s! > > amdgpu: [powerplay] [PowerOffAsic] Failed to disable DPM! > > [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of IP > > block failed -5 > > Do you have more information on what's going wrong here since this is a r= eally > important patch for KFD debugging. > > > > > Signed-off-by: Kent Russell > > Reviewed-by: Christian K=F6nig > > > --- > > drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 26 ---------------------= - > > 1 file changed, 26 deletions(-) > > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > index cf5d6e585634..a3f997f84020 100644 > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > @@ -254,32 +254,6 @@ void amdgpu_device_vram_access(struct > amdgpu_device *adev, loff_t pos, > > uint32_t hi =3D ~0; > > uint64_t last; > > > > - > > -#ifdef CONFIG_64BIT > > - last =3D min(pos + size, adev->gmc.visible_vram_size); > > - if (last > pos) { > > - void __iomem *addr =3D adev->mman.aper_base_kaddr + pos; > > - size_t count =3D last - pos; > > - > > - if (write) { > > - memcpy_toio(addr, buf, count); > > - mb(); > > - amdgpu_asic_flush_hdp(adev, NULL); > > - } else { > > - amdgpu_asic_invalidate_hdp(adev, NULL); > > - mb(); > > - memcpy_fromio(buf, addr, count); > > - } > > - > > - if (count =3D=3D size) > > - return; > > - > > - pos +=3D count; > > - buf +=3D count / 4; > > - size -=3D count; > > - } > > -#endif > > - > > spin_lock_irqsave(&adev->mmio_idx_lock, flags); > > for (last =3D pos + size; pos < last; pos +=3D 4) { > > uint32_t tmp =3D pos >> 31; _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flists.f= reedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=3D02%7C01%7Calexande= r.deucher%40amd.com%7C68e0bfea2a5f4a909ab108d7e07ed164%7C3dd8961fe4884e608e= 11a82d994e183d%7C0%7C0%7C637224707637289768&sdata=3DttNOHJt0IwywpOIWahK= jjuC6OkT1jxduc6iMzYzndpg%3D&reserved=3D0 Am 14.04.2020 16:35 schrieb "Deucher, Alexander" : [AMD Public Use] If this causes an issue, any access to vram via the BAR could cause an issu= e. Alex ________________________________ From: amd-gfx on behalf of Russell,= Kent Sent: Tuesday, April 14, 2020 10:19 AM To: Koenig, Christian ; amd-gfx@lists.freedesktop= .org Cc: Kuehling, Felix ; Kim, Jonathan Subject: RE: [PATCH] Revert "drm/amdgpu: use the BAR if possible in amdgpu_= device_vram_access v2" [AMD Official Use Only - Internal Distribution Only] On VG20 or MI100, as soon as we run the subtest, we get the dmesg output be= low, and then the kernel ends up hanging. I don't know enough about the tes= t itself to know why this is occurring, but Jon Kim and Felix were discussi= ng it on a separate thread when the issue was first reported, so they can h= opefully provide some additional information. Kent > -----Original Message----- > From: Christian K=F6nig > Sent: Tuesday, April 14, 2020 9:52 AM > To: Russell, Kent ; amd-gfx@lists.freedesktop.org > Subject: Re: [PATCH] Revert "drm/amdgpu: use the BAR if possible in > amdgpu_device_vram_access v2" > > Am 13.04.20 um 20:20 schrieb Kent Russell: > > This reverts commit c12b84d6e0d70f1185e6daddfd12afb671791b6e. > > The original patch causes a RAS event and subsequent kernel hard-hang > > when running the KFDMemoryTest.PtraceAccessInvisibleVram on VG20 and > > Arcturus > > > > dmesg output at hang time: > > [drm] RAS event of type ERREVENT_ATHUB_INTERRUPT detected! > > amdgpu 0000:67:00.0: GPU reset begin! > > Evicting PASID 0x8000 queues > > Started evicting pasid 0x8000 > > qcm fence wait loop timeout expired > > The cp might be in an unrecoverable state due to an unsuccessful > > queues preemption Failed to evict process queues Failed to suspend > > process 0x8000 Finished evicting pasid 0x8000 Started restoring pasid > > 0x8000 Finished restoring pasid 0x8000 [drm] UVD VCPU state may lost > > due to RAS ERREVENT_ATHUB_INTERRUPT > > amdgpu: [powerplay] Failed to send message 0x26, response 0x0 > > amdgpu: [powerplay] Failed to set soft min gfxclk ! > > amdgpu: [powerplay] Failed to upload DPM Bootup Levels! > > amdgpu: [powerplay] Failed to send message 0x7, response 0x0 > > amdgpu: [powerplay] [DisableAllSMUFeatures] Failed to disable all smu > features! > > amdgpu: [powerplay] [DisableDpmTasks] Failed to disable all smu feature= s! > > amdgpu: [powerplay] [PowerOffAsic] Failed to disable DPM! > > [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of IP > > block failed -5 > > Do you have more information on what's going wrong here since this is a r= eally > important patch for KFD debugging. > > > > > Signed-off-by: Kent Russell > > Reviewed-by: Christian K=F6nig > > > --- > > drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 26 ---------------------= - > > 1 file changed, 26 deletions(-) > > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > index cf5d6e585634..a3f997f84020 100644 > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > @@ -254,32 +254,6 @@ void amdgpu_device_vram_access(struct > amdgpu_device *adev, loff_t pos, > > uint32_t hi =3D ~0; > > uint64_t last; > > > > - > > -#ifdef CONFIG_64BIT > > - last =3D min(pos + size, adev->gmc.visible_vram_size); > > - if (last > pos) { > > - void __iomem *addr =3D adev->mman.aper_base_kaddr + pos; > > - size_t count =3D last - pos; > > - > > - if (write) { > > - memcpy_toio(addr, buf, count); > > - mb(); > > - amdgpu_asic_flush_hdp(adev, NULL); > > - } else { > > - amdgpu_asic_invalidate_hdp(adev, NULL); > > - mb(); > > - memcpy_fromio(buf, addr, count); > > - } > > - > > - if (count =3D=3D size) > > - return; > > - > > - pos +=3D count; > > - buf +=3D count / 4; > > - size -=3D count; > > - } > > -#endif > > - > > spin_lock_irqsave(&adev->mmio_idx_lock, flags); > > for (last =3D pos + size; pos < last; pos +=3D 4) { > > uint32_t tmp =3D pos >> 31; _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flists.f= reedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=3D02%7C01%7Calexande= r.deucher%40amd.com%7C68e0bfea2a5f4a909ab108d7e07ed164%7C3dd8961fe4884e608e= 11a82d994e183d%7C0%7C0%7C637224707637289768&sdata=3DttNOHJt0IwywpOIWahK= jjuC6OkT1jxduc6iMzYzndpg%3D&reserved=3D0 Am 14.04.2020 16:35 schrieb "Deucher, Alexander" : [AMD Public Use] If this causes an issue, any access to vram via the BAR could cause an issu= e. Alex ________________________________ From: amd-gfx on behalf of Russell,= Kent Sent: Tuesday, April 14, 2020 10:19 AM To: Koenig, Christian ; amd-gfx@lists.freedesktop= .org Cc: Kuehling, Felix ; Kim, Jonathan Subject: RE: [PATCH] Revert "drm/amdgpu: use the BAR if possible in amdgpu_= device_vram_access v2" [AMD Official Use Only - Internal Distribution Only] On VG20 or MI100, as soon as we run the subtest, we get the dmesg output be= low, and then the kernel ends up hanging. I don't know enough about the tes= t itself to know why this is occurring, but Jon Kim and Felix were discussi= ng it on a separate thread when the issue was first reported, so they can h= opefully provide some additional information. Kent > -----Original Message----- > From: Christian K=F6nig > Sent: Tuesday, April 14, 2020 9:52 AM > To: Russell, Kent ; amd-gfx@lists.freedesktop.org > Subject: Re: [PATCH] Revert "drm/amdgpu: use the BAR if possible in > amdgpu_device_vram_access v2" > > Am 13.04.20 um 20:20 schrieb Kent Russell: > > This reverts commit c12b84d6e0d70f1185e6daddfd12afb671791b6e. > > The original patch causes a RAS event and subsequent kernel hard-hang > > when running the KFDMemoryTest.PtraceAccessInvisibleVram on VG20 and > > Arcturus > > > > dmesg output at hang time: > > [drm] RAS event of type ERREVENT_ATHUB_INTERRUPT detected! > > amdgpu 0000:67:00.0: GPU reset begin! > > Evicting PASID 0x8000 queues > > Started evicting pasid 0x8000 > > qcm fence wait loop timeout expired > > The cp might be in an unrecoverable state due to an unsuccessful > > queues preemption Failed to evict process queues Failed to suspend > > process 0x8000 Finished evicting pasid 0x8000 Started restoring pasid > > 0x8000 Finished restoring pasid 0x8000 [drm] UVD VCPU state may lost > > due to RAS ERREVENT_ATHUB_INTERRUPT > > amdgpu: [powerplay] Failed to send message 0x26, response 0x0 > > amdgpu: [powerplay] Failed to set soft min gfxclk ! > > amdgpu: [powerplay] Failed to upload DPM Bootup Levels! > > amdgpu: [powerplay] Failed to send message 0x7, response 0x0 > > amdgpu: [powerplay] [DisableAllSMUFeatures] Failed to disable all smu > features! > > amdgpu: [powerplay] [DisableDpmTasks] Failed to disable all smu feature= s! > > amdgpu: [powerplay] [PowerOffAsic] Failed to disable DPM! > > [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of IP > > block failed -5 > > Do you have more information on what's going wrong here since this is a r= eally > important patch for KFD debugging. > > > > > Signed-off-by: Kent Russell > > Reviewed-by: Christian K=F6nig > > > --- > > drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 26 ---------------------= - > > 1 file changed, 26 deletions(-) > > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > index cf5d6e585634..a3f997f84020 100644 > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > @@ -254,32 +254,6 @@ void amdgpu_device_vram_access(struct > amdgpu_device *adev, loff_t pos, > > uint32_t hi =3D ~0; > > uint64_t last; > > > > - > > -#ifdef CONFIG_64BIT > > - last =3D min(pos + size, adev->gmc.visible_vram_size); > > - if (last > pos) { > > - void __iomem *addr =3D adev->mman.aper_base_kaddr + pos; > > - size_t count =3D last - pos; > > - > > - if (write) { > > - memcpy_toio(addr, buf, count); > > - mb(); > > - amdgpu_asic_flush_hdp(adev, NULL); > > - } else { > > - amdgpu_asic_invalidate_hdp(adev, NULL); > > - mb(); > > - memcpy_fromio(buf, addr, count); > > - } > > - > > - if (count =3D=3D size) > > - return; > > - > > - pos +=3D count; > > - buf +=3D count / 4; > > - size -=3D count; > > - } > > -#endif > > - > > spin_lock_irqsave(&adev->mmio_idx_lock, flags); > > for (last =3D pos + size; pos < last; pos +=3D 4) { > > uint32_t tmp =3D pos >> 31; _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flists.f= reedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=3D02%7C01%7Calexande= r.deucher%40amd.com%7C68e0bfea2a5f4a909ab108d7e07ed164%7C3dd8961fe4884e608e= 11a82d994e183d%7C0%7C0%7C637224707637289768&sdata=3DttNOHJt0IwywpOIWahK= jjuC6OkT1jxduc6iMzYzndpg%3D&reserved=3D0 Am 14.04.2020 16:35 schrieb "Deucher, Alexander" : [AMD Public Use] If this causes an issue, any access to vram via the BAR could cause an issu= e. Alex ________________________________ From: amd-gfx on behalf of Russell,= Kent Sent: Tuesday, April 14, 2020 10:19 AM To: Koenig, Christian ; amd-gfx@lists.freedesktop= .org Cc: Kuehling, Felix ; Kim, Jonathan Subject: RE: [PATCH] Revert "drm/amdgpu: use the BAR if possible in amdgpu_= device_vram_access v2" [AMD Official Use Only - Internal Distribution Only] On VG20 or MI100, as soon as we run the subtest, we get the dmesg output be= low, and then the kernel ends up hanging. I don't know enough about the tes= t itself to know why this is occurring, but Jon Kim and Felix were discussi= ng it on a separate thread when the issue was first reported, so they can h= opefully provide some additional information. Kent > -----Original Message----- > From: Christian K=F6nig > Sent: Tuesday, April 14, 2020 9:52 AM > To: Russell, Kent ; amd-gfx@lists.freedesktop.org > Subject: Re: [PATCH] Revert "drm/amdgpu: use the BAR if possible in > amdgpu_device_vram_access v2" > > Am 13.04.20 um 20:20 schrieb Kent Russell: > > This reverts commit c12b84d6e0d70f1185e6daddfd12afb671791b6e. > > The original patch causes a RAS event and subsequent kernel hard-hang > > when running the KFDMemoryTest.PtraceAccessInvisibleVram on VG20 and > > Arcturus > > > > dmesg output at hang time: > > [drm] RAS event of type ERREVENT_ATHUB_INTERRUPT detected! > > amdgpu 0000:67:00.0: GPU reset begin! > > Evicting PASID 0x8000 queues > > Started evicting pasid 0x8000 > > qcm fence wait loop timeout expired > > The cp might be in an unrecoverable state due to an unsuccessful > > queues preemption Failed to evict process queues Failed to suspend > > process 0x8000 Finished evicting pasid 0x8000 Started restoring pasid > > 0x8000 Finished restoring pasid 0x8000 [drm] UVD VCPU state may lost > > due to RAS ERREVENT_ATHUB_INTERRUPT > > amdgpu: [powerplay] Failed to send message 0x26, response 0x0 > > amdgpu: [powerplay] Failed to set soft min gfxclk ! > > amdgpu: [powerplay] Failed to upload DPM Bootup Levels! > > amdgpu: [powerplay] Failed to send message 0x7, response 0x0 > > amdgpu: [powerplay] [DisableAllSMUFeatures] Failed to disable all smu > features! > > amdgpu: [powerplay] [DisableDpmTasks] Failed to disable all smu feature= s! > > amdgpu: [powerplay] [PowerOffAsic] Failed to disable DPM! > > [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of IP > > block failed -5 > > Do you have more information on what's going wrong here since this is a r= eally > important patch for KFD debugging. > > > > > Signed-off-by: Kent Russell > > Reviewed-by: Christian K=F6nig > > > --- > > drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 26 ---------------------= - > > 1 file changed, 26 deletions(-) > > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > index cf5d6e585634..a3f997f84020 100644 > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > @@ -254,32 +254,6 @@ void amdgpu_device_vram_access(struct > amdgpu_device *adev, loff_t pos, > > uint32_t hi =3D ~0; > > uint64_t last; > > > > - > > -#ifdef CONFIG_64BIT > > - last =3D min(pos + size, adev->gmc.visible_vram_size); > > - if (last > pos) { > > - void __iomem *addr =3D adev->mman.aper_base_kaddr + pos; > > - size_t count =3D last - pos; > > - > > - if (write) { > > - memcpy_toio(addr, buf, count); > > - mb(); > > - amdgpu_asic_flush_hdp(adev, NULL); > > - } else { > > - amdgpu_asic_invalidate_hdp(adev, NULL); > > - mb(); > > - memcpy_fromio(buf, addr, count); > > - } > > - > > - if (count =3D=3D size) > > - return; > > - > > - pos +=3D count; > > - buf +=3D count / 4; > > - size -=3D count; > > - } > > -#endif > > - > > spin_lock_irqsave(&adev->mmio_idx_lock, flags); > > for (last =3D pos + size; pos < last; pos +=3D 4) { > > uint32_t tmp =3D pos >> 31; _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flists.f= reedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=3D02%7C01%7Calexande= r.deucher%40amd.com%7C68e0bfea2a5f4a909ab108d7e07ed164%7C3dd8961fe4884e608e= 11a82d994e183d%7C0%7C0%7C637224707637289768&sdata=3DttNOHJt0IwywpOIWahK= jjuC6OkT1jxduc6iMzYzndpg%3D&reserved=3D0 --_000_146d9570724e423d931e24c96821aaaeemailandroidcom_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
That's exactly my concern as well.

This looks a bit like the test creates erroneous data som= ehow, but there doesn't seems to be a RAS check in the MM data path.

And now that we use the BAR path it goes up in flames.

I just don't see how we can create erroneous data in a te= st case?

Christian.

Am 14.04.2020 16:35 schrieb "Deucher, Alexa= nder" <Alexander.Deucher@amd.com>:

[AMD Public Use]


If this causes an issue, any access to vram via the BAR could cause an issu= e.

Alex

From: amd-gfx <amd-gfx-bounces@lists.freedesk= top.org> on behalf of Russell, Kent <Kent.Russell@amd.com>
Sent: Tuesday, April 14, 2020 10:19 AM
To: Koenig, Christian <Christian.Koenig@amd.com>; amd-gfx@list= s.freedesktop.org <amd-gfx@lists.freedesktop.org>
Cc: Kuehling, Felix <Felix.Kuehling@amd.com>; Kim, Jonathan &l= t;Jonathan.Kim@amd.com>
Subject: RE: [PATCH] Revert "drm/amdgpu: use the BAR if possibl= e in amdgpu_device_vram_access v2"
 
[AMD Official Use Only - Internal Distribution Only]

On VG20 or MI100, as soon as we run the subtest, we get the dmesg output be= low, and then the kernel ends up hanging. I don't know enough about the tes= t itself to know why this is occurring, but Jon Kim and Felix were discussi= ng it on a separate thread when the issue was first reported, so they can hopefully provide some additiona= l information.

 Kent

> -----Original Message-----
> From: Christian K=F6nig <ckoenig.leichtzumerken@gmail.com>
> Sent: Tuesday, April 14, 2020 9:52 AM
> To: Russell, Kent <Kent.Russell@amd.com>; amd-gfx@lists.freedesk= top.org
> Subject: Re: [PATCH] Revert "drm/amdgpu: use the BAR if possible = in
> amdgpu_device_vram_access v2"
>
> Am 13.04.20 um 20:20 schrieb Kent Russell:
> > This reverts commit c12b84d6e0d70f1185e6daddfd12afb671791b6e.
> > The original patch causes a RAS event and subsequent kernel hard-= hang
> > when running the KFDMemoryTest.PtraceAccessInvisibleVram on VG20 = and
> > Arcturus
> >
> > dmesg output at hang time:
> > [drm] RAS event of type ERREVENT_ATHUB_INTERRUPT detected!
> > amdgpu 0000:67:00.0: GPU reset begin!
> > Evicting PASID 0x8000 queues
> > Started evicting pasid 0x8000
> > qcm fence wait loop timeout expired
> > The cp might be in an unrecoverable state due to an unsuccessful<= br> > > queues preemption Failed to evict process queues Failed to suspen= d
> > process 0x8000 Finished evicting pasid 0x8000 Started restoring p= asid
> > 0x8000 Finished restoring pasid 0x8000 [drm] UVD VCPU state may l= ost
> > due to RAS ERREVENT_ATHUB_INTERRUPT
> > amdgpu: [powerplay] Failed to send message 0x26, response 0x0
> > amdgpu: [powerplay] Failed to set soft min gfxclk !
> > amdgpu: [powerplay] Failed to upload DPM Bootup Levels!
> > amdgpu: [powerplay] Failed to send message 0x7, response 0x0
> > amdgpu: [powerplay] [DisableAllSMUFeatures] Failed to disable all= smu
> features!
> > amdgpu: [powerplay] [DisableDpmTasks] Failed to disable all smu f= eatures!
> > amdgpu: [powerplay] [PowerOffAsic] Failed to disable DPM!
> > [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of= IP
> > block <powerplay> failed -5
>
> Do you have more information on what's going wrong here since this is = a really
> important patch for KFD debugging.
>
> >
> > Signed-off-by: Kent Russell <kent.russell@amd.com>
>
> Reviewed-by: Christian K=F6nig <christian.koenig@amd.com>
>
> > ---
> >   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 26 -----= -----------------
> >   1 file changed, 26 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > index cf5d6e585634..a3f997f84020 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > @@ -254,32 +254,6 @@ void amdgpu_device_vram_access(struct > amdgpu_device *adev, loff_t pos,
> >      uint32_t hi =3D ~0;
> >      uint64_t last;
> >
> > -
> > -#ifdef CONFIG_64BIT
> > -   last =3D min(pos + size, adev->gmc.visible_v= ram_size);
> > -   if (last > pos) {
> > -           voi= d __iomem *addr =3D adev->mman.aper_base_kaddr + pos;
> > -           siz= e_t count =3D last - pos;
> > -
> > -           if = (write) {
> > -          &nbs= p;        memcpy_toio(addr, buf, count);=
> > -          &nbs= p;        mb();
> > -          &nbs= p;        amdgpu_asic_flush_hdp(adev, NU= LL);
> > -           } e= lse {
> > -          &nbs= p;        amdgpu_asic_invalidate_hdp(ade= v, NULL);
> > -          &nbs= p;        mb();
> > -          &nbs= p;        memcpy_fromio(buf, addr, count= );
> > -           } > > -
> > -           if = (count =3D=3D size)
> > -          &nbs= p;        return;
> > -
> > -           pos= +=3D count;
> > -           buf= +=3D count / 4;
> > -           siz= e -=3D count;
> > -   }
> > -#endif
> > -
> >      spin_lock_irqsave(&adev->mmi= o_idx_lock, flags);
> >      for (last =3D pos + size; pos &= lt; last; pos +=3D 4) {
> >           =    uint32_t tmp =3D pos >> 31;
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https:= //nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flists.freedes= ktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=3D02%7C01%7Calexander.= deucher%40amd.com%7C68e0bfea2a5f4a909ab108d7e07ed164%7C3dd8961fe4884e608e11= a82d994e183d%7C0%7C0%7C637224707637289768&amp;sdata=3DttNOHJt0IwywpOIWa= hKjjuC6OkT1jxduc6iMzYzndpg%3D&amp;reserved=3D0


Am 14.04.2020 16:35 schrieb "Deucher, Alexa= nder" <Alexander.Deucher@amd.com>:

[AMD Public Use]


If this causes an issue, any access to vram via the BAR could cause an issu= e.

Alex

From: amd-gfx <amd-gfx-bounces@lists.freedesk= top.org> on behalf of Russell, Kent <Kent.Russell@amd.com>
Sent: Tuesday, April 14, 2020 10:19 AM
To: Koenig, Christian <Christian.Koenig@amd.com>; amd-gfx@list= s.freedesktop.org <amd-gfx@lists.freedesktop.org>
Cc: Kuehling, Felix <Felix.Kuehling@amd.com>; Kim, Jonathan &l= t;Jonathan.Kim@amd.com>
Subject: RE: [PATCH] Revert "drm/amdgpu: use the BAR if possibl= e in amdgpu_device_vram_access v2"
 
[AMD Official Use Only - Internal Distribution Only]

On VG20 or MI100, as soon as we run the subtest, we get the dmesg output be= low, and then the kernel ends up hanging. I don't know enough about the tes= t itself to know why this is occurring, but Jon Kim and Felix were discussi= ng it on a separate thread when the issue was first reported, so they can hopefully provide some additiona= l information.

 Kent

> -----Original Message-----
> From: Christian K=F6nig <ckoenig.leichtzumerken@gmail.com>
> Sent: Tuesday, April 14, 2020 9:52 AM
> To: Russell, Kent <Kent.Russell@amd.com>; amd-gfx@lists.freedesk= top.org
> Subject: Re: [PATCH] Revert "drm/amdgpu: use the BAR if possible = in
> amdgpu_device_vram_access v2"
>
> Am 13.04.20 um 20:20 schrieb Kent Russell:
> > This reverts commit c12b84d6e0d70f1185e6daddfd12afb671791b6e.
> > The original patch causes a RAS event and subsequent kernel hard-= hang
> > when running the KFDMemoryTest.PtraceAccessInvisibleVram on VG20 = and
> > Arcturus
> >
> > dmesg output at hang time:
> > [drm] RAS event of type ERREVENT_ATHUB_INTERRUPT detected!
> > amdgpu 0000:67:00.0: GPU reset begin!
> > Evicting PASID 0x8000 queues
> > Started evicting pasid 0x8000
> > qcm fence wait loop timeout expired
> > The cp might be in an unrecoverable state due to an unsuccessful<= br> > > queues preemption Failed to evict process queues Failed to suspen= d
> > process 0x8000 Finished evicting pasid 0x8000 Started restoring p= asid
> > 0x8000 Finished restoring pasid 0x8000 [drm] UVD VCPU state may l= ost
> > due to RAS ERREVENT_ATHUB_INTERRUPT
> > amdgpu: [powerplay] Failed to send message 0x26, response 0x0
> > amdgpu: [powerplay] Failed to set soft min gfxclk !
> > amdgpu: [powerplay] Failed to upload DPM Bootup Levels!
> > amdgpu: [powerplay] Failed to send message 0x7, response 0x0
> > amdgpu: [powerplay] [DisableAllSMUFeatures] Failed to disable all= smu
> features!
> > amdgpu: [powerplay] [DisableDpmTasks] Failed to disable all smu f= eatures!
> > amdgpu: [powerplay] [PowerOffAsic] Failed to disable DPM!
> > [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of= IP
> > block <powerplay> failed -5
>
> Do you have more information on what's going wrong here since this is = a really
> important patch for KFD debugging.
>
> >
> > Signed-off-by: Kent Russell <kent.russell@amd.com>
>
> Reviewed-by: Christian K=F6nig <christian.koenig@amd.com>
>
> > ---
> >   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 26 -----= -----------------
> >   1 file changed, 26 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > index cf5d6e585634..a3f997f84020 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > @@ -254,32 +254,6 @@ void amdgpu_device_vram_access(struct > amdgpu_device *adev, loff_t pos,
> >      uint32_t hi =3D ~0;
> >      uint64_t last;
> >
> > -
> > -#ifdef CONFIG_64BIT
> > -   last =3D min(pos + size, adev->gmc.visible_v= ram_size);
> > -   if (last > pos) {
> > -           voi= d __iomem *addr =3D adev->mman.aper_base_kaddr + pos;
> > -           siz= e_t count =3D last - pos;
> > -
> > -           if = (write) {
> > -          &nbs= p;        memcpy_toio(addr, buf, count);=
> > -          &nbs= p;        mb();
> > -          &nbs= p;        amdgpu_asic_flush_hdp(adev, NU= LL);
> > -           } e= lse {
> > -          &nbs= p;        amdgpu_asic_invalidate_hdp(ade= v, NULL);
> > -          &nbs= p;        mb();
> > -          &nbs= p;        memcpy_fromio(buf, addr, count= );
> > -           } > > -
> > -           if = (count =3D=3D size)
> > -          &nbs= p;        return;
> > -
> > -           pos= +=3D count;
> > -           buf= +=3D count / 4;
> > -           siz= e -=3D count;
> > -   }
> > -#endif
> > -
> >      spin_lock_irqsave(&adev->mmi= o_idx_lock, flags);
> >      for (last =3D pos + size; pos &= lt; last; pos +=3D 4) {
> >           =    uint32_t tmp =3D pos >> 31;
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https:= //nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flists.freedes= ktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=3D02%7C01%7Calexander.= deucher%40amd.com%7C68e0bfea2a5f4a909ab108d7e07ed164%7C3dd8961fe4884e608e11= a82d994e183d%7C0%7C0%7C637224707637289768&amp;sdata=3DttNOHJt0IwywpOIWa= hKjjuC6OkT1jxduc6iMzYzndpg%3D&amp;reserved=3D0


Am 14.04.2020 16:35 schrieb "Deucher, Alexa= nder" <Alexander.Deucher@amd.com>:

[AMD Public Use]


If this causes an issue, any access to vram via the BAR could cause an issu= e.

Alex

From: amd-gfx <amd-gfx-bounces@lists.freedesk= top.org> on behalf of Russell, Kent <Kent.Russell@amd.com>
Sent: Tuesday, April 14, 2020 10:19 AM
To: Koenig, Christian <Christian.Koenig@amd.com>; amd-gfx@list= s.freedesktop.org <amd-gfx@lists.freedesktop.org>
Cc: Kuehling, Felix <Felix.Kuehling@amd.com>; Kim, Jonathan &l= t;Jonathan.Kim@amd.com>
Subject: RE: [PATCH] Revert "drm/amdgpu: use the BAR if possibl= e in amdgpu_device_vram_access v2"
 
[AMD Official Use Only - Internal Distribution Only]

On VG20 or MI100, as soon as we run the subtest, we get the dmesg output be= low, and then the kernel ends up hanging. I don't know enough about the tes= t itself to know why this is occurring, but Jon Kim and Felix were discussi= ng it on a separate thread when the issue was first reported, so they can hopefully provide some additiona= l information.

 Kent

> -----Original Message-----
> From: Christian K=F6nig <ckoenig.leichtzumerken@gmail.com>
> Sent: Tuesday, April 14, 2020 9:52 AM
> To: Russell, Kent <Kent.Russell@amd.com>; amd-gfx@lists.freedesk= top.org
> Subject: Re: [PATCH] Revert "drm/amdgpu: use the BAR if possible = in
> amdgpu_device_vram_access v2"
>
> Am 13.04.20 um 20:20 schrieb Kent Russell:
> > This reverts commit c12b84d6e0d70f1185e6daddfd12afb671791b6e.
> > The original patch causes a RAS event and subsequent kernel hard-= hang
> > when running the KFDMemoryTest.PtraceAccessInvisibleVram on VG20 = and
> > Arcturus
> >
> > dmesg output at hang time:
> > [drm] RAS event of type ERREVENT_ATHUB_INTERRUPT detected!
> > amdgpu 0000:67:00.0: GPU reset begin!
> > Evicting PASID 0x8000 queues
> > Started evicting pasid 0x8000
> > qcm fence wait loop timeout expired
> > The cp might be in an unrecoverable state due to an unsuccessful<= br> > > queues preemption Failed to evict process queues Failed to suspen= d
> > process 0x8000 Finished evicting pasid 0x8000 Started restoring p= asid
> > 0x8000 Finished restoring pasid 0x8000 [drm] UVD VCPU state may l= ost
> > due to RAS ERREVENT_ATHUB_INTERRUPT
> > amdgpu: [powerplay] Failed to send message 0x26, response 0x0
> > amdgpu: [powerplay] Failed to set soft min gfxclk !
> > amdgpu: [powerplay] Failed to upload DPM Bootup Levels!
> > amdgpu: [powerplay] Failed to send message 0x7, response 0x0
> > amdgpu: [powerplay] [DisableAllSMUFeatures] Failed to disable all= smu
> features!
> > amdgpu: [powerplay] [DisableDpmTasks] Failed to disable all smu f= eatures!
> > amdgpu: [powerplay] [PowerOffAsic] Failed to disable DPM!
> > [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of= IP
> > block <powerplay> failed -5
>
> Do you have more information on what's going wrong here since this is = a really
> important patch for KFD debugging.
>
> >
> > Signed-off-by: Kent Russell <kent.russell@amd.com>
>
> Reviewed-by: Christian K=F6nig <christian.koenig@amd.com>
>
> > ---
> >   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 26 -----= -----------------
> >   1 file changed, 26 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > index cf5d6e585634..a3f997f84020 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > @@ -254,32 +254,6 @@ void amdgpu_device_vram_access(struct > amdgpu_device *adev, loff_t pos,
> >      uint32_t hi =3D ~0;
> >      uint64_t last;
> >
> > -
> > -#ifdef CONFIG_64BIT
> > -   last =3D min(pos + size, adev->gmc.visible_v= ram_size);
> > -   if (last > pos) {
> > -           voi= d __iomem *addr =3D adev->mman.aper_base_kaddr + pos;
> > -           siz= e_t count =3D last - pos;
> > -
> > -           if = (write) {
> > -          &nbs= p;        memcpy_toio(addr, buf, count);=
> > -          &nbs= p;        mb();
> > -          &nbs= p;        amdgpu_asic_flush_hdp(adev, NU= LL);
> > -           } e= lse {
> > -          &nbs= p;        amdgpu_asic_invalidate_hdp(ade= v, NULL);
> > -          &nbs= p;        mb();
> > -          &nbs= p;        memcpy_fromio(buf, addr, count= );
> > -           } > > -
> > -           if = (count =3D=3D size)
> > -          &nbs= p;        return;
> > -
> > -           pos= +=3D count;
> > -           buf= +=3D count / 4;
> > -           siz= e -=3D count;
> > -   }
> > -#endif
> > -
> >      spin_lock_irqsave(&adev->mmi= o_idx_lock, flags);
> >      for (last =3D pos + size; pos &= lt; last; pos +=3D 4) {
> >           =    uint32_t tmp =3D pos >> 31;
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https:= //nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flists.freedes= ktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=3D02%7C01%7Calexander.= deucher%40amd.com%7C68e0bfea2a5f4a909ab108d7e07ed164%7C3dd8961fe4884e608e11= a82d994e183d%7C0%7C0%7C637224707637289768&amp;sdata=3DttNOHJt0IwywpOIWa= hKjjuC6OkT1jxduc6iMzYzndpg%3D&amp;reserved=3D0


Am 14.04.2020 16:35 schrieb "Deucher, Alexa= nder" <Alexander.Deucher@amd.com>:

[AMD Public Use]


If this causes an issue, any access to vram via the BAR could cause an issu= e.

Alex

From: amd-gfx <amd-gfx-bounces@lists.freedesk= top.org> on behalf of Russell, Kent <Kent.Russell@amd.com>
Sent: Tuesday, April 14, 2020 10:19 AM
To: Koenig, Christian <Christian.Koenig@amd.com>; amd-gfx@list= s.freedesktop.org <amd-gfx@lists.freedesktop.org>
Cc: Kuehling, Felix <Felix.Kuehling@amd.com>; Kim, Jonathan &l= t;Jonathan.Kim@amd.com>
Subject: RE: [PATCH] Revert "drm/amdgpu: use the BAR if possibl= e in amdgpu_device_vram_access v2"
 
[AMD Official Use Only - Internal Distribution Only]

On VG20 or MI100, as soon as we run the subtest, we get the dmesg output be= low, and then the kernel ends up hanging. I don't know enough about the tes= t itself to know why this is occurring, but Jon Kim and Felix were discussi= ng it on a separate thread when the issue was first reported, so they can hopefully provide some additiona= l information.

 Kent

> -----Original Message-----
> From: Christian K=F6nig <ckoenig.leichtzumerken@gmail.com>
> Sent: Tuesday, April 14, 2020 9:52 AM
> To: Russell, Kent <Kent.Russell@amd.com>; amd-gfx@lists.freedesk= top.org
> Subject: Re: [PATCH] Revert "drm/amdgpu: use the BAR if possible = in
> amdgpu_device_vram_access v2"
>
> Am 13.04.20 um 20:20 schrieb Kent Russell:
> > This reverts commit c12b84d6e0d70f1185e6daddfd12afb671791b6e.
> > The original patch causes a RAS event and subsequent kernel hard-= hang
> > when running the KFDMemoryTest.PtraceAccessInvisibleVram on VG20 = and
> > Arcturus
> >
> > dmesg output at hang time:
> > [drm] RAS event of type ERREVENT_ATHUB_INTERRUPT detected!
> > amdgpu 0000:67:00.0: GPU reset begin!
> > Evicting PASID 0x8000 queues
> > Started evicting pasid 0x8000
> > qcm fence wait loop timeout expired
> > The cp might be in an unrecoverable state due to an unsuccessful<= br> > > queues preemption Failed to evict process queues Failed to suspen= d
> > process 0x8000 Finished evicting pasid 0x8000 Started restoring p= asid
> > 0x8000 Finished restoring pasid 0x8000 [drm] UVD VCPU state may l= ost
> > due to RAS ERREVENT_ATHUB_INTERRUPT
> > amdgpu: [powerplay] Failed to send message 0x26, response 0x0
> > amdgpu: [powerplay] Failed to set soft min gfxclk !
> > amdgpu: [powerplay] Failed to upload DPM Bootup Levels!
> > amdgpu: [powerplay] Failed to send message 0x7, response 0x0
> > amdgpu: [powerplay] [DisableAllSMUFeatures] Failed to disable all= smu
> features!
> > amdgpu: [powerplay] [DisableDpmTasks] Failed to disable all smu f= eatures!
> > amdgpu: [powerplay] [PowerOffAsic] Failed to disable DPM!
> > [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of= IP
> > block <powerplay> failed -5
>
> Do you have more information on what's going wrong here since this is = a really
> important patch for KFD debugging.
>
> >
> > Signed-off-by: Kent Russell <kent.russell@amd.com>
>
> Reviewed-by: Christian K=F6nig <christian.koenig@amd.com>
>
> > ---
> >   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 26 -----= -----------------
> >   1 file changed, 26 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > index cf5d6e585634..a3f997f84020 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > @@ -254,32 +254,6 @@ void amdgpu_device_vram_access(struct > amdgpu_device *adev, loff_t pos,
> >      uint32_t hi =3D ~0;
> >      uint64_t last;
> >
> > -
> > -#ifdef CONFIG_64BIT
> > -   last =3D min(pos + size, adev->gmc.visible_v= ram_size);
> > -   if (last > pos) {
> > -           voi= d __iomem *addr =3D adev->mman.aper_base_kaddr + pos;
> > -           siz= e_t count =3D last - pos;
> > -
> > -           if = (write) {
> > -          &nbs= p;        memcpy_toio(addr, buf, count);=
> > -          &nbs= p;        mb();
> > -          &nbs= p;        amdgpu_asic_flush_hdp(adev, NU= LL);
> > -           } e= lse {
> > -          &nbs= p;        amdgpu_asic_invalidate_hdp(ade= v, NULL);
> > -          &nbs= p;        mb();
> > -          &nbs= p;        memcpy_fromio(buf, addr, count= );
> > -           } > > -
> > -           if = (count =3D=3D size)
> > -          &nbs= p;        return;
> > -
> > -           pos= +=3D count;
> > -           buf= +=3D count / 4;
> > -           siz= e -=3D count;
> > -   }
> > -#endif
> > -
> >      spin_lock_irqsave(&adev->mmi= o_idx_lock, flags);
> >      for (last =3D pos + size; pos &= lt; last; pos +=3D 4) {
> >           =    uint32_t tmp =3D pos >> 31;
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https:= //nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flists.freedes= ktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=3D02%7C01%7Calexander.= deucher%40amd.com%7C68e0bfea2a5f4a909ab108d7e07ed164%7C3dd8961fe4884e608e11= a82d994e183d%7C0%7C0%7C637224707637289768&amp;sdata=3DttNOHJt0IwywpOIWa= hKjjuC6OkT1jxduc6iMzYzndpg%3D&amp;reserved=3D0


Am 14.04.2020 16:35 schrieb "Deucher, Alexa= nder" <Alexander.Deucher@amd.com>:

[AMD Public Use]


If this causes an issue, any access to vram via the BAR could cause an issu= e.

Alex

From: amd-gfx <amd-gfx-b= ounces@lists.freedesktop.org> on behalf of Russell, Kent <Kent.Russel= l@amd.com>
Sent: Tuesday, April 14, 2020 10:19 AM
To: Koenig, Christian <Christian.Koenig@amd.com>; amd-gfx@list= s.freedesktop.org <amd-gfx@lists.freedesktop.org>
Cc: Kuehling, Felix <Felix.Kuehling@amd.com>; Kim, Jonathan &l= t;Jonathan.Kim@amd.com>
Subject: RE: [PATCH] Revert "drm/amdgpu: use the BAR if possibl= e in amdgpu_device_vram_access v2"
 
[AMD Official Use Only - Internal Distribution Onl= y]

On VG20 or MI100, as soon as we run the subtest, we get the dmesg output be= low, and then the kernel ends up hanging. I don't know enough about the tes= t itself to know why this is occurring, but Jon Kim and Felix were discussi= ng it on a separate thread when the issue was first reported, so they can hopefully provide some additiona= l information.

 Kent

> -----Original Message-----
> From: Christian K=F6nig <ckoenig.leichtzumerken@gmail.com>
> Sent: Tuesday, April 14, 2020 9:52 AM
> To: Russell, Kent <Kent.Russell@amd.com>; amd-gfx@lists.freedesk= top.org
> Subject: Re: [PATCH] Revert "drm/amdgpu: use the BAR if possible = in
> amdgpu_device_vram_access v2"
>
> Am 13.04.20 um 20:20 schrieb Kent Russell:
> > This reverts commit c12b84d6e0d70f1185e6daddfd12afb671791b6e.
> > The original patch causes a RAS event and subsequent kernel hard-= hang
> > when running the KFDMemoryTest.PtraceAccessInvisibleVram on VG20 = and
> > Arcturus
> >
> > dmesg output at hang time:
> > [drm] RAS event of type ERREVENT_ATHUB_INTERRUPT detected!
> > amdgpu 0000:67:00.0: GPU reset begin!
> > Evicting PASID 0x8000 queues
> > Started evicting pasid 0x8000
> > qcm fence wait loop timeout expired
> > The cp might be in an unrecoverable state due to an unsuccessful<= br> > > queues preemption Failed to evict process queues Failed to suspen= d
> > process 0x8000 Finished evicting pasid 0x8000 Started restoring p= asid
> > 0x8000 Finished restoring pasid 0x8000 [drm] UVD VCPU state may l= ost
> > due to RAS ERREVENT_ATHUB_INTERRUPT
> > amdgpu: [powerplay] Failed to send message 0x26, response 0x0
> > amdgpu: [powerplay] Failed to set soft min gfxclk !
> > amdgpu: [powerplay] Failed to upload DPM Bootup Levels!
> > amdgpu: [powerplay] Failed to send message 0x7, response 0x0
> > amdgpu: [powerplay] [DisableAllSMUFeatures] Failed to disable all= smu
> features!
> > amdgpu: [powerplay] [DisableDpmTasks] Failed to disable all smu f= eatures!
> > amdgpu: [powerplay] [PowerOffAsic] Failed to disable DPM!
> > [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of= IP
> > block <powerplay> failed -5
>
> Do you have more information on what's going wrong here since this is = a really
> important patch for KFD debugging.
>
> >
> > Signed-off-by: Kent Russell <kent.russell@amd.com>
>
> Reviewed-by: Christian K=F6nig <christian.koenig@amd.com>
>
> > ---
> >   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 26 -----= -----------------
> >   1 file changed, 26 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > index cf5d6e585634..a3f997f84020 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > @@ -254,32 +254,6 @@ void amdgpu_device_vram_access(struct > amdgpu_device *adev, loff_t pos,
> >      uint32_t hi =3D ~0;
> >      uint64_t last;
> >
> > -
> > -#ifdef CONFIG_64BIT
> > -   last =3D min(pos + size, adev->gmc.visible_v= ram_size);
> > -   if (last > pos) {
> > -           voi= d __iomem *addr =3D adev->mman.aper_base_kaddr + pos;
> > -           siz= e_t count =3D last - pos;
> > -
> > -           if = (write) {
> > -          &nbs= p;        memcpy_toio(addr, buf, count);=
> > -          &nbs= p;        mb();
> > -          &nbs= p;        amdgpu_asic_flush_hdp(adev, NU= LL);
> > -           } e= lse {
> > -          &nbs= p;        amdgpu_asic_invalidate_hdp(ade= v, NULL);
> > -          &nbs= p;        mb();
> > -          &nbs= p;        memcpy_fromio(buf, addr, count= );
> > -           } > > -
> > -           if = (count =3D=3D size)
> > -          &nbs= p;        return;
> > -
> > -           pos= +=3D count;
> > -           buf= +=3D count / 4;
> > -           siz= e -=3D count;
> > -   }
> > -#endif
> > -
> >      spin_lock_irqsave(&adev->mmi= o_idx_lock, flags);
> >      for (last =3D pos + size; pos &= lt; last; pos +=3D 4) {
> >           =    uint32_t tmp =3D pos >> 31;
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https:= //nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flists.freedes= ktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=3D02%7C01%7Calexander.= deucher%40amd.com%7C68e0bfea2a5f4a909ab108d7e07ed164%7C3dd8961fe4884e608e11= a82d994e183d%7C0%7C0%7C637224707637289768&amp;sdata=3DttNOHJt0IwywpOIWa= hKjjuC6OkT1jxduc6iMzYzndpg%3D&amp;reserved=3D0
--_000_146d9570724e423d931e24c96821aaaeemailandroidcom_-- --===============0738367152== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx --===============0738367152==--