Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 347D5C0218A for ; Thu, 30 Jan 2025 18:39:04 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4YkSTy2SVlz30VF; Fri, 31 Jan 2025 05:39:02 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1738262342; cv=none; b=VpPfQGWy7OAOlsepPMmWsT6v2doz8YI0NigvXG0OVYQe26NI7e/ezcYrpQfaV7k2sZJ7+omzYKrvPT4uB/3nCxqFU/Ex475t65w4mWIMe9xPHzXuBtRhpuYFREsXQhCJXJszsrJdkeVYbx5kYdpHyOtzFbt4RBNwESTst/vvxzpFhjQOkSPzjB5Mg0oXPt4B2xQmOzlxUsycGAsdhDF9ohbsqiB2hFoJOcDcA5KuHpflrcmeWwy7S0jXpLzYTFCprmcAjsUuXyFxC85to5WOa5ALugiyDCkTdg/8J2J0gu1/BpX1DHIw7eJlZUGkrUy9N5yLQCVUqk4RK+A/SHT2rw== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1738262342; c=relaxed/relaxed; bh=7UAJEyhAW492CSssW4BRzSNFka9LYJ4BAF5aeAs1BDQ=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version; b=aFs0eaIg2K8GZbdiTjDTIjZXA3/Jnoprv0rrZiTRASCYlpbkGZsBTJ2MbVLnnK1uzfnOWiQsHngoGI00XmDxQ8qqJX+ttNFquv4r9N/a/n38F8P+MyClTGGGY+ky/Y2ni1kDK/QdZSLITxMX9k7+BSHiO6VOvVvdfRc6gFSO1yiRud+T8TVLL5U5//3hDOtl7z7fm3mXhmvxOUsWbUXyoz2HO0uvMfT7Hd2ZPY5oemNmN9PnsVFisIMyGtu/MV2oPSkq0WrVVsDKoVlH1CTzeHb83wvnEV5Z8ymQaDMOTxKqJXAyTYjPu2wGx/UWMt+uDfrh0zYvSdFWh8ZPIhf43g== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=IC2YsInT; dkim-atps=neutral; spf=pass (client-ip=148.163.158.5; helo=mx0b-001b2d01.pphosted.com; envelope-from=gbatra@linux.ibm.com; receiver=lists.ozlabs.org) smtp.mailfrom=linux.ibm.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=IC2YsInT; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.158.5; helo=mx0b-001b2d01.pphosted.com; envelope-from=gbatra@linux.ibm.com; receiver=lists.ozlabs.org) Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4YkSTx025Qz2yjR for ; Fri, 31 Jan 2025 05:39:00 +1100 (AEDT) Received: from pps.filterd (m0356516.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 50UFe8W7011372 for ; Thu, 30 Jan 2025 18:38:57 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:message-id:mime-version :subject:to; s=pp1; bh=7UAJEyhAW492CSssW4BRzSNFka9LYJ4BAF5aeAs1B DQ=; b=IC2YsInToCG233eYDYfcmlxoz5IAe9o+XJoNkQPDmUf1tpsNNZPfesvgP I4n/W0v2OXPzsrhPW/WqGUpWx9Gi1BvgGNZwseseBdkoGpfYFEeIGw0xFcxga/Xx 1GkkVPz9cIkkq4AXtowtz/tfIH0sCy2LZ8y5UvxDxGVLxkfWzhpiD1ZnT514qGrT Lg7ZTwyK6VXeEApZVA1I6do3ds7ZJynu9EjRw1SB9IpuydwOKZ7tE3iIsca/MWpe EsD2nJnKk0qHanyaQjuAzYFdTeEaw3/HXkKJZM1klZ9kFTsFufEekrDH35FkXXw7 V56z8VeXutiKF3BJSYU88/6Vz2djg== Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 44gc8trupa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 30 Jan 2025 18:38:57 +0000 (GMT) Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 50UG49hA019348 for ; Thu, 30 Jan 2025 18:38:56 GMT Received: from smtprelay06.dal12v.mail.ibm.com ([172.16.1.8]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 44db9n7fee-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 30 Jan 2025 18:38:56 +0000 Received: from smtpav05.wdc07v.mail.ibm.com (smtpav05.wdc07v.mail.ibm.com [10.39.53.232]) by smtprelay06.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 50UIctg89961994 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 30 Jan 2025 18:38:55 GMT Received: from smtpav05.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 53CC658063; Thu, 30 Jan 2025 18:38:55 +0000 (GMT) Received: from smtpav05.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 003605805F; Thu, 30 Jan 2025 18:38:55 +0000 (GMT) Received: from localhost.localdomain (unknown [9.24.23.139]) by smtpav05.wdc07v.mail.ibm.com (Postfix) with ESMTP; Thu, 30 Jan 2025 18:38:54 +0000 (GMT) From: Gaurav Batra To: linuxppc-dev@lists.ozlabs.org Cc: donettom@linux.ibm.com, Gaurav Batra Subject: [PATCH] powerpc/pseries/iommu: memory notifier incorrectly adds TCEs for pmemory Date: Thu, 30 Jan 2025 12:38:54 -0600 Message-Id: <20250130183854.92258-1-gbatra@linux.ibm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: hEXen9QxMmgeLd4agy88Ob2Wqm6dukxI X-Proofpoint-GUID: hEXen9QxMmgeLd4agy88Ob2Wqm6dukxI X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-01-30_08,2025-01-30_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 mlxscore=0 bulkscore=0 adultscore=0 impostorscore=0 lowpriorityscore=0 mlxlogscore=969 clxscore=1015 priorityscore=1501 malwarescore=0 spamscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2411120000 definitions=main-2501300140 iommu_mem_notifier() is invoked when RAM is dynamically added/removed. This notifier call is responsible to add/remove TCEs from the Dynamic DMA Window (DDW) when TCEs are pre-mapped. TCEs are pre-mapped only for RAM and not for persistent memory (pmemory). For DMA buffers in pmemory, TCEs are dynamically mapped when the device driver instructs to do so. The issue is 'daxctl' command is capable of adding pmemory as "System RAM" after LPAR boot. The command to do so is - daxctl reconfigure-device --mode=system-ram dax0.0 --force This will dynamically add pmemory range to LPAR RAM eventually invoking iommu_mem_notifier(). The address range of pmemory is way beyond the Max RAM that the LPAR can have. Which means, this range is beyond the DDW created for the device, at device initialization time. As a result when TCEs are pre-mapped for the pmemory range, by iommu_mem_notifier(), PHYP HCALL returns H_PARAMETER. This failed the command, daxctl, to add pmemory as RAM. The solution is to not pre-map TCEs for pmemory. Signed-off-by: Gaurav Batra --- arch/powerpc/include/asm/mmzone.h | 1 + arch/powerpc/mm/numa.c | 2 +- arch/powerpc/platforms/pseries/iommu.c | 29 ++++++++++++++------------ 3 files changed, 18 insertions(+), 14 deletions(-) diff --git a/arch/powerpc/include/asm/mmzone.h b/arch/powerpc/include/asm/mmzone.h index d99863cd6cde..049152f8d597 100644 --- a/arch/powerpc/include/asm/mmzone.h +++ b/arch/powerpc/include/asm/mmzone.h @@ -29,6 +29,7 @@ extern cpumask_var_t node_to_cpumask_map[]; #ifdef CONFIG_MEMORY_HOTPLUG extern unsigned long max_pfn; u64 memory_hotplug_max(void); +u64 hot_add_drconf_memory_max(void); #else #define memory_hotplug_max() memblock_end_of_DRAM() #endif diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c index 3c1da08304d0..603a0f652ba6 100644 --- a/arch/powerpc/mm/numa.c +++ b/arch/powerpc/mm/numa.c @@ -1336,7 +1336,7 @@ int hot_add_scn_to_nid(unsigned long scn_addr) return nid; } -static u64 hot_add_drconf_memory_max(void) +u64 hot_add_drconf_memory_max(void) { struct device_node *memory = NULL; struct device_node *dn = NULL; diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c index 29f1a0cc59cd..abd9529a8f41 100644 --- a/arch/powerpc/platforms/pseries/iommu.c +++ b/arch/powerpc/platforms/pseries/iommu.c @@ -1284,17 +1284,13 @@ static LIST_HEAD(failed_ddw_pdn_list); static phys_addr_t ddw_memory_hotplug_max(void) { - resource_size_t max_addr = memory_hotplug_max(); - struct device_node *memory; + resource_size_t max_addr; - for_each_node_by_type(memory, "memory") { - struct resource res; - - if (of_address_to_resource(memory, 0, &res)) - continue; - - max_addr = max_t(resource_size_t, max_addr, res.end + 1); - } +#if defined(CONFIG_NUMA) && defined(CONFIG_MEMORY_HOTPLUG) + max_addr = hot_add_drconf_memory_max(); +#else + max_addr = memblock_end_of_DRAM(); +#endif return max_addr; } @@ -1600,7 +1596,7 @@ static bool enable_ddw(struct pci_dev *dev, struct device_node *pdn) if (direct_mapping) { /* DDW maps the whole partition, so enable direct DMA mapping */ - ret = walk_system_ram_range(0, memblock_end_of_DRAM() >> PAGE_SHIFT, + ret = walk_system_ram_range(0, ddw_memory_hotplug_max() >> PAGE_SHIFT, win64->value, tce_setrange_multi_pSeriesLP_walk); if (ret) { dev_info(&dev->dev, "failed to map DMA window for %pOF: %d\n", @@ -2346,11 +2342,17 @@ static int iommu_mem_notifier(struct notifier_block *nb, unsigned long action, struct memory_notify *arg = data; int ret = 0; + /* This notifier can get called when onlining persistent memory as well. + * TCEs are not pre-mapped for persistent memory. Persistent memory will + * always be above ddw_memory_hotplug_max() + */ + switch (action) { case MEM_GOING_ONLINE: spin_lock(&dma_win_list_lock); list_for_each_entry(window, &dma_win_list, list) { - if (window->direct) { + if (window->direct && (arg->start_pfn << PAGE_SHIFT) < + ddw_memory_hotplug_max()) { ret |= tce_setrange_multi_pSeriesLP(arg->start_pfn, arg->nr_pages, window->prop); } @@ -2362,7 +2364,8 @@ static int iommu_mem_notifier(struct notifier_block *nb, unsigned long action, case MEM_OFFLINE: spin_lock(&dma_win_list_lock); list_for_each_entry(window, &dma_win_list, list) { - if (window->direct) { + if (window->direct && (arg->start_pfn << PAGE_SHIFT) < + ddw_memory_hotplug_max()) { ret |= tce_clearrange_multi_pSeriesLP(arg->start_pfn, arg->nr_pages, window->prop); } base-commit: 95ec54a420b8f445e04a7ca0ea8deb72c51fe1d3 -- 2.39.3 (Apple Git-146)