Received: from list by lists.gnu.org with archive (Exim 4.90_1) id 1m2vFE-0004EU-87 for mharc-grub-devel@gnu.org; Mon, 12 Jul 2021 08:41:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42682) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m2vFB-00049n-G1 for grub-devel@gnu.org; Mon, 12 Jul 2021 08:41:05 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:42924) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m2vF8-0004s3-Bv for grub-devel@gnu.org; Mon, 12 Jul 2021 08:41:05 -0400 Received: from pps.filterd (m0127361.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16CCY1jI069210; Mon, 12 Jul 2021 08:41:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=subject : to : cc : references : from : message-id : date : mime-version : in-reply-to : content-type : content-transfer-encoding; s=pp1; bh=FvgDqgHfRA2WQhsMBGd74ZVP3BNPT7fRsfsZAnZWPL8=; b=rX18ae0LnAD3IJpXL9HZLO1ffJGF8hD8K+2ccjZ4y8n+kij0PDrfj/ZmO54gyZybKjbW w7bxSXCrGkZg3V7WGBC0SGB6dOktwYZ84VpS160N3gmML3u5FGhZ2BszIpEQaRZ15/o0 L+9p833qNpwe8ki+C7AWF7kuOi4XXqGyPhOxfyWAA8CU+lga/l/ZcIYDDh/KQnkIxKnx f+1blRexbToJkB9OBC0+nKtJxSEbj36OZpuW3Vjm9jBGxWSVX4hDNA/xfDcf9yCp1lIR 16C7GKU47dyzk9lQVy9AwLAGbev5xzDDR1Qux/Ebgk++6rTJ8F93K7xsGi8pDSxEq/9Y 7A== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 39qrsxrnu9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 12 Jul 2021 08:40:59 -0400 Received: from m0127361.ppops.net (m0127361.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 16CCcTQk086931; Mon, 12 Jul 2021 08:40:59 -0400 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com with ESMTP id 39qrsxrntw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 12 Jul 2021 08:40:59 -0400 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 16CCcGUN015775; Mon, 12 Jul 2021 12:40:59 GMT Received: from b03cxnp07029.gho.boulder.ibm.com (b03cxnp07029.gho.boulder.ibm.com [9.17.130.16]) by ppma02dal.us.ibm.com with ESMTP id 39qt39sjbk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 12 Jul 2021 12:40:58 +0000 Received: from b03ledav002.gho.boulder.ibm.com (b03ledav002.gho.boulder.ibm.com [9.17.130.233]) by b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 16CCevw648366064 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Jul 2021 12:40:57 GMT Received: from b03ledav002.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B78E0136053; Mon, 12 Jul 2021 12:40:57 +0000 (GMT) Received: from b03ledav002.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 011DC136061; Mon, 12 Jul 2021 12:40:56 +0000 (GMT) Received: from [9.47.158.152] (unknown [9.47.158.152]) by b03ledav002.gho.boulder.ibm.com (Postfix) with ESMTP; Mon, 12 Jul 2021 12:40:56 +0000 (GMT) Subject: Re: [PATCH v2 03/22] ieee1275: request memory with ibm, client-architecture-support To: The development of GNU GRUB , Daniel Axtens Cc: rashmica.g@gmail.com, alastair@d-silva.org, nayna@linux.ibm.com References: <20210630084031.2663622-1-dja@axtens.net> <20210630084031.2663622-4-dja@axtens.net> From: Stefan Berger Message-ID: <10cdcfb1-83cf-dd10-ecd3-fd7e1a723f76@linux.ibm.com> Date: Mon, 12 Jul 2021 08:40:56 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <20210630084031.2663622-4-dja@axtens.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 7kq5QGRGccIWn9mu5GRZkpwEWsTC9O2L X-Proofpoint-GUID: vvO2NL2ZbFHck5Aonbht9vTmrs3JxaZD X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-07-12_07:2021-07-12, 2021-07-12 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 suspectscore=0 mlxscore=0 bulkscore=0 lowpriorityscore=0 adultscore=0 impostorscore=0 spamscore=0 phishscore=0 priorityscore=1501 clxscore=1015 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107120099 Received-SPF: pass client-ip=148.163.158.5; envelope-from=stefanb@linux.ibm.com; helo=mx0b-001b2d01.pphosted.com X-Spam_score_int: -34 X-Spam_score: -3.5 X-Spam_bar: --- X-Spam_report: (-3.5 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, NICE_REPLY_A=-1.479, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: grub-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: The development of GNU GRUB List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jul 2021 12:41:05 -0000 On 6/30/21 4:40 AM, Daniel Axtens wrote: > On PowerVM, the first time we boot a Linux partition, we may only get > 256MB of real memory area, even if the partition has more memory. > > This isn't really enough. Fortunately, the Power Architecture Platform > Reference (PAPR) defines a method we can call to ask for more memory. > This is part of the broad and powerful ibm,client-architecture-support > (CAS) method. > > CAS can do an enormous amount of things on a PAPR platform: as well as > asking for memory, you can set the supported processor level, the interrupt > controller, hash vs radix mmu, and so on. We want to touch as little of > this as possible because we don't want to step on the toes of the future OS. > > If: > > - we are running under what we think is PowerVM (compatible property of / > begins with "IBM"), and > > - the full amount of RMA is less than 512MB (as determined by the reg > property of /memory) > > then call CAS as follows: (refer to the Linux on Power Architecture > Reference, LoPAR, which is public, at B.5.2.3): > > - Use the "any" PVR value and supply 2 option vectors. > > - Set option vector 1 (PowerPC Server Processor Architecture Level) > to "ignore". > > - Set option vector 2 with default or Linux-like options, including a > min-rma-size of 512MB. > > This will cause a CAS reboot and the partition will restart with 512MB > of RMA. Grub will notice the 512MB and not call CAS again. > > (A partition can be configured with only 256MB of memory, which would > mean this request couldn't be satisfied, but PFW refuses to load with > only 256MB of memory, so it's a bit moot. SLOF will run fine with 256MB, > but we will never call CAS under qemu/SLOF because /compatible won't > begin with "IBM".) > > One of the first things Linux does while still running under OpenFirmware > is to call CAS with a much fuller set of options (including asking for > 512MB of memory). This includes a much more restrictive set of PVR values > and processor support levels, and this will induce another reboot. On this > reboot grub will again notice the higher RMA, and not call CAS. We will get > to Linux, Linux will call CAS but because the values are now set for Linux > this will not induce another CAS reboot and we will finally boot. > > On all subsequent boots, everything will be configured with 512MB of RMA > and all the settings Linux likes, so there will be no further CAS reboots. > > (phyp is super sticky with the RMA size - it persists even on cold boots. > So if you've ever booted Linux in a partition, you'll probably never have > grub call CAS. It'll only ever fire the first time a partition loads grub, > or if you deliberately lower the amount of memory your partition has below > 512MB.) > > Signed-off-by: Daniel Axtens Tested-by: Stefan Berger > --- > grub-core/kern/ieee1275/cmain.c | 3 + > grub-core/kern/ieee1275/init.c | 144 ++++++++++++++++++++++++++++++- > include/grub/ieee1275/ieee1275.h | 6 ++ > 3 files changed, 151 insertions(+), 2 deletions(-) > > diff --git a/grub-core/kern/ieee1275/cmain.c b/grub-core/kern/ieee1275/cmain.c > index 20cbbd761ec3..cc98811f4f99 100644 > --- a/grub-core/kern/ieee1275/cmain.c > +++ b/grub-core/kern/ieee1275/cmain.c > @@ -124,6 +124,9 @@ grub_ieee1275_find_options (void) > break; > } > } > + > + if (grub_strncmp (tmp, "IBM,", 4) == 0) > + grub_ieee1275_set_flag (GRUB_IEEE1275_FLAG_CAN_TRY_CAS_FOR_MORE_MEMORY); > } > > if (is_smartfirmware) > diff --git a/grub-core/kern/ieee1275/init.c b/grub-core/kern/ieee1275/init.c > index 4162b5949df4..4586bec939b2 100644 > --- a/grub-core/kern/ieee1275/init.c > +++ b/grub-core/kern/ieee1275/init.c > @@ -240,6 +240,135 @@ heap_init (grub_uint64_t addr, grub_uint64_t len, grub_memory_type_t type, > return 0; > } > > +/* How much memory does OF believe it has? (regardless of whether > + it's accessible or not) */ > +static grub_err_t > +grub_ieee1275_total_mem (grub_uint64_t *total) > +{ > + grub_ieee1275_phandle_t root; > + grub_ieee1275_phandle_t memory; > + grub_uint32_t reg[4]; > + grub_ssize_t reg_size; > + grub_uint32_t address_cells = 1; > + grub_uint32_t size_cells = 1; > + grub_uint64_t size; > + > + /* If we fail to get to the end, report 0. */ > + *total = 0; > + > + /* Determine the format of each entry in `reg'. */ > + grub_ieee1275_finddevice ("/", &root); > + grub_ieee1275_get_integer_property (root, "#address-cells", &address_cells, > + sizeof address_cells, 0); > + grub_ieee1275_get_integer_property (root, "#size-cells", &size_cells, > + sizeof size_cells, 0); > + > + if (size_cells > address_cells) > + address_cells = size_cells; > + > + /* Load `/memory/reg'. */ > + if (grub_ieee1275_finddevice ("/memory", &memory)) > + return grub_error (GRUB_ERR_UNKNOWN_DEVICE, > + "couldn't find /memory node"); > + if (grub_ieee1275_get_integer_property (memory, "reg", reg, > + sizeof reg, ®_size)) > + return grub_error (GRUB_ERR_UNKNOWN_DEVICE, > + "couldn't examine /memory/reg property"); > + if (reg_size < 0 || (grub_size_t) reg_size > sizeof (reg)) > + return grub_error (GRUB_ERR_UNKNOWN_DEVICE, > + "/memory response buffer exceeded"); > + > + if (grub_ieee1275_test_flag (GRUB_IEEE1275_FLAG_BROKEN_ADDRESS_CELLS)) > + { > + address_cells = 1; > + size_cells = 1; > + } > + > + /* Decode only the size */ > + size = reg[address_cells]; > + if (size_cells == 2) > + size = (size << 32) | reg[address_cells + 1]; > + > + *total = size; > + > + return grub_errno; > +} > + > +/* Based on linux - arch/powerpc/kernel/prom_init.c */ > +struct option_vector2 { > + grub_uint8_t byte1; > + grub_uint16_t reserved; > + grub_uint32_t real_base; > + grub_uint32_t real_size; > + grub_uint32_t virt_base; > + grub_uint32_t virt_size; > + grub_uint32_t load_base; > + grub_uint32_t min_rma; > + grub_uint32_t min_load; > + grub_uint8_t min_rma_percent; > + grub_uint8_t max_pft_size; > +} __attribute__((packed)); > + > +struct pvr_entry { > + grub_uint32_t mask; > + grub_uint32_t entry; > +}; > + > +struct cas_vector { > + struct { > + struct pvr_entry terminal; > + } pvr_list; > + grub_uint8_t num_vecs; > + grub_uint8_t vec1_size; > + grub_uint8_t vec1; > + grub_uint8_t vec2_size; > + struct option_vector2 vec2; > +} __attribute__((packed)); > + > +/* Call ibm,client-architecture-support to try to get more RMA. > + We ask for 512MB which should be enough to verify a distro kernel. > + We ignore most errors: if we don't succeed we'll proceed with whatever > + memory we have. */ > +static void > +grub_ieee1275_ibm_cas (void) > +{ > + int rc; > + grub_ieee1275_ihandle_t root; > + struct cas_args { > + struct grub_ieee1275_common_hdr common; > + grub_ieee1275_cell_t method; > + grub_ieee1275_ihandle_t ihandle; > + grub_ieee1275_cell_t cas_addr; > + grub_ieee1275_cell_t result; > + } args; > + struct cas_vector vector = { > + .pvr_list = { { 0x00000000, 0xffffffff } }, /* any processor */ > + .num_vecs = 2 - 1, > + .vec1_size = 0, > + .vec1 = 0x80, /* ignore */ > + .vec2_size = 1 + sizeof(struct option_vector2) - 2, > + .vec2 = { > + 0, 0, -1, -1, -1, -1, -1, 512, -1, 0, 48 > + }, > + }; > + > + INIT_IEEE1275_COMMON (&args.common, "call-method", 3, 2); > + args.method = (grub_ieee1275_cell_t)"ibm,client-architecture-support"; > + rc = grub_ieee1275_open("/", &root); > + if (rc) { > + grub_error (GRUB_ERR_IO, "could not open root when trying to call CAS"); > + return; > + } > + args.ihandle = root; > + args.cas_addr = (grub_ieee1275_cell_t)&vector; > + > + grub_printf("Calling ibm,client-architecture-support..."); > + IEEE1275_CALL_ENTRY_FN (&args); > + grub_printf("done\n"); > + > + grub_ieee1275_close(root); > +} > + > static void > grub_claim_heap (void) > { > @@ -247,11 +376,22 @@ grub_claim_heap (void) > > if (grub_ieee1275_test_flag (GRUB_IEEE1275_FLAG_FORCE_CLAIM)) > { > - heap_init (GRUB_IEEE1275_STATIC_HEAP_START, GRUB_IEEE1275_STATIC_HEAP_LEN, > - 1, &total); > + heap_init (GRUB_IEEE1275_STATIC_HEAP_START, > + GRUB_IEEE1275_STATIC_HEAP_LEN, 1, &total); > return; > } > > + if (grub_ieee1275_test_flag (GRUB_IEEE1275_FLAG_CAN_TRY_CAS_FOR_MORE_MEMORY)) > + { > + grub_uint64_t rma_size; > + grub_err_t err; > + > + err = grub_ieee1275_total_mem (&rma_size); > + /* if we have an error, don't call CAS, just hope for the best */ > + if (!err && rma_size < (512 * 1024 * 1024)) > + grub_ieee1275_ibm_cas(); > + } > + > grub_machine_mmap_iterate (heap_size, &total); > > total = total / 4; > diff --git a/include/grub/ieee1275/ieee1275.h b/include/grub/ieee1275/ieee1275.h > index 73e2f4644751..18c479b668ce 100644 > --- a/include/grub/ieee1275/ieee1275.h > +++ b/include/grub/ieee1275/ieee1275.h > @@ -148,6 +148,12 @@ enum grub_ieee1275_flag > GRUB_IEEE1275_FLAG_CURSORONOFF_ANSI_BROKEN, > > GRUB_IEEE1275_FLAG_RAW_DEVNAMES, > + > + /* On PFW, the first time we boot a Linux partition, we may only get 256MB > + of real memory area, even if the partition has more memory. Set this flag > + if we think we're running under PFW. Then, if this flag is set, and the > + RMA is only 256MB in size, try asking for more with CAS. */ > + GRUB_IEEE1275_FLAG_CAN_TRY_CAS_FOR_MORE_MEMORY, > }; > > extern int EXPORT_FUNC(grub_ieee1275_test_flag) (enum grub_ieee1275_flag flag);