mm, x86/cc, efi: Implement support for unaccepted memory
Subsystems (10)
v14 cover mm, x86/cc, efi: Implement support for unaccepted memory
UEFI Specification version 2.9 introduces the concept of memory
acceptance: some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, requiring memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific for the Virtual Machine platform.
Accepting memory is costly and it makes VMM allocate memory for the accepted guest physical address range. It's better to postpone memory acceptance until memory is needed. It lowers boot time and reduces memory overhead.
The kernel needs to know what memory has been accepted. Firmware communicates this information via memory map: a new memory type -- EFI_UNACCEPTED_MEMORY -- indicates such memory.
Range-based tracking works fine for firmware, but it gets bulky for the kernel: e820 has to be modified on every page acceptance. It leads to table fragmentation, but there's a limited number of entries in the e820 table
Another option is to mark such memory as usable in e820 and track if the range has been accepted in a bitmap. One bit in the bitmap represents 2MiB in the address space: one 4k page is enough to track 64GiB or physical address space.
In the worst-case scenario -- a huge hole in the middle of the address space -- It needs 256MiB to handle 4PiB of the address space.
Any unaccepted memory that is not aligned to 2M gets accepted upfront.
The approach lowers boot time substantially. Boot to shell is ~2.5x faster for 4G TDX VM and ~4x faster for 64G.
TDX-specific code isolated from the core of unaccepted memory support. It supposed to help to plug-in different implementation of unaccepted memory such as SEV-SNP.
-- Fragmentation study --
Vlastimil and Mel were concern about effect of unaccepted memory on fragmentation prevention measures in page allocator. I tried to evaluate it, but it is tricky. As suggested I tried to run multiple parallel kernel builds and follow how often kmem:mm_page_alloc_extfrag gets hit.
See results in the v9 of the patchset[1][2]
[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]
--
The tree can be found here:
https://github.com/intel/tdx.git guest-unaccepted-memory
v14:
- Fix error handling in arch_accept_memory() (Tom);
- Address Borislav's feedback:
+ code restructure;
+ added/adjusted comments;
v13:
- Fix few boot issues discovered by 0day;
- Simplify tdx_accept_memory(): no need in MAP_GPA hypercall;
- Update commit message for the first patch;
- Add Reviewed-bys from Tom and Ard;
v12:
- Re-initialize 'unaccepted_table' variable from decompressor to cover some boot scenarios;
- Add missing memblock_reserve() for the unaccepted memory configuration table (Mika);
- Add efi.unaccepted into efi_tables (Tom);
- Do not build tdx-shared.o for !TDX (Tom);
- Typo fix (Liam)
- Whitespace fix;
- Reviewed-bys from Liam, Tom and Ard;
v11:
- Restructure the code to make it less x86-specific (suggested by Ard):
+ use EFI configuration table instead of zero-page to pass down bitmap;
+ do not imply 1bit == 2M in bitmap;
+ move bulk of the code under driver/firmware/efi;
- The bitmap only covers unaccpeted memory now. All memory that is not covered by the bitmap assumed accepted;
- Reviewed-by from Ard;
v10:
- Restructure code around zones_with_unaccepted_pages static brach to avoid unnecessary function calls (Suggested by Vlastimil);
- Drop mentions of PageUnaccepted();
- Drop patches that add fake unaccepted memory support and sysfs handle to accept memory manually;
- Add Reviewed-by from Vlastimil;
v9:
- Accept memory up to high watermark when kernel runs out of free memory;
- Treat unaccepted memory as unusable in __zone_watermark_unusable_free();
- Per-zone unaccepted memory accounting;
- All pages on unaccepted list are MAX_ORDER now;
- accept_memory=eager in cmdline to pre-accept memory during the boot;
- Implement fake unaccepted memory;
- Sysfs handle to accept memory manually;
- Drop PageUnaccepted();
- Rename unaccepted_pages static key to zones_with_unaccepted_pages;
v8:
- Rewrite core-mm support for unaccepted memory (patch 02/14);
- s/UnacceptedPages/Unaccepted/ in meminfo;
- Drop arch/x86/boot/compressed/compiler.h;
- Fix build errors;
- Adjust commit messages and comments;
- Reviewed-bys from Dave and Borislav;
- Rebased to tip/master.
v7:
- Rework meminfo counter to use PageUnaccepted() and move to generic code;
- Fix range_contains_unaccepted_memory() on machines without unaccepted memory;
- Add Reviewed-by from David;
v6:
- Fix load_unaligned_zeropad() on machine with unaccepted memory;
- Clear PageUnaccepted() on merged pages, leaving it only on head;
- Clarify error handling in allocate_e820();
- Fix build with CONFIG_UNACCEPTED_MEMORY=y, but without TDX;
- Disable kexec at boottime instead of build conflict;
- Rebased to tip/master;
- Spelling fixes;
- Add Reviewed-by from Mike and David;
v5:
- Updates comments and commit messages;
+ Explain options for unaccepted memory handling;
- Expose amount of unaccepted memory in /proc/meminfo
- Adjust check in page_expected_state();
- Fix error code handling in allocate_e820();
- Centralize __pa()/__va() definitions in the boot stub;
- Avoid includes from the main kernel in the boot stub;
- Use an existing hole in boot_param for unaccepted_memory, instead of adding to the end of the structure;
- Extract allocate_unaccepted_memory() form allocate_e820();
- Complain if there's unaccepted memory, but kernel does not support it;
- Fix vmstat counter;
- Split up few preparatory patches;
- Random readability adjustments;
v4:
- PageBuddyUnaccepted() -> PageUnaccepted;
- Use separate page_type, not shared with offline;
- Rework interface between core-mm and arch code;
- Adjust commit messages;
- Ack from Mike;
Kirill A. Shutemov (9):
mm: Add support for unaccepted memory
efi/x86: Get full memory map in allocate_e820()
efi/libstub: Implement support for unaccepted memory x86/boot/compressed: Handle unaccepted memory
efi: Add unaccepted memory support
efi/unaccepted: Avoid load_unaligned_zeropad() stepping into
unaccepted memory
x86/tdx: Make _tdx_hypercall() and __tdx_module_call() available in
boot stub
x86/tdx: Refactor try_accept_one()
x86/tdx: Add unaccepted memory support
arch/x86/Kconfig | 2 + arch/x86/boot/compressed/Makefile | 3 +- arch/x86/boot/compressed/efi.h | 10 + arch/x86/boot/compressed/error.c | 19 ++ arch/x86/boot/compressed/error.h | 1 + arch/x86/boot/compressed/kaslr.c | 40 +++- arch/x86/boot/compressed/mem.c | 83 +++++++ arch/x86/boot/compressed/misc.c | 6 + arch/x86/boot/compressed/misc.h | 10 + arch/x86/boot/compressed/tdx-shared.c | 2 + arch/x86/coco/tdx/Makefile | 2 +- arch/x86/coco/tdx/tdx-shared.c | 71 ++++++ arch/x86/coco/tdx/tdx.c | 102 +------- arch/x86/include/asm/efi.h | 2 + arch/x86/include/asm/shared/tdx.h | 53 +++++ arch/x86/include/asm/tdx.h | 19 -- arch/x86/include/asm/unaccepted_memory.h | 24 ++ arch/x86/platform/efi/efi.c | 3 + drivers/base/node.c | 7 + drivers/firmware/efi/Kconfig | 14 ++ drivers/firmware/efi/Makefile | 1 + drivers/firmware/efi/efi.c | 26 ++ drivers/firmware/efi/libstub/Makefile | 2 + drivers/firmware/efi/libstub/bitmap.c | 41 ++++ drivers/firmware/efi/libstub/efistub.h | 6 + drivers/firmware/efi/libstub/find.c | 43 ++++ .../firmware/efi/libstub/unaccepted_memory.c | 222 ++++++++++++++++++ drivers/firmware/efi/libstub/x86-stub.c | 39 +-- drivers/firmware/efi/unaccepted_memory.c | 147 ++++++++++++ fs/proc/meminfo.c | 5 + include/linux/efi.h | 13 +- include/linux/mm.h | 19 ++ include/linux/mmzone.h | 8 + mm/memblock.c | 9 + mm/mm_init.c | 7 + mm/page_alloc.c | 173 ++++++++++++++ mm/vmstat.c | 3 + 37 files changed, 1089 insertions(+), 148 deletions(-) create mode 100644 arch/x86/boot/compressed/mem.c create mode 100644 arch/x86/boot/compressed/tdx-shared.c create mode 100644 arch/x86/coco/tdx/tdx-shared.c create mode 100644 arch/x86/include/asm/unaccepted_memory.h create mode 100644 drivers/firmware/efi/libstub/bitmap.c create mode 100644 drivers/firmware/efi/libstub/find.c create mode 100644 drivers/firmware/efi/libstub/unaccepted_memory.c create mode 100644 drivers/firmware/efi/unaccepted_memory.c
Signature · 1 line
2.39.3
PATCH v14 1/9 mm: Add support for unaccepted memory
UEFI Specification version 2.9 introduces the concept of memory acceptance. Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, require memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific to the Virtual Machine platform.
There are several ways kernel can deal with unaccepted memory:
1. Accept all the memory during the boot. It is easy to implement and it doesn't have runtime cost once the system is booted. The downside is very long boot time.
Accept can be parallelized to multiple CPUs to keep it manageable (i.e. via DEFERRED_STRUCT_PAGE_INIT), but it tends to saturate memory bandwidth and does not scale beyond the point.
2. Accept a block of memory on the first use. It requires more infrastructure and changes in page allocator to make it work, but it provides good boot time.
On-demand memory accept means latency spikes every time kernel steps onto a new memory block. The spikes will go away once workload data set size gets stabilized or all memory gets accepted.
3. Accept all memory in background. Introduce a thread (or multiple) that gets memory accepted proactively. It will minimize time the system experience latency spikes on memory allocation while keeping low boot time.
This approach cannot function on its own. It is an extension of #2:
background memory acceptance requires functional scheduler, but the page allocator may need to tap into unaccepted memory before that.
The downside of the approach is that these threads also steal CPU cycles and memory bandwidth from the user's workload and may hurt user experience.
Implement #1 and #2 for now. #2 is the default. Some workloads may want to use #1 with accept_memory=eager in kernel command line. #3 can be implemented later based on user's demands.
Support of unaccepted memory requires a few changes in core-mm code:
- memblock accepts memory on allocation. It serves early boot memory allocations and doesn't limit them to pre-accepted pool of memory.
- page allocator accepts memory on the first allocation of the page. When kernel runs out of accepted memory, it accepts memory until the high watermark is reached. It helps to minimize fragmentation.
EFI code will provide two helpers if the platform supports unaccepted memory:
- accept_memory() makes a range of physical addresses accepted.
- range_contains_unaccepted_memory() checks anything within the range of physical addresses requires acceptance.
--- drivers/base/node.c | 7 ++ fs/proc/meminfo.c | 5 ++ include/linux/mm.h | 19 +++++ include/linux/mmzone.h | 8 ++ mm/memblock.c | 9 +++ mm/mm_init.c | 7 ++ mm/page_alloc.c | 173 +++++++++++++++++++++++++++++++++++++++++ mm/vmstat.c | 3 + 8 files changed, 231 insertions(+)
drivers/base/node.c +7 -0
@@ -448,6 +448,9 @@ static ssize_t node_read_meminfo(struct device *dev,"Node %d ShmemPmdMapped: %8lu kB\n" "Node %d FileHugePages: %8lu kB\n" "Node %d FilePmdMapped: %8lu kB\n" +#endif +#ifdef CONFIG_UNACCEPTED_MEMORY +"Node %d Unaccepted: %8lu kB\n" #endif , nid, K(node_page_state(pgdat, NR_FILE_DIRTY)),
@@ -477,6 +480,10 @@ static ssize_t node_read_meminfo(struct device *dev, nid,K (node_page_state (pgdat,NR_SHMEM_PMDMAPPED )), nid,K (node_page_state (pgdat,NR_FILE_THPS )), nid, K(node_page_state(pgdat, NR_FILE_PMDMAPPED)) +#endif +#ifdef CONFIG_UNACCEPTED_MEMORY + , + nid,K (sum_zone_node_page_state (nid,NR_UNACCEPTED )) #endif ); len +=hugetlb_report_node_meminfo (buf, len, nid);
fs/proc/meminfo.c +5 -0
@@ -168,6 +168,11 @@ static int meminfo_proc_show(struct seq_file *m, void *v) global_zone_page_state(NR_FREE_CMA_PAGES));#endif +#ifdef CONFIG_UNACCEPTED_MEMORY +show_val_kb (m,"Unaccepted: " , +global_zone_page_state (NR_UNACCEPTED )); +#endif +hugetlb_report_meminfo (m);arch_report_meminfo (m);
include/linux/mm.h +19 -0
@@ -3816,4 +3816,23 @@ madvise_set_anon_name(struct mm_struct *mm, unsigned long start, }#endif +#ifdef CONFIG_UNACCEPTED_MEMORY + +bool range_contains_unaccepted_memory (phys_addr_t start,phys_addr_t end); +void accept_memory (phys_addr_t start,phys_addr_t end); + +#else + +static inline bool range_contains_unaccepted_memory (phys_addr_t start, +phys_addr_t end) +{ +return false; +} + +static inline void accept_memory (phys_addr_t start,phys_addr_t end) +{ +} + +#endif +#endif /* _LINUX_MM_H */
include/linux/mmzone.h +8 -0
@@ -143,6 +143,9 @@ enum zone_stat_item { NR_ZSPAGES, /* allocated in zsmalloc */ #endif NR_FREE_CMA_PAGES, +#ifdef CONFIG_UNACCEPTED_MEMORY + NR_UNACCEPTED, +#endif NR_VM_ZONE_STAT_ITEMS }; enum node_stat_item {
@@ -910,6 +913,11 @@ struct zone {/* free areas of different sizes */ struct free_area free_area[MAX_ORDER +1 ]; +#ifdef CONFIG_UNACCEPTED_MEMORY +/* Pages to be accepted. All pages on the list are MAX_ORDER */ +struct list_head unaccepted_pages; +#endif +/* zone flags, see below */ unsigned long flags;
mm/memblock.c +9 -0
@@ -1436,6 +1436,15 @@ phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size,*/ kmemleak_alloc_phys (found, size,0 ); +/* +* Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, +* require memory to be accepted before it can be used by the +* guest. +* +* Accept the memory of the allocated buffer. +*/ +accept_memory (found, found + size); +return found; }
mm/mm_init.c +7 -0
@@ -1375,6 +1375,10 @@ static void __meminit zone_init_free_lists(struct zone *zone)INIT_LIST_HEAD (&zone->free_area[order].free_list[t]); zone->free_area[order].nr_free =0 ; } + +#ifdef CONFIG_UNACCEPTED_MEMORY +INIT_LIST_HEAD (&zone->unaccepted_pages); +#endif } void __meminit init_currently_empty_zone(struct zone *zone,
@@ -1960,6 +1964,9 @@ static void __init deferred_free_range(unsigned long pfn,return ; } +/* Accept chunks smaller than MAX_ORDER upfront */ +accept_memory (PFN_PHYS (pfn),PFN_PHYS (pfn + nr_pages)); +for (i =0 ; i < nr_pages; i++, page++, pfn++) {if (pageblock_aligned (pfn))set_pageblock_migratetype (page,MIGRATE_MOVABLE );
mm/page_alloc.c +173 -0
206 lines not on this page · raw message
mm/vmstat.c +3 -0
@@ -1180,6 +1180,9 @@ const char * const vmstat_text[] = {"nr_zspages" , #endif"nr_free_cma" , +#ifdef CONFIG_UNACCEPTED_MEMORY +"nr_unaccepted" , +#endif /* enum numa_stat_item counters */ #ifdef CONFIG_NUMA
Signature · 1 line
2.39.3
[tip: x86/cc] mm: Add support for unaccepted memory
The following commit has been merged into the x86/cc branch of tip:
The following commit has been merged into the x86/cc branch of tip:
Commit-ID: dcdfdd40fa82b6704d2841938e5c8ec3051eb0d6 Gitweb: https://git.kernel.org/tip/dcdfdd40fa82b6704d2841938e5c8ec3051eb0d6 Author: Kirill A. Shutemov <[email protected]> AuthorDate: Tue, 06 Jun 2023 17:26:29 +03:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 06 Jun 2023 16:38:22 +02:00
mm: Add support for unaccepted memory
UEFI Specification version 2.9 introduces the concept of memory acceptance. Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, require memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific to the Virtual Machine platform.
There are several ways the kernel can deal with unaccepted memory:
1. Accept all the memory during boot. It is easy to implement and it doesn't have runtime cost once the system is booted. The downside is very long boot time.
Accept can be parallelized to multiple CPUs to keep it manageable (i.e. via DEFERRED_STRUCT_PAGE_INIT), but it tends to saturate memory bandwidth and does not scale beyond the point.
2. Accept a block of memory on the first use. It requires more infrastructure and changes in page allocator to make it work, but it provides good boot time.
On-demand memory accept means latency spikes every time kernel steps onto a new memory block. The spikes will go away once workload data set size gets stabilized or all memory gets accepted.
3. Accept all memory in background. Introduce a thread (or multiple) that gets memory accepted proactively. It will minimize time the system experience latency spikes on memory allocation while keeping low boot time.
This approach cannot function on its own. It is an extension of #2:
background memory acceptance requires functional scheduler, but the page allocator may need to tap into unaccepted memory before that.
The downside of the approach is that these threads also steal CPU cycles and memory bandwidth from the user's workload and may hurt user experience.
Implement #1 and #2 for now. #2 is the default. Some workloads may want to use #1 with accept_memory=eager in kernel command line. #3 can be implemented later based on user's demands.
Support of unaccepted memory requires a few changes in core-mm code:
- memblock accepts memory on allocation. It serves early boot memory allocations and doesn't limit them to pre-accepted pool of memory.
- page allocator accepts memory on the first allocation of the page. When kernel runs out of accepted memory, it accepts memory until the high watermark is reached. It helps to minimize fragmentation.
EFI code will provide two helpers if the platform supports unaccepted memory:
- accept_memory() makes a range of physical addresses accepted.
- range_contains_unaccepted_memory() checks anything within the range of physical addresses requires acceptance.
--- drivers/base/node.c | 7 ++- fs/proc/meminfo.c | 5 +- include/linux/mm.h | 19 ++++- include/linux/mmzone.h | 8 ++- mm/memblock.c | 9 ++- mm/mm_init.c | 7 ++- mm/page_alloc.c | 173 ++++++++++++++++++++++++++++++++++++++++- mm/vmstat.c | 3 +- 8 files changed, 231 insertions(+)
drivers/base/node.c +7 -0
@@ -449,6 +449,9 @@ static ssize_t node_read_meminfo(struct device *dev,"Node %d FileHugePages: %8lu kB\n" "Node %d FilePmdMapped: %8lu kB\n"#endif +#ifdef CONFIG_UNACCEPTED_MEMORY +"Node %d Unaccepted: %8lu kB\n" +#endif , nid,K (node_page_state (pgdat,NR_FILE_DIRTY )), nid, K(node_page_state(pgdat, NR_WRITEBACK)),
@@ -478,6 +481,10 @@ static ssize_t node_read_meminfo(struct device *dev, nid,K (node_page_state (pgdat,NR_FILE_THPS )), nid, K(node_page_state(pgdat, NR_FILE_PMDMAPPED))#endif +#ifdef CONFIG_UNACCEPTED_MEMORY + , + nid,K (sum_zone_node_page_state (nid,NR_UNACCEPTED )) +#endif ); len +=hugetlb_report_node_meminfo (buf, len, nid);return len;
fs/proc/meminfo.c +5 -0
@@ -168,6 +168,11 @@ static int meminfo_proc_show(struct seq_file *m, void *v) global_zone_page_state(NR_FREE_CMA_PAGES));#endif +#ifdef CONFIG_UNACCEPTED_MEMORY +show_val_kb (m,"Unaccepted: " , +global_zone_page_state (NR_UNACCEPTED )); +#endif +hugetlb_report_meminfo (m);arch_report_meminfo (m);
include/linux/mm.h +19 -0
@@ -3816,4 +3816,23 @@ madvise_set_anon_name(struct mm_struct *mm, unsigned long start, }#endif +#ifdef CONFIG_UNACCEPTED_MEMORY + +bool range_contains_unaccepted_memory (phys_addr_t start,phys_addr_t end); +void accept_memory (phys_addr_t start,phys_addr_t end); + +#else + +static inline bool range_contains_unaccepted_memory (phys_addr_t start, +phys_addr_t end) +{ +return false; +} + +static inline void accept_memory (phys_addr_t start,phys_addr_t end) +{ +} + +#endif +#endif /* _LINUX_MM_H */
include/linux/mmzone.h +8 -0
@@ -143,6 +143,9 @@ enum zone_stat_item { NR_ZSPAGES, /* allocated in zsmalloc */ #endif NR_FREE_CMA_PAGES, +#ifdef CONFIG_UNACCEPTED_MEMORY + NR_UNACCEPTED, +#endif NR_VM_ZONE_STAT_ITEMS }; enum node_stat_item {
@@ -910,6 +913,11 @@ struct zone {/* free areas of different sizes */ struct free_area free_area[MAX_ORDER +1 ]; +#ifdef CONFIG_UNACCEPTED_MEMORY +/* Pages to be accepted. All pages on the list are MAX_ORDER */ +struct list_head unaccepted_pages; +#endif +/* zone flags, see below */ unsigned long flags;
mm/memblock.c +9 -0
@@ -1436,6 +1436,15 @@ done:*/ kmemleak_alloc_phys (found, size,0 ); +/* +* Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, +* require memory to be accepted before it can be used by the +* guest. +* +* Accept the memory of the allocated buffer. +*/ +accept_memory (found, found + size); +return found; }
mm/mm_init.c +7 -0
@@ -1375,6 +1375,10 @@ static void __meminit zone_init_free_lists(struct zone *zone)INIT_LIST_HEAD (&zone->free_area[order].free_list[t]); zone->free_area[order].nr_free =0 ; } + +#ifdef CONFIG_UNACCEPTED_MEMORY +INIT_LIST_HEAD (&zone->unaccepted_pages); +#endif } void __meminit init_currently_empty_zone(struct zone *zone,
@@ -1960,6 +1964,9 @@ static void __init deferred_free_range(unsigned long pfn,return ; } +/* Accept chunks smaller than MAX_ORDER upfront */ +accept_memory (PFN_PHYS (pfn),PFN_PHYS (pfn + nr_pages)); +for (i =0 ; i < nr_pages; i++, page++, pfn++) {if (pageblock_aligned (pfn))set_pageblock_migratetype (page,MIGRATE_MOVABLE );
mm/page_alloc.c +173 -0
206 lines not on this page · raw message
mm/vmstat.c +3 -0
@@ -1180,6 +1180,9 @@ const char * const vmstat_text[] = {"nr_zspages" , #endif"nr_free_cma" , +#ifdef CONFIG_UNACCEPTED_MEMORY +"nr_unaccepted" , +#endif /* enum numa_stat_item counters */ #ifdef CONFIG_NUMA
PATCH v14 2/9 efi/x86: Get full memory map in allocate_e820()
In reply to Kirill A. ShutemovCurrently allocate_e820() is only interested in the size of map and size of memory descriptor to determine how many e820 entries the kernel needs.
UEFI Specification version 2.9 introduces a new memory type -- unaccepted memory. To track unaccepted memory kernel needs to allocate a bitmap. The size of the bitmap is dependent on the maximum physical address present in the system. A full memory map is required to find the maximum address.
Modify allocate_e820() to get a full memory map.
--- drivers/firmware/efi/libstub/x86-stub.c | 26 +++++++++++-------------- 1 file changed, 11 insertions(+), 15 deletions(-)
drivers/firmware/efi/libstub/x86-stub.c +11 -15
@@ -681,28 +681,24 @@ static efi_status_t allocate_e820(struct boot_params *params,struct setup_data **e820ext, u32 *e820ext_size) { - unsigned long map_size, desc_size, map_key; + struct efi_boot_memmap *map;efi_status_t status; -__u32 nr_desc, desc_version; +__u32 nr_desc; -/* Only need the size of the mem map and size of each mem descriptor */ - map_size =0 ; - status =efi_bs_call (get_memory_map, &map_size,NULL , &map_key, - &desc_size, &desc_version); -if (status !=EFI_BUFFER_TOO_SMALL ) -return (status !=EFI_SUCCESS ) ? status :EFI_UNSUPPORTED ; + status =efi_get_memory_map (&map, false); +if (status !=EFI_SUCCESS ) +return status; - nr_desc = map_size / desc_size +EFI_MMAP_NR_SLACK_SLOTS ; - -if (nr_desc >ARRAY_SIZE (params->e820_table)) { -u32 nr_e820ext = nr_desc -ARRAY_SIZE (params->e820_table); + nr_desc = map->map_size / map->desc_size; +if (nr_desc >ARRAY_SIZE (params->e820_table) -EFI_MMAP_NR_SLACK_SLOTS ) { +u32 nr_e820ext = nr_desc -ARRAY_SIZE (params->e820_table) + +EFI_MMAP_NR_SLACK_SLOTS ; status =alloc_e820ext (nr_e820ext, e820ext, e820ext_size); -if (status !=EFI_SUCCESS ) -return status; } -return EFI_SUCCESS ; +efi_bs_call (free_pool, map); +return status; } struct exit_boot_struct {
Signature · 1 line
2.39.3
[tip: x86/cc] efi/x86: Get full memory map in allocate_e820()
The following commit has been merged into the x86/cc branch of tip:
The following commit has been merged into the x86/cc branch of tip:
Commit-ID: 2e9f46ee1599be8a50a5366eb3ef4a4b5acff0b7 Gitweb: https://git.kernel.org/tip/2e9f46ee1599be8a50a5366eb3ef4a4b5acff0b7 Author: Kirill A. Shutemov <[email protected]> AuthorDate: Tue, 06 Jun 2023 17:26:30 +03:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 06 Jun 2023 16:45:14 +02:00
efi/x86: Get full memory map in allocate_e820()
Currently allocate_e820() is only interested in the size of map and size of memory descriptor to determine how many e820 entries the kernel needs.
UEFI Specification version 2.9 introduces a new memory type -- unaccepted memory. To track unaccepted memory, the kernel needs to allocate a bitmap. The size of the bitmap is dependent on the maximum physical address present in the system. A full memory map is required to find the maximum address.
Modify allocate_e820() to get a full memory map.
--- drivers/firmware/efi/libstub/x86-stub.c | 26 ++++++++++-------------- 1 file changed, 11 insertions(+), 15 deletions(-)
drivers/firmware/efi/libstub/x86-stub.c +11 -15
@@ -681,28 +681,24 @@ static efi_status_t allocate_e820(struct boot_params *params,struct setup_data **e820ext, u32 *e820ext_size) { - unsigned long map_size, desc_size, map_key; + struct efi_boot_memmap *map;efi_status_t status; -__u32 nr_desc, desc_version; +__u32 nr_desc; -/* Only need the size of the mem map and size of each mem descriptor */ - map_size =0 ; - status =efi_bs_call (get_memory_map, &map_size,NULL , &map_key, - &desc_size, &desc_version); -if (status !=EFI_BUFFER_TOO_SMALL ) -return (status !=EFI_SUCCESS ) ? status :EFI_UNSUPPORTED ; - - nr_desc = map_size / desc_size +EFI_MMAP_NR_SLACK_SLOTS ; + status =efi_get_memory_map (&map, false); +if (status !=EFI_SUCCESS ) +return status; -if (nr_desc >ARRAY_SIZE (params->e820_table)) { -u32 nr_e820ext = nr_desc -ARRAY_SIZE (params->e820_table); + nr_desc = map->map_size / map->desc_size; +if (nr_desc >ARRAY_SIZE (params->e820_table) -EFI_MMAP_NR_SLACK_SLOTS ) { +u32 nr_e820ext = nr_desc -ARRAY_SIZE (params->e820_table) + +EFI_MMAP_NR_SLACK_SLOTS ; status =alloc_e820ext (nr_e820ext, e820ext, e820ext_size); -if (status !=EFI_SUCCESS ) -return status; } -return EFI_SUCCESS ; +efi_bs_call (free_pool, map); +return status; } struct exit_boot_struct {
PATCH v14 3/9 efi/libstub: Implement support for unaccepted memory
In reply to Kirill A. ShutemovUEFI Specification version 2.9 introduces the concept of memory
acceptance: Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, requiring memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific for the Virtual Machine platform.
Accepting memory is costly and it makes VMM allocate memory for the accepted guest physical address range. It's better to postpone memory acceptance until memory is needed. It lowers boot time and reduces memory overhead.
The kernel needs to know what memory has been accepted. Firmware communicates this information via memory map: a new memory type -- EFI_UNACCEPTED_MEMORY -- indicates such memory.
Range-based tracking works fine for firmware, but it gets bulky for the kernel: e820 (or whatever the arch uses) has to be modified on every page acceptance. It leads to table fragmentation and there's a limited number of entries in the e820 table.
Another option is to mark such memory as usable in e820 and track if the range has been accepted in a bitmap. One bit in the bitmap represents a naturally aligned power-2-sized region of address space -- unit.
For x86, unit size is 2MiB: 4k of the bitmap is enough to track 64GiB or physical address space.
In the worst-case scenario -- a huge hole in the middle of the address space -- It needs 256MiB to handle 4PiB of the address space.
Any unaccepted memory that is not aligned to unit_size gets accepted upfront.
The bitmap is allocated and constructed in the EFI stub and passed down to the kernel via EFI configuration table. allocate_e820() allocates the bitmap if unaccepted memory is present, according to the size of unaccepted region.
--- arch/x86/boot/compressed/Makefile | 1 + arch/x86/boot/compressed/mem.c | 9 + arch/x86/include/asm/efi.h | 2 + drivers/firmware/efi/Kconfig | 14 ++ drivers/firmware/efi/efi.c | 1 + drivers/firmware/efi/libstub/Makefile | 2 + drivers/firmware/efi/libstub/bitmap.c | 41 ++++ drivers/firmware/efi/libstub/efistub.h | 6 + drivers/firmware/efi/libstub/find.c | 43 ++++ .../firmware/efi/libstub/unaccepted_memory.c | 222 ++++++++++++++++++ drivers/firmware/efi/libstub/x86-stub.c | 13 + include/linux/efi.h | 12 +- 12 files changed, 365 insertions(+), 1 deletion(-) create mode 100644 arch/x86/boot/compressed/mem.c create mode 100644 drivers/firmware/efi/libstub/bitmap.c create mode 100644 drivers/firmware/efi/libstub/find.c create mode 100644 drivers/firmware/efi/libstub/unaccepted_memory.c
arch/x86/boot/compressed/Makefile +1 -0
@@ -107,6 +107,7 @@ endif vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o vmlinux-objs-$(CONFIG_INTEL_TDX_GUEST) += $(obj)/tdx.o $(obj)/tdcall.o +vmlinux-objs-$(CONFIG_UNACCEPTED_MEMORY) += $(obj)/mem.o vmlinux-objs-$(CONFIG_EFI) += $(obj)/efi.o vmlinux-objs-$(CONFIG_EFI_MIXED) += $(obj)/efi_mixed.o
arch/x86/boot/compressed/mem.c new file +9 -0
@@ -0,0 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include "error.h" + +void arch_accept_memory (phys_addr_t start,phys_addr_t end) +{ +/* Platform-specific memory-acceptance call goes here */ +error ("Cannot accept memory" ); +}
arch/x86/include/asm/efi.h +2 -0
@@ -31,6 +31,8 @@ extern unsigned long efi_mixed_mode_stack_pa;#define ARCH_EFI_IRQ_FLAGS_MASK X86_EFLAGS_IF +#define EFI_UNACCEPTED_UNIT_SIZE PMD_SIZE +/* * The EFI services are called through variadic functions in many cases. These * functions are implemented in assembler and support only a fixed number of
drivers/firmware/efi/Kconfig +14 -0
@@ -269,6 +269,20 @@ config EFI_COCO_SECRET virt/coco/efi_secret module to access the secrets, which in turn allows userspace programs to access the injected secrets. +config UNACCEPTED_MEMORY + bool + depends on EFI_STUB + help + Some Virtual Machine platforms, such as Intel TDX, require + some memory to be "accepted" by the guest before it can be used. + This mechanism helps prevent malicious hosts from making changes + to guest memory. + + UEFI specification v2.9 introduced EFI_UNACCEPTED_MEMORY memory type. + + This option adds support for unaccepted memory and makes such memory + usable by the kernel. + config EFI_EMBEDDED_FIRMWARE bool select CRYPTO_LIB_SHA256
drivers/firmware/efi/efi.c +1 -0
@@ -843,6 +843,7 @@ static __initdata char memory_type_name[][13] = {"MMIO Port" ,"PAL Code" ,"Persistent" , + "Unaccepted", }; char * __init efi_md_typeattr_format(char *buf, size_t size,
drivers/firmware/efi/libstub/Makefile +2 -0
@@ -96,6 +96,8 @@ CFLAGS_arm32-stub.o := -DTEXT_OFFSET=$(TEXT_OFFSET) zboot-obj-$(CONFIG_RISCV) := lib-clz_ctz.o lib-ashldi3.o lib-$(CONFIG_EFI_ZBOOT) += zboot.o $(zboot-obj-y) +lib-$(CONFIG_UNACCEPTED_MEMORY) += unaccepted_memory.o bitmap.o find.o + extra-y := $(lib-y) lib-y := $(patsubst %.o,%.stub.o,$(lib-y))
drivers/firmware/efi/libstub/bitmap.c new file +41 -0
41 lines not on this page · raw message
drivers/firmware/efi/libstub/efistub.h +6 -0
@@ -1133,4 +1133,10 @@ const u8 *__efi_get_smbios_string(const struct efi_smbios_record *record,void efi_remap_image (unsigned long image_base,unsigned alloc_size,unsigned long code_size); +efi_status_t allocate_unaccepted_bitmap (__u32 nr_desc, +struct efi_boot_memmap *map); +void process_unaccepted_memory (u64 start,u64 end); +void accept_memory (phys_addr_t start,phys_addr_t end); +void arch_accept_memory (phys_addr_t start,phys_addr_t end); +#endif
drivers/firmware/efi/libstub/find.c new file +43 -0
43 lines not on this page · raw message
drivers/firmware/efi/libstub/unaccepted_memory.c new file +222 -0
222 lines not on this page · raw message
drivers/firmware/efi/libstub/x86-stub.c +13 -0
@@ -613,6 +613,16 @@ setup_e820(struct boot_params *params, struct setup_data *e820ext, u32 e820ext_s e820_type =E820_TYPE_PMEM ;break ; +case EFI_UNACCEPTED_MEMORY : +if (!IS_ENABLED (CONFIG_UNACCEPTED_MEMORY )) { +efi_warn_once ( +"The system has unaccepted memory, but kernel does not support it\nConsider enabling CONFIG_UNACCEPTED_MEMORY\n" ); +continue ; + } + e820_type =E820_TYPE_RAM ; +process_unaccepted_memory (d->phys_addr, + d->phys_addr +PAGE_SIZE * d->num_pages); +break ;default :continue ; }
@@ -697,6 +707,9 @@ static efi_status_t allocate_e820(struct boot_params *params, status =alloc_e820ext (nr_e820ext, e820ext, e820ext_size); } +if (IS_ENABLED (CONFIG_UNACCEPTED_MEMORY ) && status ==EFI_SUCCESS ) + status =allocate_unaccepted_bitmap (nr_desc, map); +efi_bs_call (free_pool, map);return status; }
include/linux/efi.h +11 -1
@@ -108,7 +108,8 @@ typedef struct {#define EFI_MEMORY_MAPPED_IO_PORT_SPACE 12#define EFI_PAL_CODE 13#define EFI_PERSISTENT_MEMORY 14 -#define EFI_MAX_MEMORY_TYPE 15 +#define EFI_UNACCEPTED_MEMORY 15 +#define EFI_MAX_MEMORY_TYPE 16/* Attribute values: */ #define EFI_MEMORY_UC ((u64)0x0000000000000001ULL)/* uncached */
@@ -417,6 +418,7 @@ void efi_native_runtime_setup(void);#define LINUX_EFI_MOK_VARIABLE_TABLE_GUID EFI_GUID(0xc451ed2b, 0x9694, 0x45d3, 0xba, 0xba, 0xed, 0x9f, 0x89, 0x88, 0xa3, 0x89)#define LINUX_EFI_COCO_SECRET_AREA_GUID EFI_GUID(0xadf956ad, 0xe98c, 0x484c, 0xae, 0x11, 0xb5, 0x1c, 0x7d, 0x33, 0x64, 0x47)#define LINUX_EFI_BOOT_MEMMAP_GUID EFI_GUID(0x800f683f, 0xd08b, 0x423a, 0xa2, 0x93, 0x96, 0x5c, 0x3c, 0x6f, 0xe2, 0xb4) +#define LINUX_EFI_UNACCEPTED_MEM_TABLE_GUID EFI_GUID(0xd5d1de3c, 0x105c, 0x44f9, 0x9e, 0xa9, 0xbc, 0xef, 0x98, 0x12, 0x00, 0x31)#define RISCV_EFI_BOOT_PROTOCOL_GUID EFI_GUID(0xccd15fec, 0x6f73, 0x4eec, 0x83, 0x95, 0x3e, 0x69, 0xe4, 0xb9, 0x40, 0xbf)
@@ -534,6 +536,14 @@ struct efi_boot_memmap {efi_memory_desc_t map[]; }; +struct efi_unaccepted_memory { +u32 version; +u32 unit_size; +u64 phys_base; +u64 size; +unsigned long bitmap[]; +}; +/* * Architecture independent structure for describing a memory map for the * benefit of efi_memmap_init_early(), and for passing context between
Signature · 1 line
2.39.3
[tip: x86/cc] efi/libstub: Implement support for unaccepted memory
The following commit has been merged into the x86/cc branch of tip:
The following commit has been merged into the x86/cc branch of tip:
Commit-ID: 745e3ed85f71a6382a239b03d9278a8025f2beae Gitweb: https://git.kernel.org/tip/745e3ed85f71a6382a239b03d9278a8025f2beae Author: Kirill A. Shutemov <[email protected]> AuthorDate: Tue, 06 Jun 2023 17:26:31 +03:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 06 Jun 2023 16:58:23 +02:00
efi/libstub: Implement support for unaccepted memory
UEFI Specification version 2.9 introduces the concept of memory
acceptance: Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, requiring memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific for the Virtual Machine platform.
Accepting memory is costly and it makes VMM allocate memory for the accepted guest physical address range. It's better to postpone memory acceptance until memory is needed. It lowers boot time and reduces memory overhead.
The kernel needs to know what memory has been accepted. Firmware communicates this information via memory map: a new memory type -- EFI_UNACCEPTED_MEMORY -- indicates such memory.
Range-based tracking works fine for firmware, but it gets bulky for the kernel: e820 (or whatever the arch uses) has to be modified on every page acceptance. It leads to table fragmentation and there's a limited number of entries in the e820 table.
Another option is to mark such memory as usable in e820 and track if the range has been accepted in a bitmap. One bit in the bitmap represents a naturally aligned power-2-sized region of address space -- unit.
For x86, unit size is 2MiB: 4k of the bitmap is enough to track 64GiB or physical address space.
In the worst-case scenario -- a huge hole in the middle of the address space -- It needs 256MiB to handle 4PiB of the address space.
Any unaccepted memory that is not aligned to unit_size gets accepted upfront.
The bitmap is allocated and constructed in the EFI stub and passed down to the kernel via EFI configuration table. allocate_e820() allocates the bitmap if unaccepted memory is present, according to the size of unaccepted region.
--- arch/x86/boot/compressed/Makefile | 1 +- arch/x86/boot/compressed/mem.c | 9 +- arch/x86/include/asm/efi.h | 2 +- drivers/firmware/efi/Kconfig | 14 +- drivers/firmware/efi/efi.c | 1 +- drivers/firmware/efi/libstub/Makefile | 2 +- drivers/firmware/efi/libstub/bitmap.c | 41 +++- drivers/firmware/efi/libstub/efistub.h | 6 +- drivers/firmware/efi/libstub/find.c | 43 +++- drivers/firmware/efi/libstub/unaccepted_memory.c | 222 ++++++++++++++- drivers/firmware/efi/libstub/x86-stub.c | 13 +- include/linux/efi.h | 12 +- 12 files changed, 365 insertions(+), 1 deletion(-) create mode 100644 arch/x86/boot/compressed/mem.c create mode 100644 drivers/firmware/efi/libstub/bitmap.c create mode 100644 drivers/firmware/efi/libstub/find.c create mode 100644 drivers/firmware/efi/libstub/unaccepted_memory.c
arch/x86/boot/compressed/Makefile +1 -0
@@ -107,6 +107,7 @@ endif vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o vmlinux-objs-$(CONFIG_INTEL_TDX_GUEST) += $(obj)/tdx.o $(obj)/tdcall.o +vmlinux-objs-$(CONFIG_UNACCEPTED_MEMORY) += $(obj)/mem.o vmlinux-objs-$(CONFIG_EFI) += $(obj)/efi.o vmlinux-objs-$(CONFIG_EFI_MIXED) += $(obj)/efi_mixed.o
arch/x86/boot/compressed/mem.c new file +9 -0
@@ -0,0 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include "error.h" + +void arch_accept_memory (phys_addr_t start,phys_addr_t end) +{ +/* Platform-specific memory-acceptance call goes here */ +error ("Cannot accept memory" ); +}
arch/x86/include/asm/efi.h +2 -0
@@ -31,6 +31,8 @@ extern unsigned long efi_mixed_mode_stack_pa;#define ARCH_EFI_IRQ_FLAGS_MASK X86_EFLAGS_IF +#define EFI_UNACCEPTED_UNIT_SIZE PMD_SIZE +/* * The EFI services are called through variadic functions in many cases. These * functions are implemented in assembler and support only a fixed number of
drivers/firmware/efi/Kconfig +14 -0
@@ -269,6 +269,20 @@ config EFI_COCO_SECRET virt/coco/efi_secret module to access the secrets, which in turn allows userspace programs to access the injected secrets. +config UNACCEPTED_MEMORY + bool + depends on EFI_STUB + help + Some Virtual Machine platforms, such as Intel TDX, require + some memory to be "accepted" by the guest before it can be used. + This mechanism helps prevent malicious hosts from making changes + to guest memory. + + UEFI specification v2.9 introduced EFI_UNACCEPTED_MEMORY memory type. + + This option adds support for unaccepted memory and makes such memory + usable by the kernel. + config EFI_EMBEDDED_FIRMWARE bool select CRYPTO_LIB_SHA256
drivers/firmware/efi/efi.c +1 -0
@@ -843,6 +843,7 @@ static __initdata char memory_type_name[][13] = {"MMIO Port" ,"PAL Code" ,"Persistent" , + "Unaccepted", }; char * __init efi_md_typeattr_format(char *buf, size_t size,
drivers/firmware/efi/libstub/Makefile +2 -0
@@ -96,6 +96,8 @@ CFLAGS_arm32-stub.o := -DTEXT_OFFSET=$(TEXT_OFFSET) zboot-obj-$(CONFIG_RISCV) := lib-clz_ctz.o lib-ashldi3.o lib-$(CONFIG_EFI_ZBOOT) += zboot.o $(zboot-obj-y) +lib-$(CONFIG_UNACCEPTED_MEMORY) += unaccepted_memory.o bitmap.o find.o + extra-y := $(lib-y) lib-y := $(patsubst %.o,%.stub.o,$(lib-y))
drivers/firmware/efi/libstub/bitmap.c new file +41 -0
41 lines not on this page · raw message
drivers/firmware/efi/libstub/efistub.h +6 -0
@@ -1136,4 +1136,10 @@ void efi_remap_image(unsigned long image_base, unsigned alloc_size, asmlinkage efi_status_t __efiapiefi_zboot_entry (efi_handle_t handle,efi_system_table_t *systab); +efi_status_t allocate_unaccepted_bitmap (__u32 nr_desc, +struct efi_boot_memmap *map); +void process_unaccepted_memory (u64 start,u64 end); +void accept_memory (phys_addr_t start,phys_addr_t end); +void arch_accept_memory (phys_addr_t start,phys_addr_t end); +#endif
drivers/firmware/efi/libstub/find.c new file +43 -0
43 lines not on this page · raw message
drivers/firmware/efi/libstub/unaccepted_memory.c new file +222 -0
222 lines not on this page · raw message
drivers/firmware/efi/libstub/x86-stub.c +13 -0
@@ -613,6 +613,16 @@ setup_e820(struct boot_params *params, struct setup_data *e820ext, u32 e820ext_s e820_type =E820_TYPE_PMEM ;break ; +case EFI_UNACCEPTED_MEMORY : +if (!IS_ENABLED (CONFIG_UNACCEPTED_MEMORY )) { +efi_warn_once ( +"The system has unaccepted memory, but kernel does not support it\nConsider enabling CONFIG_UNACCEPTED_MEMORY\n" ); +continue ; + } + e820_type =E820_TYPE_RAM ; +process_unaccepted_memory (d->phys_addr, + d->phys_addr +PAGE_SIZE * d->num_pages); +break ;default :continue ; }
@@ -697,6 +707,9 @@ static efi_status_t allocate_e820(struct boot_params *params, status =alloc_e820ext (nr_e820ext, e820ext, e820ext_size); } +if (IS_ENABLED (CONFIG_UNACCEPTED_MEMORY ) && status ==EFI_SUCCESS ) + status =allocate_unaccepted_bitmap (nr_desc, map); +efi_bs_call (free_pool, map);return status; }
include/linux/efi.h +11 -1
@@ -108,7 +108,8 @@ typedef struct {#define EFI_MEMORY_MAPPED_IO_PORT_SPACE 12#define EFI_PAL_CODE 13#define EFI_PERSISTENT_MEMORY 14 -#define EFI_MAX_MEMORY_TYPE 15 +#define EFI_UNACCEPTED_MEMORY 15 +#define EFI_MAX_MEMORY_TYPE 16/* Attribute values: */ #define EFI_MEMORY_UC ((u64)0x0000000000000001ULL)/* uncached */
@@ -417,6 +418,7 @@ void efi_native_runtime_setup(void);#define LINUX_EFI_MOK_VARIABLE_TABLE_GUID EFI_GUID(0xc451ed2b, 0x9694, 0x45d3, 0xba, 0xba, 0xed, 0x9f, 0x89, 0x88, 0xa3, 0x89)#define LINUX_EFI_COCO_SECRET_AREA_GUID EFI_GUID(0xadf956ad, 0xe98c, 0x484c, 0xae, 0x11, 0xb5, 0x1c, 0x7d, 0x33, 0x64, 0x47)#define LINUX_EFI_BOOT_MEMMAP_GUID EFI_GUID(0x800f683f, 0xd08b, 0x423a, 0xa2, 0x93, 0x96, 0x5c, 0x3c, 0x6f, 0xe2, 0xb4) +#define LINUX_EFI_UNACCEPTED_MEM_TABLE_GUID EFI_GUID(0xd5d1de3c, 0x105c, 0x44f9, 0x9e, 0xa9, 0xbc, 0xef, 0x98, 0x12, 0x00, 0x31)#define RISCV_EFI_BOOT_PROTOCOL_GUID EFI_GUID(0xccd15fec, 0x6f73, 0x4eec, 0x83, 0x95, 0x3e, 0x69, 0xe4, 0xb9, 0x40, 0xbf)
@@ -534,6 +536,14 @@ struct efi_boot_memmap {efi_memory_desc_t map[]; }; +struct efi_unaccepted_memory { +u32 version; +u32 unit_size; +u64 phys_base; +u64 size; +unsigned long bitmap[]; +}; +/* * Architecture independent structure for describing a memory map for the * benefit of efi_memmap_init_early(), and for passing context between
PATCH v14 4/9 x86/boot/compressed: Handle unaccepted memory
In reply to Kirill A. ShutemovThe firmware will pre-accept the memory used to run the stub. But, the stub is responsible for accepting the memory into which it decompresses the main kernel. Accept memory just before decompression starts.
The stub is also responsible for choosing a physical address in which to place the decompressed kernel image. The KASLR mechanism will randomize this physical address. Since the accepted memory region is relatively small, KASLR would be quite ineffective if it only used the pre-accepted area (EFI_CONVENTIONAL_MEMORY). Ensure that KASLR randomizes among the entire physical address space by also including EFI_UNACCEPTED_MEMORY.
--- arch/x86/boot/compressed/efi.h | 10 ++++++++ arch/x86/boot/compressed/kaslr.c | 40 +++++++++++++++++++++---------- arch/x86/boot/compressed/mem.c | 41 ++++++++++++++++++++++++++++++++ arch/x86/boot/compressed/misc.c | 6 +++++ arch/x86/boot/compressed/misc.h | 10 ++++++++ 5 files changed, 95 insertions(+), 12 deletions(-)
arch/x86/boot/compressed/efi.h +10 -0
@@ -16,6 +16,7 @@ typedef guid_t efi_guid_t __aligned(__alignof__(u32));#define ACPI_TABLE_GUID EFI_GUID(0xeb9d2d30, 0x2d88, 0x11d3, 0x9a, 0x16, 0x00, 0x90, 0x27, 0x3f, 0xc1, 0x4d)#define ACPI_20_TABLE_GUID EFI_GUID(0x8868e871, 0xe4f1, 0x11d3, 0xbc, 0x22, 0x00, 0x80, 0xc7, 0x3c, 0x88, 0x81)#define EFI_CC_BLOB_GUID EFI_GUID(0x067b1f5f, 0xcf26, 0x44c5, 0x85, 0x54, 0x93, 0xd7, 0x77, 0x91, 0x2d, 0x42) +#define LINUX_EFI_UNACCEPTED_MEM_TABLE_GUID EFI_GUID(0xd5d1de3c, 0x105c, 0x44f9, 0x9e, 0xa9, 0xbc, 0xef, 0x98, 0x12, 0x00, 0x31)#define EFI32_LOADER_SIGNATURE "EL32"#define EFI64_LOADER_SIGNATURE "EL64"
@@ -32,6 +33,7 @@ typedef struct { } efi_table_hdr_t;#define EFI_CONVENTIONAL_MEMORY 7 +#define EFI_UNACCEPTED_MEMORY 15#define EFI_MEMORY_MORE_RELIABLE \ ((u64)0x0000000000010000ULL)/* higher reliability */
@@ -104,6 +106,14 @@ struct efi_setup_data {u64 reserved[8 ]; }; +struct efi_unaccepted_memory { +u32 version; +u32 unit_size; +u64 phys_base; +u64 size; +unsigned long bitmap[]; +}; +static inline int efi_guidcmp (efi_guid_t left,efi_guid_t right) {return memcmp (&left, &right,sizeof (efi_guid_t));
arch/x86/boot/compressed/kaslr.c +28 -12
@@ -672,6 +672,33 @@ static bool process_mem_region(struct mem_vector *region, }#ifdef CONFIG_EFI + +/* +* Only EFI_CONVENTIONAL_MEMORY and EFI_UNACCEPTED_MEMORY (if supported) are +* guaranteed to be free. +* +* Pick free memory more conservatively than the EFI spec allows: according to +* the spec, EFI_BOOT_SERVICES_{CODE|DATA} are also free memory and thus +* available to place the kernel image into, but in practice there's firmware +* where using that memory leads to crashes. Buggy vendor EFI code registers +* for an event that triggers on SetVirtualAddressMap(). The handler assumes +* that EFI_BOOT_SERVICES_DATA memory has not been touched by loader yet, which +* is probably true for Windows. +* +* Preserve EFI_BOOT_SERVICES_* regions until after SetVirtualAddressMap(). +*/ +static inline bool memory_type_is_free (efi_memory_desc_t *md) +{ +if (md->type ==EFI_CONVENTIONAL_MEMORY ) +return true; + +if (IS_ENABLED (CONFIG_UNACCEPTED_MEMORY ) && + md->type ==EFI_UNACCEPTED_MEMORY ) +return true; + +return false; +} +/* * Returns true if we processed the EFI memmap, which we prefer over the E820 * table if it is available.
@@ -716,18 +743,7 @@ process_efi_entries(unsigned long minimum, unsigned long image_size)for (i =0 ; i < nr_desc; i++) { md =efi_early_memdesc_ptr (pmap, e->efi_memdesc_size, i); -/* -* Here we are more conservative in picking free memory than -* the EFI spec allows: -* -* According to the spec, EFI_BOOT_SERVICES_{CODE|DATA} are also -* free memory and thus available to place the kernel image into, -* but in practice there's firmware where using that memory leads -* to crashes. -* -* Only EFI_CONVENTIONAL_MEMORY is guaranteed to be free. -*/ -if (md->type !=EFI_CONVENTIONAL_MEMORY ) +if (!memory_type_is_free (md))continue ; if (efi_soft_reserve_enabled() &&
arch/x86/boot/compressed/mem.c +41 -0
50 lines not on this page · raw message
arch/x86/boot/compressed/misc.c +6 -0
@@ -455,6 +455,12 @@ asmlinkage __visible void *extract_kernel(void *rmode, memptr heap,#endif debug_putstr ("\nDecompressing Linux... " ); + +if (init_unaccepted_memory ()) { +debug_putstr ("Accepting memory... " ); +accept_memory (__pa (output),__pa (output) + needed_size); + } +__decompress (input_data, input_len,NULL ,NULL , output, output_len,NULL , error); entry_offset =parse_elf (output);
arch/x86/boot/compressed/misc.h +10 -0
@@ -247,4 +247,14 @@ static inline unsigned long efi_find_vendor_table(struct boot_params *bp, }#endif /* CONFIG_EFI */ +#ifdef CONFIG_UNACCEPTED_MEMORY +bool init_unaccepted_memory (void ); +#else +static inline bool init_unaccepted_memory (void ) {return false; } +#endif + +/* Defined in EFI stub */ +extern struct efi_unaccepted_memory *unaccepted_table; +void accept_memory (phys_addr_t start,phys_addr_t end); +#endif /* BOOT_COMPRESSED_MISC_H */
Signature · 1 line
2.39.3
[tip: x86/cc] x86/boot/compressed: Handle unaccepted memory
The following commit has been merged into the x86/cc branch of tip:
The following commit has been merged into the x86/cc branch of tip:
Commit-ID: 3fd1239a783522e7158a1f141fabc7b3b5dc84c6 Gitweb: https://git.kernel.org/tip/3fd1239a783522e7158a1f141fabc7b3b5dc84c6 Author: Kirill A. Shutemov <[email protected]> AuthorDate: Tue, 06 Jun 2023 17:26:32 +03:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 06 Jun 2023 17:17:24 +02:00
x86/boot/compressed: Handle unaccepted memory
The firmware will pre-accept the memory used to run the stub. But, the stub is responsible for accepting the memory into which it decompresses the main kernel. Accept memory just before decompression starts.
The stub is also responsible for choosing a physical address in which to place the decompressed kernel image. The KASLR mechanism will randomize this physical address. Since the accepted memory region is relatively small, KASLR would be quite ineffective if it only used the pre-accepted area (EFI_CONVENTIONAL_MEMORY). Ensure that KASLR randomizes among the entire physical address space by also including EFI_UNACCEPTED_MEMORY.
--- arch/x86/boot/compressed/efi.h | 10 ++++++++- arch/x86/boot/compressed/kaslr.c | 40 +++++++++++++++++++++--------- arch/x86/boot/compressed/mem.c | 41 +++++++++++++++++++++++++++++++- arch/x86/boot/compressed/misc.c | 6 +++++- arch/x86/boot/compressed/misc.h | 10 ++++++++- 5 files changed, 95 insertions(+), 12 deletions(-)
arch/x86/boot/compressed/efi.h +10 -0
@@ -16,6 +16,7 @@ typedef guid_t efi_guid_t __aligned(__alignof__(u32));#define ACPI_TABLE_GUID EFI_GUID(0xeb9d2d30, 0x2d88, 0x11d3, 0x9a, 0x16, 0x00, 0x90, 0x27, 0x3f, 0xc1, 0x4d)#define ACPI_20_TABLE_GUID EFI_GUID(0x8868e871, 0xe4f1, 0x11d3, 0xbc, 0x22, 0x00, 0x80, 0xc7, 0x3c, 0x88, 0x81)#define EFI_CC_BLOB_GUID EFI_GUID(0x067b1f5f, 0xcf26, 0x44c5, 0x85, 0x54, 0x93, 0xd7, 0x77, 0x91, 0x2d, 0x42) +#define LINUX_EFI_UNACCEPTED_MEM_TABLE_GUID EFI_GUID(0xd5d1de3c, 0x105c, 0x44f9, 0x9e, 0xa9, 0xbc, 0xef, 0x98, 0x12, 0x00, 0x31)#define EFI32_LOADER_SIGNATURE "EL32"#define EFI64_LOADER_SIGNATURE "EL64"
@@ -32,6 +33,7 @@ typedef struct { } efi_table_hdr_t;#define EFI_CONVENTIONAL_MEMORY 7 +#define EFI_UNACCEPTED_MEMORY 15#define EFI_MEMORY_MORE_RELIABLE \ ((u64)0x0000000000010000ULL)/* higher reliability */
@@ -104,6 +106,14 @@ struct efi_setup_data {u64 reserved[8 ]; }; +struct efi_unaccepted_memory { +u32 version; +u32 unit_size; +u64 phys_base; +u64 size; +unsigned long bitmap[]; +}; +static inline int efi_guidcmp (efi_guid_t left,efi_guid_t right) {return memcmp (&left, &right,sizeof (efi_guid_t));
arch/x86/boot/compressed/kaslr.c +28 -12
@@ -672,6 +672,33 @@ static bool process_mem_region(struct mem_vector *region, }#ifdef CONFIG_EFI + +/* +* Only EFI_CONVENTIONAL_MEMORY and EFI_UNACCEPTED_MEMORY (if supported) are +* guaranteed to be free. +* +* Pick free memory more conservatively than the EFI spec allows: according to +* the spec, EFI_BOOT_SERVICES_{CODE|DATA} are also free memory and thus +* available to place the kernel image into, but in practice there's firmware +* where using that memory leads to crashes. Buggy vendor EFI code registers +* for an event that triggers on SetVirtualAddressMap(). The handler assumes +* that EFI_BOOT_SERVICES_DATA memory has not been touched by loader yet, which +* is probably true for Windows. +* +* Preserve EFI_BOOT_SERVICES_* regions until after SetVirtualAddressMap(). +*/ +static inline bool memory_type_is_free (efi_memory_desc_t *md) +{ +if (md->type ==EFI_CONVENTIONAL_MEMORY ) +return true; + +if (IS_ENABLED (CONFIG_UNACCEPTED_MEMORY ) && + md->type ==EFI_UNACCEPTED_MEMORY ) +return true; + +return false; +} +/* * Returns true if we processed the EFI memmap, which we prefer over the E820 * table if it is available.
@@ -716,18 +743,7 @@ process_efi_entries(unsigned long minimum, unsigned long image_size)for (i =0 ; i < nr_desc; i++) { md =efi_early_memdesc_ptr (pmap, e->efi_memdesc_size, i); -/* -* Here we are more conservative in picking free memory than -* the EFI spec allows: -* -* According to the spec, EFI_BOOT_SERVICES_{CODE|DATA} are also -* free memory and thus available to place the kernel image into, -* but in practice there's firmware where using that memory leads -* to crashes. -* -* Only EFI_CONVENTIONAL_MEMORY is guaranteed to be free. -*/ -if (md->type !=EFI_CONVENTIONAL_MEMORY ) +if (!memory_type_is_free (md))continue ; if (efi_soft_reserve_enabled() &&
arch/x86/boot/compressed/mem.c +41 -0
50 lines not on this page · raw message
arch/x86/boot/compressed/misc.c +6 -0
@@ -455,6 +455,12 @@ asmlinkage __visible void *extract_kernel(void *rmode, memptr heap,#endif debug_putstr ("\nDecompressing Linux... " ); + +if (init_unaccepted_memory ()) { +debug_putstr ("Accepting memory... " ); +accept_memory (__pa (output),__pa (output) + needed_size); + } +__decompress (input_data, input_len,NULL ,NULL , output, output_len,NULL , error); entry_offset =parse_elf (output);
arch/x86/boot/compressed/misc.h +10 -0
@@ -247,4 +247,14 @@ static inline unsigned long efi_find_vendor_table(struct boot_params *bp, }#endif /* CONFIG_EFI */ +#ifdef CONFIG_UNACCEPTED_MEMORY +bool init_unaccepted_memory (void ); +#else +static inline bool init_unaccepted_memory (void ) {return false; } +#endif + +/* Defined in EFI stub */ +extern struct efi_unaccepted_memory *unaccepted_table; +void accept_memory (phys_addr_t start,phys_addr_t end); +#endif /* BOOT_COMPRESSED_MISC_H */
PATCH v14 5/9 efi: Add unaccepted memory support
In reply to Kirill A. Shutemovefi_config_parse_tables() reserves memory that holds unaccepted memory configuration table so it won't be reused by page allocator.
Core-mm requires few helpers to support unaccepted memory:
- accept_memory() checks the range of addresses against the bitmap and accept memory if needed.
- range_contains_unaccepted_memory() checks if anything within the range requires acceptance.
Architectural code has to provide efi_get_unaccepted_table() that returns pointer to the unaccepted memory configuration table.
arch_accept_memory() handles arch-specific part of memory acceptance.
--- arch/x86/platform/efi/efi.c | 3 + drivers/firmware/efi/Makefile | 1 + drivers/firmware/efi/efi.c | 25 +++++ drivers/firmware/efi/unaccepted_memory.c | 112 +++++++++++++++++++++++ include/linux/efi.h | 1 + 5 files changed, 142 insertions(+) create mode 100644 drivers/firmware/efi/unaccepted_memory.c
arch/x86/platform/efi/efi.c +3 -0
@@ -96,6 +96,9 @@ static const unsigned long * const efi_tables[] = {#ifdef CONFIG_EFI_COCO_SECRET &efi.coco_secret,#endif +#ifdef CONFIG_UNACCEPTED_MEMORY + &efi.unaccepted, +#endif };u64 efi_setup;/* efi setup_data physical address */
drivers/firmware/efi/Makefile +1 -0
@@ -41,3 +41,4 @@ obj-$(CONFIG_EFI_CAPSULE_LOADER) += capsule-loader.o obj-$(CONFIG_EFI_EARLYCON) += earlycon.o obj-$(CONFIG_UEFI_CPER_ARM) += cper-arm.o obj-$(CONFIG_UEFI_CPER_X86) += cper-x86.o +obj-$(CONFIG_UNACCEPTED_MEMORY) += unaccepted_memory.o
drivers/firmware/efi/efi.c +25 -0
@@ -50,6 +50,9 @@ struct efi __read_mostly efi = {#ifdef CONFIG_EFI_COCO_SECRET .coco_secret = EFI_INVALID_TABLE_ADDR,#endif +#ifdef CONFIG_UNACCEPTED_MEMORY + .unaccepted = EFI_INVALID_TABLE_ADDR, +#endif };EXPORT_SYMBOL (efi);
@@ -605,6 +608,9 @@ static const efi_config_table_type_t common_tables[] __initconst = { #ifdef CONFIG_EFI_COCO_SECRET {LINUX_EFI_COCO_SECRET_AREA_GUID, &efi.coco_secret, "CocoSecret" }, #endif +#ifdef CONFIG_UNACCEPTED_MEMORY + {LINUX_EFI_UNACCEPTED_MEM_TABLE_GUID, &efi.unaccepted, "Unaccepted" }, +#endif #ifdef CONFIG_EFI_GENERIC_STUB {LINUX_EFI_SCREEN_INFO_TABLE_GUID, &screen_info_table }, #endif
@@ -759,6 +765,25 @@ int __init efi_config_parse_tables(const efi_config_table_t *config_tables, } } +if (IS_ENABLED (CONFIG_UNACCEPTED_MEMORY ) && + efi.unaccepted !=EFI_INVALID_TABLE_ADDR ) { +struct efi_unaccepted_memory *unaccepted; + + unaccepted =early_memremap (efi.unaccepted,sizeof (*unaccepted)); +if (unaccepted) { +unsigned long size; + +if (unaccepted->version ==1 ) { + size =sizeof (*unaccepted) + unaccepted->size; +memblock_reserve (efi.unaccepted, size); + }else { + efi.unaccepted =EFI_INVALID_TABLE_ADDR ; + } + +early_memunmap (unaccepted,sizeof (*unaccepted)); + } + } +return 0 ; }
drivers/firmware/efi/unaccepted_memory.c new file +112 -0
112 lines not on this page · raw message
include/linux/efi.h +1 -0
@@ -646,6 +646,7 @@ extern struct efi {unsigned long tpm_final_log;/* TPM2 Final Events Log table */ unsigned long mokvar_table;/* MOK variable config table */ unsigned long coco_secret;/* Confidential computing secret table */ +unsigned long unaccepted;/* Unaccepted memory table */ efi_get_time_t *get_time;efi_set_time_t *set_time;
Signature · 1 line
2.39.3
[tip: x86/cc] efi: Add unaccepted memory support
The following commit has been merged into the x86/cc branch of tip:
The following commit has been merged into the x86/cc branch of tip:
Commit-ID: 2053bc57f36763febced0b5cd91821698bcf6b3d Gitweb: https://git.kernel.org/tip/2053bc57f36763febced0b5cd91821698bcf6b3d Author: Kirill A. Shutemov <[email protected]> AuthorDate: Tue, 06 Jun 2023 17:26:33 +03:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 06 Jun 2023 17:22:20 +02:00
efi: Add unaccepted memory support
efi_config_parse_tables() reserves memory that holds unaccepted memory configuration table so it won't be reused by page allocator.
Core-mm requires few helpers to support unaccepted memory:
- accept_memory() checks the range of addresses against the bitmap and accept memory if needed.
- range_contains_unaccepted_memory() checks if anything within the range requires acceptance.
Architectural code has to provide efi_get_unaccepted_table() that returns pointer to the unaccepted memory configuration table.
arch_accept_memory() handles arch-specific part of memory acceptance.
--- arch/x86/platform/efi/efi.c | 3 +- drivers/firmware/efi/Makefile | 1 +- drivers/firmware/efi/efi.c | 25 +++++- drivers/firmware/efi/unaccepted_memory.c | 112 ++++++++++++++++++++++- include/linux/efi.h | 1 +- 5 files changed, 142 insertions(+) create mode 100644 drivers/firmware/efi/unaccepted_memory.c
arch/x86/platform/efi/efi.c +3 -0
@@ -96,6 +96,9 @@ static const unsigned long * const efi_tables[] = {#ifdef CONFIG_EFI_COCO_SECRET &efi.coco_secret,#endif +#ifdef CONFIG_UNACCEPTED_MEMORY + &efi.unaccepted, +#endif };u64 efi_setup;/* efi setup_data physical address */
drivers/firmware/efi/Makefile +1 -0
@@ -41,3 +41,4 @@ obj-$(CONFIG_EFI_CAPSULE_LOADER) += capsule-loader.o obj-$(CONFIG_EFI_EARLYCON) += earlycon.o obj-$(CONFIG_UEFI_CPER_ARM) += cper-arm.o obj-$(CONFIG_UEFI_CPER_X86) += cper-x86.o +obj-$(CONFIG_UNACCEPTED_MEMORY) += unaccepted_memory.o
drivers/firmware/efi/efi.c +25 -0
@@ -50,6 +50,9 @@ struct efi __read_mostly efi = {#ifdef CONFIG_EFI_COCO_SECRET .coco_secret = EFI_INVALID_TABLE_ADDR,#endif +#ifdef CONFIG_UNACCEPTED_MEMORY + .unaccepted = EFI_INVALID_TABLE_ADDR, +#endif };EXPORT_SYMBOL (efi);
@@ -605,6 +608,9 @@ static const efi_config_table_type_t common_tables[] __initconst = { #ifdef CONFIG_EFI_COCO_SECRET {LINUX_EFI_COCO_SECRET_AREA_GUID, &efi.coco_secret, "CocoSecret" }, #endif +#ifdef CONFIG_UNACCEPTED_MEMORY + {LINUX_EFI_UNACCEPTED_MEM_TABLE_GUID, &efi.unaccepted, "Unaccepted" }, +#endif #ifdef CONFIG_EFI_GENERIC_STUB {LINUX_EFI_SCREEN_INFO_TABLE_GUID, &screen_info_table }, #endif
@@ -759,6 +765,25 @@ int __init efi_config_parse_tables(const efi_config_table_t *config_tables, } } +if (IS_ENABLED (CONFIG_UNACCEPTED_MEMORY ) && + efi.unaccepted !=EFI_INVALID_TABLE_ADDR ) { +struct efi_unaccepted_memory *unaccepted; + + unaccepted =early_memremap (efi.unaccepted,sizeof (*unaccepted)); +if (unaccepted) { +unsigned long size; + +if (unaccepted->version ==1 ) { + size =sizeof (*unaccepted) + unaccepted->size; +memblock_reserve (efi.unaccepted, size); + }else { + efi.unaccepted =EFI_INVALID_TABLE_ADDR ; + } + +early_memunmap (unaccepted,sizeof (*unaccepted)); + } + } +return 0 ; }
drivers/firmware/efi/unaccepted_memory.c new file +112 -0
112 lines not on this page · raw message
include/linux/efi.h +1 -0
@@ -646,6 +646,7 @@ extern struct efi {unsigned long tpm_final_log;/* TPM2 Final Events Log table */ unsigned long mokvar_table;/* MOK variable config table */ unsigned long coco_secret;/* Confidential computing secret table */ +unsigned long unaccepted;/* Unaccepted memory table */ efi_get_time_t *get_time;efi_set_time_t *set_time;
Re: [PATCHv14 5/9] efi: Add unaccepted memory support
In reply to Kirill A. ShutemovQuoted 11 lines · Kirill A. Shutemov
efi_config_parse_tables()reserves memory that holds unaccepted memory configuration table so it won't be reused by page allocator.Core-mm requires few helpers to support unaccepted memory:
-
accept_memory()checks the range of addresses against the bitmap and accept memory if needed.-
range_contains_unaccepted_memory()checks if anything within the range requires acceptance.
Quoted 8 lines
Architectural code has to provide
efi_get_unaccepted_table()that returns pointer to the unaccepted memory configuration table.
arch_accept_memory()handles arch-specific part of memory acceptance.Signed-off-by Kirill A. Shutemov <[email protected]>Reviewed-by Ard Biesheuvel <[email protected]>Reviewed-by Tom Lendacky <[email protected]>
By and large, this looks ok from the page allocator perspective as the checks for unaccepted are mostly after watermark checks. However, if you look in the initial fast path, you'll see this
/*
* Forbid the first pass from falling back to types that fragment
* memory until all local zones are considered.
*/
alloc_flags |= alloc_flags_nofragment(ac.preferred_zoneref->zone, gfp);While checking watermarks should be fine from a functional perspective and the fast paths are unaffected, there is a risk of premature fragmentation until all memory has been accepted. Meeting watermarks does not necessarily mean that fragmentation is avoided as pageblocks can get mixed while still meeting watermarks. This will be tricky to detect in most cases so it may be worth considering augmenting the watermark checks with a check in this block for unaccepted memory
/*
* It's possible on a UMA machine to get through all zones that are
* fragmented. If avoiding fragmentation, reset and try again. */
if (no_fallback) {
alloc_flags &= ~ALLOC_NOFRAGMENT;
goto retry;
}I think the watermark checks will still be needed because hypothetically if 100% of allocations were MIGRATE_UNMOVABLE then there never would be a request that fragments memory and memory would not be accepted.
It's not a blocker to the series, simply a suggestion for follow-on work.
Signature · 2 lines
Mel Gorman SUSE Labs
Re: [PATCHv14 5/9] efi: Add unaccepted memory support
Quoted 31 lines · Mel Gorman
Quoted 11 lines · Kirill A. Shutemov
efi_config_parse_tables()reserves memory that holds unaccepted memory configuration table so it won't be reused by page allocator.Core-mm requires few helpers to support unaccepted memory:
-
accept_memory()checks the range of addresses against the bitmap and accept memory if needed.-
range_contains_unaccepted_memory()checks if anything within the range requires acceptance.Quoted 8 lines
Architectural code has to provide
efi_get_unaccepted_table()that returns pointer to the unaccepted memory configuration table.
arch_accept_memory()handles arch-specific part of memory acceptance.Signed-off-by Kirill A. Shutemov <[email protected]>Reviewed-by Ard Biesheuvel <[email protected]>Reviewed-by Tom Lendacky <[email protected]>By and large, this looks ok from the page allocator perspective as the checks for unaccepted are mostly after watermark checks. However, if you look in the initial fast path, you'll see this
/* * Forbid the first pass from falling back to types that fragment * memory until all local zones are considered. */ alloc_flags |= alloc_flags_nofragment(ac.preferred_zoneref->zone, gfp);
Quoted 5 lines
While checking watermarks should be fine from a functional perspective and the fast paths are unaffected, there is a risk of premature fragmentation until all memory has been accepted. Meeting watermarks does not necessarily mean that fragmentation is avoided as pageblocks can get mixed while still meeting watermarks.
Could you elaborate on this scenario?
Current code checks the watermark, if it is met, try rmqueue().
If rmqueue() fails anyway, try to accept more pages and retry the zone if it is successful.
I'm not sure how we can get to the 'if (no_fallback) {' case with any unaccepted memory in the allowed zones.
I see that there's preferred_zoneref and spread_dirty_pages cases, but unaccepted memory seems change nothing for them.
Hm?
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCHv14 5/9] efi: Add unaccepted memory support
Quoted 40 lines · Kirill A. Shutemov
Quoted 31 lines · Mel Gorman
Quoted 11 lines · Kirill A. Shutemov
efi_config_parse_tables()reserves memory that holds unaccepted memory configuration table so it won't be reused by page allocator.Core-mm requires few helpers to support unaccepted memory:
-
accept_memory()checks the range of addresses against the bitmap and accept memory if needed.-
range_contains_unaccepted_memory()checks if anything within the range requires acceptance.Quoted 8 lines
Architectural code has to provide
efi_get_unaccepted_table()that returns pointer to the unaccepted memory configuration table.
arch_accept_memory()handles arch-specific part of memory acceptance.Signed-off-by Kirill A. Shutemov <[email protected]>Reviewed-by Ard Biesheuvel <[email protected]>Reviewed-by Tom Lendacky <[email protected]>By and large, this looks ok from the page allocator perspective as the checks for unaccepted are mostly after watermark checks. However, if you look in the initial fast path, you'll see this
/* * Forbid the first pass from falling back to types that fragment * memory until all local zones are considered. */ alloc_flags |= alloc_flags_nofragment(ac.preferred_zoneref->zone, gfp);Quoted 5 lines
While checking watermarks should be fine from a functional perspective and the fast paths are unaffected, there is a risk of premature fragmentation until all memory has been accepted. Meeting watermarks does not necessarily mean that fragmentation is avoided as pageblocks can get mixed while still meeting watermarks.
Could you elaborate on this scenario?
Quoted 8 lines
Current code checks the watermark, if it is met, try
rmqueue().If
rmqueue()fails anyway, try to accept more pages and retry the zone if it is successful.I'm not sure how we can get to the 'if (
no_fallback) {' case with any unaccepted memory in the allowed zones.
Lets take an extreme example and assume that the low watermark is lower than 2MB (one pageblock). Just before the watermark is reached (free count between 1MB and 2MB), it is unlikely that all free pages are within pageblocks of the same migratetype (e.g. MIGRATE_MOVABLE). If there is an allocation near the watermark of a different type (e.g. MIGRATE_UNMOVABLE) then the page allocation could fallback to a different pageblock and now it is mixed. It's a condition that is only obvious if you are explicitly checking for it via tracepoints. This can happen in the normal case, but unaccepted memory makes it worse because the "pageblock mixing" could have been avoided if the "no_fallback" case accepted at least one new pageblock instead of mixing pageblocks.
That is an extreme example but the same logic applies when the free count is at or near MIGRATE_TYPES*pageblock_nr_pages as it is not guaranteed that the pageblocks with free pages are a migratetype that matches the allocation request.
Hence, it may be more robust from a fragmentation perspective if ALLOC_NOFRAGMENT requests accept memory if it is available and retries before clearing ALLOC_NOFRAGMENT and mixing pageblocks before the watermarks are reached.
Quoted 3 lines
I see that there's
preferred_zonerefandspread_dirty_pagescases, but unaccepted memory seems change nothing for them.
preferred_zoneref is about premature zone exhaustion and spread_dirty_pages is about avoiding premature stalls on a node/zone due to an imbalance in the number of pages waiting for writeback to complete. There is an arguement to be made that they also should accept memory but it's less clear how much of a problem this is. Both are very obvious when they "fail" and likely are covered by the existing watermark checks. Premature pageblock mixing is more subtle as the final impact (root cause of a premature THP allocation failure) is harder to detect.
Signature · 2 lines
Mel Gorman SUSE Labs
Re: [PATCHv14 5/9] efi: Add unaccepted memory support
In reply to Kirill A. ShutemovQuoted 92 lines · Kirill A. Shutemov
efi_config_parse_tables()reserves memory that holds unaccepted memory configuration table so it won't be reused by page allocator.Core-mm requires few helpers to support unaccepted memory:
-
accept_memory()checks the range of addresses against the bitmap and accept memory if needed.-
range_contains_unaccepted_memory()checks if anything within the range requires acceptance.Architectural code has to provide
efi_get_unaccepted_table()that returns pointer to the unaccepted memory configuration table.
arch_accept_memory()handles arch-specific part of memory acceptance.Signed-off-by Kirill A. Shutemov <[email protected]>Reviewed-by Ard Biesheuvel <[email protected]>Reviewed-by Tom Lendacky <[email protected]>--- arch/x86/platform/efi/efi.c | 3 + drivers/firmware/efi/Makefile | 1 + drivers/firmware/efi/efi.c | 25 +++++ drivers/firmware/efi/unaccepted_memory.c | 112 +++++++++++++++++++++++ include/linux/efi.h | 1 + 5 files changed, 142 insertions(+) create mode 100644 drivers/firmware/efi/unaccepted_memory.cdrivers/firmware/efi/unaccepted_memory.c new file +58
-0@@ -0,0 +1,112 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include <linux/efi.h> +#include <linux/memblock.h> +#include <linux/spinlock.h> +#include <asm/unaccepted_memory.h> + +/* Protects unaccepted memory bitmap */ +static DEFINE_SPINLOCK (unaccepted_memory_lock); + +/* +* accept_memory() -- Consult bitmap and accept the memory if needed. +* +* Only memory that is explicitly marked as unaccepted in the bitmap requires +* an action. All the remaining memory is implicitly accepted and doesn't need +* acceptance. +* +* No need to accept: +* - anything if the system has no unaccepted table; +* - memory that is below phys_base; +* - memory that is above the memory that addressable by the bitmap; +*/ +void accept_memory (phys_addr_t start,phys_addr_t end) +{ +struct efi_unaccepted_memory *unaccepted; +unsigned long range_start, range_end; +unsigned long flags; +u64 unit_size; + + unaccepted =efi_get_unaccepted_table (); +if (!unaccepted) +return ; + + unit_size = unaccepted->unit_size; + +/* +* Only care for the part of the range that is represented +* in the bitmap. +*/ +if (start < unaccepted->phys_base) + start = unaccepted->phys_base; +if (end < unaccepted->phys_base) +return ; + +/* Translate to offsets from the beginning of the bitmap */ + start -= unaccepted->phys_base; + end -= unaccepted->phys_base; + +/* Make sure not to overrun the bitmap */ +if (end > unaccepted->size * unit_size *BITS_PER_BYTE ) + end = unaccepted->size * unit_size *BITS_PER_BYTE ; + + range_start = start / unit_size; + +spin_lock_irqsave (&unaccepted_memory_lock, flags); +for_each_set_bitrange_from (range_start, range_end, unaccepted->bitmap, +DIV_ROUND_UP (end, unit_size)) { +unsigned long phys_start, phys_end;
drivers/firmware/efi/unaccepted_memory.c line 68
Quoted 10 lines
drivers/firmware/efi/unaccepted_memory.c new file +10
-0+unsigned long len = range_end - range_start; + + phys_start = range_start * unit_size + unaccepted->phys_base; + phys_end = range_end * unit_size + unaccepted->phys_base; + +arch_accept_memory (phys_start, phys_end); +bitmap_clear (unaccepted->bitmap, range_start, len); + } +spin_unlock_irqrestore (&unaccepted_memory_lock, flags); +}
While testing SNP guests running today's tip/master (ef19bc9dddc3) I ran into what seems to be fairly significant lock contention due to the unaccepted_memory_lock spinlock above, which results in a constant stream of soft-lockups until the workload gets all its memory accepted/faulted in if the guest has around 16+ vCPUs.
I've included the guest dmesg traces I was seeing below.
In this case I was running a 32 vCPU guest with 200GB of memory running on a 256 thread EPYC (Milan) system, and can trigger the above situation fairly reliably by running the following workload in a freshly-booted guests:
stress --vm 32 --vm-bytes 5G --vm-keep
Scaling up the number of stress threads and vCPUs should make it easier to reproduce.
Other than unresponsiveness/lockup messages until the memory is accepted, the guest seems to continue running fine, but for large guests where unaccepted memory is more likely to be useful, it seems like it could be an issue, especially when consider 100+ vCPU guests.
-Mike
[ 105.753073] watchdog: BUG: soft lockup - CPU#3 stuck for 27s! [stress:1590] [ 105.756109] Modules linked in: btrfs(E) blake2b_generic(E) raid10(E) raid456(E) async_raid6_recov(E) async_memcpy(E) async_pq(E) async_xor(E) async_tx(E) xor(E) raid6_pq(E) libcrc32c(E) raid1(E) raid0(E) multipath(E) linear(E) crc32_pclmul(E) virtio_net(E) i2c_i801(E) net_failover(E) i2c_smbus(E) psmouse(E) failover(E) virtio_scsi(E) lpc_ich(E) [ 105.771644] irq event stamp: 392274 [ 105.773852] hardirqs last enabled at (392273): [<ffffffff9fbddf9a>] _raw_spin_unlock_irqrestore+0x5a/0x70 [ 105.780058] hardirqs last disabled at (392274): [<ffffffff9fbddca9>] _raw_spin_lock_irqsave+0x69/0x70 [ 105.785486] softirqs last enabled at (392158): [<ffffffff9fbdf663>] __do_softirq+0x2a3/0x357 [ 105.790627] softirqs last disabled at (392149): [<ffffffff9ecd0f5c>] irq_exit_rcu+0x8c/0xb0 [ 105.795972] CPU: 3 PID: 1590 Comm: stress Tainted: G EL 6.6.0-rc5-snp-guest-v10n+ #1 [ 105.802416] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 2/2/2022 [ 105.807040] RIP: 0010:_raw_spin_unlock_irqrestore+0x45/0x70 [ 105.810327] Code: b1 b1 18 ff 4c 89 e7 e8 79 e7 18 ff 81 e3 00 02 00 00 75 26 9c 58 0f 1f 40 00 f6 c4 02 75 22 48 85 db 74 06 fb 0f 1f 44 00 00 <65> ff 0d fc 5d 46 60 5b 41 5c 5d c3 cc cc cc cc e8 c6 9f 28 ff eb [ 105.818749] RSP: 0000:ffffc9000585faa8 EFLAGS: 00000206 [ 105.821155] RAX: 0000000000000046 RBX: 0000000000000200 RCX: 00000000000028ce [ 105.825258] RDX: ffffffff9f8c7830 RSI: ffffffff9fbddf9a RDI: ffffffff9fbddf9a [ 105.828287] RBP: ffffc9000585fab8 R08: 0000000000000000 R09: 0000000000000000 [ 105.831322] R10: 000000ffffffffff R11: 0000000000000000 R12: ffffffffa091a160 [ 105.834713] R13: 00000000000028cd R14: ffff88807c459030 R15: ffff88807c459018 [ 105.838230] FS: 00007fa4a8ee8740(0000) GS:ffff88b1aff80000(0000) knlGS:0000000000000000 [ 105.841677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 105.844115] CR2: 00007fa3719ec010 CR3: 000800010f7c2003 CR4: 0000000000770ef0 [ 105.847160] PKRU: 55555554 [ 105.848490] Call Trace: [ 105.849634] <IRQ> [ 105.850517] ? show_regs+0x68/0x70 [ 105.851986] ? watchdog_timer_fn+0x1dc/0x240 [ 105.853868] ? __pfx_watchdog_timer_fn+0x10/0x10 [ 105.855827] ? __hrtimer_run_queues+0x1c3/0x340 [ 105.857821] ? hrtimer_interrupt+0x109/0x240 [ 105.859673] ? __sysvec_apic_timer_interrupt+0x67/0x170 [ 105.861941] ? sysvec_apic_timer_interrupt+0x7b/0x90 [ 105.864048] </IRQ> [ 105.864981] <TASK> [ 105.865901] ? asm_sysvec_apic_timer_interrupt+0x1b/0x20 [ 105.868110] ? accept_memory+0x150/0x170 [ 105.869795] ? _raw_spin_unlock_irqrestore+0x5a/0x70 [ 105.871885] ? _raw_spin_unlock_irqrestore+0x5a/0x70 [ 105.873966] ? _raw_spin_unlock_irqrestore+0x45/0x70 [ 105.876047] accept_memory+0x150/0x170 [ 105.877665] try_to_accept_memory+0x134/0x1b0 [ 105.879532] get_page_from_freelist+0xa3e/0x1370 [ 105.881479] ? lock_acquire+0xd8/0x2b0 [ 105.883045] __alloc_pages+0x1b7/0x390 [ 105.884619] __folio_alloc+0x1b/0x50 [ 105.886151] ? policy_node+0x57/0x70 [ 105.887673] vma_alloc_folio+0xa6/0x360 [ 105.889325] do_pte_missing+0x1a5/0x8b0 [ 105.890948] __handle_mm_fault+0x75e/0xda0 [ 105.892693] handle_mm_fault+0xe1/0x2d0 [ 105.894304] do_user_addr_fault+0x1ce/0xa40 [ 105.896050] ? exit_to_user_mode_prepare+0xa4/0x230 [ 105.898113] exc_page_fault+0x84/0x200 [ 105.899693] asm_exc_page_fault+0x27/0x30 [ 105.901405] RIP: 0033:0x55ebd4602cf0 [ 105.902926] Code: 8b 54 24 0c 31 c0 85 d2 0f 94 c0 89 04 24 41 83 fd 02 0f 8f 3e 02 00 00 48 85 ed 48 89 d8 7e 1b 66 2e 0f 1f 84 00 00 00 00 00 <c6> 00 5a 4c 01 f8 48 89 c2 48 29 da 48 39 d5 7f ef 49 83 fc 00 0f [ 105.910671] RSP: 002b:00007ffebfede470 EFLAGS: 00010206 [ 105.912897] RAX: 00007fa3719ec010 RBX: 00007fa3683ff010 RCX: 00007fa3683ff010 [ 105.915859] RDX: 00000000095ed000 RSI: 0000000140001000 RDI: 0000000000000000 [ 105.918853] RBP: 0000000140000000 R08: 00000000ffffffff R09: 0000000000000000 [ 105.921836] R10: 0000000000000022 R11: 0000000000000246 R12: ffffffffffffffff [ 105.924810] R13: 0000000000000002 R14: fffffffffffff000 R15: 0000000000001000 [ 105.927771] </TASK> [ 110.367774] watchdog: BUG: soft lockup - CPU#15 stuck for 32s! [stress:1594] [ 110.369415] watchdog: BUG: soft lockup - CPU#13 stuck for 36s! [stress:1602]
Re: [PATCHv14 5/9] efi: Add unaccepted memory support
Quoted 118 lines · Michael Roth
Quoted 92 lines · Kirill A. Shutemov
efi_config_parse_tables()reserves memory that holds unaccepted memory configuration table so it won't be reused by page allocator.Core-mm requires few helpers to support unaccepted memory:
-
accept_memory()checks the range of addresses against the bitmap and accept memory if needed.-
range_contains_unaccepted_memory()checks if anything within the range requires acceptance.Architectural code has to provide
efi_get_unaccepted_table()that returns pointer to the unaccepted memory configuration table.
arch_accept_memory()handles arch-specific part of memory acceptance.Signed-off-by Kirill A. Shutemov <[email protected]>Reviewed-by Ard Biesheuvel <[email protected]>Reviewed-by Tom Lendacky <[email protected]>--- arch/x86/platform/efi/efi.c | 3 + drivers/firmware/efi/Makefile | 1 + drivers/firmware/efi/efi.c | 25 +++++ drivers/firmware/efi/unaccepted_memory.c | 112 +++++++++++++++++++++++ include/linux/efi.h | 1 + 5 files changed, 142 insertions(+) create mode 100644 drivers/firmware/efi/unaccepted_memory.cdrivers/firmware/efi/unaccepted_memory.c new file +58
-0@@ -0,0 +1,112 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include <linux/efi.h> +#include <linux/memblock.h> +#include <linux/spinlock.h> +#include <asm/unaccepted_memory.h> + +/* Protects unaccepted memory bitmap */ +static DEFINE_SPINLOCK (unaccepted_memory_lock); + +/* +* accept_memory() -- Consult bitmap and accept the memory if needed. +* +* Only memory that is explicitly marked as unaccepted in the bitmap requires +* an action. All the remaining memory is implicitly accepted and doesn't need +* acceptance. +* +* No need to accept: +* - anything if the system has no unaccepted table; +* - memory that is below phys_base; +* - memory that is above the memory that addressable by the bitmap; +*/ +void accept_memory (phys_addr_t start,phys_addr_t end) +{ +struct efi_unaccepted_memory *unaccepted; +unsigned long range_start, range_end; +unsigned long flags; +u64 unit_size; + + unaccepted =efi_get_unaccepted_table (); +if (!unaccepted) +return ; + + unit_size = unaccepted->unit_size; + +/* +* Only care for the part of the range that is represented +* in the bitmap. +*/ +if (start < unaccepted->phys_base) + start = unaccepted->phys_base; +if (end < unaccepted->phys_base) +return ; + +/* Translate to offsets from the beginning of the bitmap */ + start -= unaccepted->phys_base; + end -= unaccepted->phys_base; + +/* Make sure not to overrun the bitmap */ +if (end > unaccepted->size * unit_size *BITS_PER_BYTE ) + end = unaccepted->size * unit_size *BITS_PER_BYTE ; + + range_start = start / unit_size; + +spin_lock_irqsave (&unaccepted_memory_lock, flags); +for_each_set_bitrange_from (range_start, range_end, unaccepted->bitmap, +DIV_ROUND_UP (end, unit_size)) { +unsigned long phys_start, phys_end;drivers/firmware/efi/unaccepted_memory.c line 68
Quoted 10 lines
drivers/firmware/efi/unaccepted_memory.c new file +10
-0+unsigned long len = range_end - range_start; + + phys_start = range_start * unit_size + unaccepted->phys_base; + phys_end = range_end * unit_size + unaccepted->phys_base; + +arch_accept_memory (phys_start, phys_end); +bitmap_clear (unaccepted->bitmap, range_start, len); + } +spin_unlock_irqrestore (&unaccepted_memory_lock, flags); +}While testing SNP guests running today's tip/master (
ef19bc9dddc3) I ran into what seems to be fairly significant lock contention due to theunaccepted_memory_lockspinlock above, which results in a constant stream of soft-lockups until the workload gets all its memory accepted/faulted in if the guest has around 16+ vCPUs.I've included the guest dmesg traces I was seeing below.
In this case I was running a 32 vCPU guest with 200GB of memory running on a 256 thread EPYC (Milan) system, and can trigger the above situation fairly reliably by running the following workload in a freshly-booted guests:
stress --vm 32 --vm-bytes 5G --vm-keep
Quoted 7 lines
Scaling up the number of stress threads and vCPUs should make it easier to reproduce.
Other than unresponsiveness/lockup messages until the memory is accepted, the guest seems to continue running fine, but for large guests where unaccepted memory is more likely to be useful, it seems like it could be an issue, especially when consider 100+ vCPU guests.
Okay, sorry for delay. It took time to reproduce it with TDX.
I will look what can be done.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCHv14 5/9] efi: Add unaccepted memory support
Quoted 17 lines · Kirill A. Shutemov
Quoted 17 lines
While testing SNP guests running today's tip/master (
ef19bc9dddc3) I ran into what seems to be fairly significant lock contention due to theunaccepted_memory_lockspinlock above, which results in a constant stream of soft-lockups until the workload gets all its memory accepted/faulted in if the guest has around 16+ vCPUs.I've included the guest dmesg traces I was seeing below.
In this case I was running a 32 vCPU guest with 200GB of memory running on a 256 thread EPYC (Milan) system, and can trigger the above situation fairly reliably by running the following workload in a freshly-booted guests:
stress --vm 32 --vm-bytes 5G --vm-keepScaling up the number of stress threads and vCPUs should make it easier to reproduce.
Quoted 8 lines
Quoted 4 lines
Other than unresponsiveness/lockup messages until the memory is accepted, the guest seems to continue running fine, but for large guests where unaccepted memory is more likely to be useful, it seems like it could be an issue, especially when consider 100+ vCPU guests.
Okay, sorry for delay. It took time to reproduce it with TDX.
I will look what can be done.
Could you check if the patch below helps?
drivers/firmware/efi/unaccepted_memory.c +33 -0
@@ -8,6 +8,14 @@/* Protects unaccepted memory bitmap */ static DEFINE_SPINLOCK (unaccepted_memory_lock); +struct accept_range { +struct list_head list; +unsigned long start; +unsigned long end; +}; + +static LIST_HEAD (accepting_list); +/* * accept_memory() -- Consult bitmap and accept the memory if needed. *
@@ -24,6 +32,7 @@ void accept_memory(phys_addr_t start, phys_addr_t end) {struct efi_unaccepted_memory *unaccepted;unsigned long range_start, range_end; +struct accept_range range, *entry;unsigned long flags;u64 unit_size;
@@ -80,7 +89,25 @@ void accept_memory(phys_addr_t start, phys_addr_t end) range_start = start / unit_size; + range.start = start; + range.end = end; +retry:spin_lock_irqsave (&unaccepted_memory_lock, flags); + +list_for_each_entry (entry, &accepting_list, list) { +if (entry->end < start) +continue ; +if (entry->start > end) +continue ; +spin_unlock_irqrestore (&unaccepted_memory_lock, flags); + +/* Somebody else accepting the range */ +cpu_relax (); + goto retry; + } + +list_add (&range.list, &accepting_list); +for_each_set_bitrange_from (range_start, range_end, unaccepted->bitmap,DIV_ROUND_UP (end, unit_size)) {unsigned long phys_start, phys_end;
@@ -89,9 +116,15 @@ void accept_memory(phys_addr_t start, phys_addr_t end) phys_start = range_start * unit_size + unaccepted->phys_base; phys_end = range_end * unit_size + unaccepted->phys_base; +spin_unlock_irqrestore (&unaccepted_memory_lock, flags); +arch_accept_memory (phys_start, phys_end); + +spin_lock_irqsave (&unaccepted_memory_lock, flags);bitmap_clear (unaccepted->bitmap, range_start, len); } + +list_del (&range.list);spin_unlock_irqrestore (&unaccepted_memory_lock, flags); }
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCHv14 5/9] efi: Add unaccepted memory support
Quoted 64 lines · Kirill A. Shutemov
Quoted 17 lines · Kirill A. Shutemov
Quoted 17 lines
While testing SNP guests running today's tip/master (
ef19bc9dddc3) I ran into what seems to be fairly significant lock contention due to theunaccepted_memory_lockspinlock above, which results in a constant stream of soft-lockups until the workload gets all its memory accepted/faulted in if the guest has around 16+ vCPUs.I've included the guest dmesg traces I was seeing below.
In this case I was running a 32 vCPU guest with 200GB of memory running on a 256 thread EPYC (Milan) system, and can trigger the above situation fairly reliably by running the following workload in a freshly-booted guests:
stress --vm 32 --vm-bytes 5G --vm-keepScaling up the number of stress threads and vCPUs should make it easier to reproduce.
Quoted 8 lines
Quoted 4 lines
Other than unresponsiveness/lockup messages until the memory is accepted, the guest seems to continue running fine, but for large guests where unaccepted memory is more likely to be useful, it seems like it could be an issue, especially when consider 100+ vCPU guests.
Okay, sorry for delay. It took time to reproduce it with TDX.
I will look what can be done.
Could you check if the patch below helps?
drivers/firmware/efi/unaccepted_memory.c +12
-0@@ -8,6 +8,14 @@/* Protects unaccepted memory bitmap */ static DEFINE_SPINLOCK (unaccepted_memory_lock); +struct accept_range { +struct list_head list; +unsigned long start; +unsigned long end; +}; + +static LIST_HEAD (accepting_list); +/* * accept_memory() -- Consult bitmap and accept the memory if needed. * @@ -24,6 +32,7 @@ void accept_memory(phys_addr_t start, phys_addr_t end) {struct efi_unaccepted_memory *unaccepted;unsigned long range_start, range_end; +struct accept_range range, *entry;unsigned long flags;u64 unit_size;@@ -80,7 +89,25 @@ void accept_memory(phys_addr_t start, phys_addr_t end) range_start = start / unit_size; + range.start = start; + range.end = end; +retry:spin_lock_irqsave (&unaccepted_memory_lock, flags);
drivers/firmware/efi/unaccepted_memory.c void accept_memory(phys_addr_t start, phys_addr_t end) · line 105
Quoted 10 lines
drivers/firmware/efi/unaccepted_memory.c +10
-0+ +list_for_each_entry (entry, &accepting_list, list) { +if (entry->end < start) +continue ; +if (entry->start > end) +continue ; +spin_unlock_irqrestore (&unaccepted_memory_lock, flags); + +/* Somebody else accepting the range */ +cpu_relax ();
Should this be rather cond_resched()? I think cpu_relax() isn't enough to prevent soft lockups.
Although IIUC hitting this should be rare, as the contending tasks will pick different ranges via try_to_accept_memory_one(), right?
Quoted 24 lines
drivers/firmware/efi/unaccepted_memory.c +11
-0+ goto retry; + } + +list_add (&range.list, &accepting_list); +for_each_set_bitrange_from (range_start, range_end, unaccepted->bitmap,DIV_ROUND_UP (end, unit_size)) {unsigned long phys_start, phys_end;@@ -89,9 +116,15 @@ void accept_memory(phys_addr_t start, phys_addr_t end) phys_start = range_start * unit_size + unaccepted->phys_base; phys_end = range_end * unit_size + unaccepted->phys_base; +spin_unlock_irqrestore (&unaccepted_memory_lock, flags); +arch_accept_memory (phys_start, phys_end); + +spin_lock_irqsave (&unaccepted_memory_lock, flags);bitmap_clear (unaccepted->bitmap, range_start, len); } + +list_del (&range.list);spin_unlock_irqrestore (&unaccepted_memory_lock, flags); }
Re: [PATCHv14 5/9] efi: Add unaccepted memory support
Quoted 70 lines · Vlastimil Babka
Quoted 69 lines · Kirill A. Shutemov
Quoted 17 lines · Kirill A. Shutemov
Quoted 17 lines
While testing SNP guests running today's tip/master (
ef19bc9dddc3) I ran into what seems to be fairly significant lock contention due to theunaccepted_memory_lockspinlock above, which results in a constant stream of soft-lockups until the workload gets all its memory accepted/faulted in if the guest has around 16+ vCPUs.I've included the guest dmesg traces I was seeing below.
In this case I was running a 32 vCPU guest with 200GB of memory running on a 256 thread EPYC (Milan) system, and can trigger the above situation fairly reliably by running the following workload in a freshly-booted guests:
stress --vm 32 --vm-bytes 5G --vm-keepScaling up the number of stress threads and vCPUs should make it easier to reproduce.
Quoted 8 lines
Quoted 4 lines
Other than unresponsiveness/lockup messages until the memory is accepted, the guest seems to continue running fine, but for large guests where unaccepted memory is more likely to be useful, it seems like it could be an issue, especially when consider 100+ vCPU guests.
Okay, sorry for delay. It took time to reproduce it with TDX.
I will look what can be done.
Could you check if the patch below helps?
drivers/firmware/efi/unaccepted_memory.c +17
-0@@ -8,6 +8,14 @@/* Protects unaccepted memory bitmap */ static DEFINE_SPINLOCK (unaccepted_memory_lock); +struct accept_range { +struct list_head list; +unsigned long start; +unsigned long end; +}; + +static LIST_HEAD (accepting_list); +/* * accept_memory() -- Consult bitmap and accept the memory if needed. * @@ -24,6 +32,7 @@ void accept_memory(phys_addr_t start, phys_addr_t end) {struct efi_unaccepted_memory *unaccepted;unsigned long range_start, range_end; +struct accept_range range, *entry;unsigned long flags;u64 unit_size;@@ -80,7 +89,25 @@ void accept_memory(phys_addr_t start, phys_addr_t end) range_start = start / unit_size; + range.start = start; + range.end = end; +retry:spin_lock_irqsave (&unaccepted_memory_lock, flags); + +list_for_each_entry (entry, &accepting_list, list) { +if (entry->end < start) +continue ; +if (entry->start > end)
Quoted 8 lines
drivers/firmware/efi/unaccepted_memory.c void accept_memory(phys_addr_t start, phys_addr_t end) · line 105
Quoted 5 lines
drivers/firmware/efi/unaccepted_memory.c +5
-0+continue ; +spin_unlock_irqrestore (&unaccepted_memory_lock, flags); + +/* Somebody else accepting the range */ +cpu_relax ();Should this be rather
cond_resched()? I thinkcpu_relax()isn't enough to prevent soft lockups.
Right. For some reason, I thought we cannot call cond_resched() from atomic context (we sometimes get there from atomic context), but we can.
Quoted 2 lines
Although IIUC hitting this should be rare, as the contending tasks will pick different ranges via
try_to_accept_memory_one(), right?
Yes, it should be rare.
Generally, with exception of memblock, we accept all memory with MAX_ORDER chunks. As long as unit_size <= MAX_ORDER page allocator should never trigger the conflict as the caller owns full range to accept.
I will test the idea with larger unit_size to see how it behaves.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCHv14 5/9] efi: Add unaccepted memory support
Quoted 1 line · Kirill A. Shutemov
I will test the idea with larger
unit_sizeto see how it behaves.
It indeed uncovered an issue. We need to record ranges on accepting_list in unit_size granularity. Otherwise, we fail to stop parallel accept requests to the same unit_size block if they don't overlap on physical addresses.
Updated patch:
drivers/firmware/efi/unaccepted_memory.c +51 -4
@@ -5,9 +5,17 @@#include <linux/spinlock.h> #include <asm/unaccepted_memory.h> -/* Protects unaccepted memory bitmap */ +/* Protects unaccepted memory bitmap and accepting_list */ static DEFINE_SPINLOCK (unaccepted_memory_lock); +struct accept_range { +struct list_head list; +unsigned long start; +unsigned long end; +}; + +static LIST_HEAD (accepting_list); +/* * accept_memory() -- Consult bitmap and accept the memory if needed. *
@@ -24,6 +32,7 @@ void accept_memory(phys_addr_t start, phys_addr_t end) {struct efi_unaccepted_memory *unaccepted;unsigned long range_start, range_end; +struct accept_range range, *entry;unsigned long flags;u64 unit_size;
@@ -78,20 +87,58 @@ void accept_memory(phys_addr_t start, phys_addr_t end)if (end > unaccepted->size * unit_size *BITS_PER_BYTE ) end = unaccepted->size * unit_size *BITS_PER_BYTE ; - range_start = start / unit_size; - + range.start = start / unit_size; + range.end =DIV_ROUND_UP (end, unit_size); +retry:spin_lock_irqsave (&unaccepted_memory_lock, flags); + +/* +* Check if anybody works on accepting the same range of the memory. +* +* The check with unit_size granularity. It is crucial to catch all +* accept requests to the same unit_size block, even if they don't +* overlap on physical address level. +*/ +list_for_each_entry (entry, &accepting_list, list) { +if (entry->end < range.start) +continue ; +if (entry->start >= range.end) +continue ; + +/* +* Somebody else accepting the range. Or at least part of it. +* +* Drop the lock and retry until it is complete. +*/ +spin_unlock_irqrestore (&unaccepted_memory_lock, flags); +cond_resched (); + goto retry; + } + +/* +* Register that the range is about to be accepted. +* Make sure nobody else will accept it. +*/ +list_add (&range.list, &accepting_list); + + range_start = range.start;for_each_set_bitrange_from (range_start, range_end, unaccepted->bitmap, -DIV_ROUND_UP (end, unit_size)) { + range.end) {unsigned long phys_start, phys_end;unsigned long len = range_end - range_start; phys_start = range_start * unit_size + unaccepted->phys_base; phys_end = range_end * unit_size + unaccepted->phys_base; +spin_unlock_irqrestore (&unaccepted_memory_lock, flags); +arch_accept_memory (phys_start, phys_end); + +spin_lock_irqsave (&unaccepted_memory_lock, flags);bitmap_clear (unaccepted->bitmap, range_start, len); } + +list_del (&range.list);spin_unlock_irqrestore (&unaccepted_memory_lock, flags); }
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCHv14 5/9] efi: Add unaccepted memory support
In reply to Kirill A. ShutemovQuoted 59 lines · Kirill A. Shutemov
Quoted 17 lines · Kirill A. Shutemov
Quoted 17 lines
While testing SNP guests running today's tip/master (
ef19bc9dddc3) I ran into what seems to be fairly significant lock contention due to theunaccepted_memory_lockspinlock above, which results in a constant stream of soft-lockups until the workload gets all its memory accepted/faulted in if the guest has around 16+ vCPUs.I've included the guest dmesg traces I was seeing below.
In this case I was running a 32 vCPU guest with 200GB of memory running on a 256 thread EPYC (Milan) system, and can trigger the above situation fairly reliably by running the following workload in a freshly-booted guests:
stress --vm 32 --vm-bytes 5G --vm-keepScaling up the number of stress threads and vCPUs should make it easier to reproduce.
Quoted 8 lines
Quoted 4 lines
Other than unresponsiveness/lockup messages until the memory is accepted, the guest seems to continue running fine, but for large guests where unaccepted memory is more likely to be useful, it seems like it could be an issue, especially when consider 100+ vCPU guests.
Okay, sorry for delay. It took time to reproduce it with TDX.
I will look what can be done.
Could you check if the patch below helps?
drivers/firmware/efi/unaccepted_memory.c +9
-0@@ -8,6 +8,14 @@/* Protects unaccepted memory bitmap */ static DEFINE_SPINLOCK (unaccepted_memory_lock); +struct accept_range { +struct list_head list; +unsigned long start; +unsigned long end; +}; + +static LIST_HEAD (accepting_list); +/* * accept_memory() -- Consult bitmap and accept the memory if needed. * @@ -24,6 +32,7 @@ void accept_memory(phys_addr_t start, phys_addr_t end) {struct efi_unaccepted_memory *unaccepted;unsigned long range_start, range_end; +struct accept_range range, *entry;unsigned long flags;u64 unit_size;@@ -80,7 +89,25 @@ void accept_memory(phys_addr_t start, phys_addr_t end) range_start = start / unit_size;
drivers/firmware/efi/unaccepted_memory.c void accept_memory(phys_addr_t start, phys_addr_t end) · line 100
Quoted 10 lines
drivers/firmware/efi/unaccepted_memory.c +8
-0+ range.start = start; + range.end = end; +retry:spin_lock_irqsave (&unaccepted_memory_lock, flags); + +list_for_each_entry (entry, &accepting_list, list) { +if (entry->end < start) +continue ; +if (entry->start > end)
Should this be a >= check since start and end are page aligned values?
drivers/firmware/efi/unaccepted_memory.c void accept_memory(phys_addr_t start, phys_addr_t end) · line 106
Quoted 6 lines
drivers/firmware/efi/unaccepted_memory.c +6
-0+continue ; +spin_unlock_irqrestore (&unaccepted_memory_lock, flags); + +/* Somebody else accepting the range */ +cpu_relax (); + goto retry;
Could you set some kind of flag here so that ...
drivers/firmware/efi/unaccepted_memory.c void accept_memory(phys_addr_t start, phys_addr_t end) · line 108
Quoted 2 lines
drivers/firmware/efi/unaccepted_memory.c +2
-0+ } +
... once you get here, that means that area was accepted and removed from the list, so I think you could just drop the lock and exit now, right? Because at that point the bitmap will have been updated and you wouldn't be accepting any memory anyway?
Thanks,
Tom
Quoted 21 lines
drivers/firmware/efi/unaccepted_memory.c +8
-0+list_add (&range.list, &accepting_list); +for_each_set_bitrange_from (range_start, range_end, unaccepted->bitmap,DIV_ROUND_UP (end, unit_size)) {unsigned long phys_start, phys_end;@@ -89,9 +116,15 @@ void accept_memory(phys_addr_t start, phys_addr_t end) phys_start = range_start * unit_size + unaccepted->phys_base; phys_end = range_end * unit_size + unaccepted->phys_base; +spin_unlock_irqrestore (&unaccepted_memory_lock, flags); +arch_accept_memory (phys_start, phys_end); + +spin_lock_irqsave (&unaccepted_memory_lock, flags);bitmap_clear (unaccepted->bitmap, range_start, len); } + +list_del (&range.list);spin_unlock_irqrestore (&unaccepted_memory_lock, flags); }
Re: [PATCHv14 5/9] efi: Add unaccepted memory support
Quoted 60 lines · Tom Lendacky
Quoted 59 lines · Kirill A. Shutemov
Quoted 17 lines · Kirill A. Shutemov
Quoted 17 lines
While testing SNP guests running today's tip/master (
ef19bc9dddc3) I ran into what seems to be fairly significant lock contention due to theunaccepted_memory_lockspinlock above, which results in a constant stream of soft-lockups until the workload gets all its memory accepted/faulted in if the guest has around 16+ vCPUs.I've included the guest dmesg traces I was seeing below.
In this case I was running a 32 vCPU guest with 200GB of memory running on a 256 thread EPYC (Milan) system, and can trigger the above situation fairly reliably by running the following workload in a freshly-booted guests:
stress --vm 32 --vm-bytes 5G --vm-keepScaling up the number of stress threads and vCPUs should make it easier to reproduce.
Quoted 8 lines
Quoted 4 lines
Other than unresponsiveness/lockup messages until the memory is accepted, the guest seems to continue running fine, but for large guests where unaccepted memory is more likely to be useful, it seems like it could be an issue, especially when consider 100+ vCPU guests.
Okay, sorry for delay. It took time to reproduce it with TDX.
I will look what can be done.
Could you check if the patch below helps?
drivers/firmware/efi/unaccepted_memory.c +12
-0@@ -8,6 +8,14 @@/* Protects unaccepted memory bitmap */ static DEFINE_SPINLOCK (unaccepted_memory_lock); +struct accept_range { +struct list_head list; +unsigned long start; +unsigned long end; +}; + +static LIST_HEAD (accepting_list); +/* * accept_memory() -- Consult bitmap and accept the memory if needed. * @@ -24,6 +32,7 @@ void accept_memory(phys_addr_t start, phys_addr_t end) {struct efi_unaccepted_memory *unaccepted;unsigned long range_start, range_end; +struct accept_range range, *entry;unsigned long flags;u64 unit_size;@@ -80,7 +89,25 @@ void accept_memory(phys_addr_t start, phys_addr_t end) range_start = start / unit_size; + range.start = start; + range.end = end; +retry:
Quoted 8 lines
drivers/firmware/efi/unaccepted_memory.c void accept_memory(phys_addr_t start, phys_addr_t end) · line 100
Quoted 6 lines
drivers/firmware/efi/unaccepted_memory.c +5
-0spin_lock_irqsave (&unaccepted_memory_lock, flags); + +list_for_each_entry (entry, &accepting_list, list) { +if (entry->end < start) +continue ; +if (entry->start > end)Should this be a >= check since start and end are page aligned values?
Right. Good catch.
Quoted 9 lines
drivers/firmware/efi/unaccepted_memory.c void accept_memory(phys_addr_t start, phys_addr_t end) · line 106
Quoted 6 lines
drivers/firmware/efi/unaccepted_memory.c +6
-0+continue ; +spin_unlock_irqrestore (&unaccepted_memory_lock, flags); + +/* Somebody else accepting the range */ +cpu_relax (); + goto retry;Could you set some kind of flag here so that ...
Quoted 7 lines
drivers/firmware/efi/unaccepted_memory.c void accept_memory(phys_addr_t start, phys_addr_t end) · line 106
Quoted 2 lines
drivers/firmware/efi/unaccepted_memory.c +2
-0+ } +... once you get here, that means that area was accepted and removed from the list, so I think you could just drop the lock and exit now, right? Because at that point the bitmap will have been updated and you wouldn't be accepting any memory anyway?
No. Consider the case if someone else accept part of the range you are accepting.
I guess we can check if the range on the list covers what we are accepting fully, but it complication. Checking bitmap at this point is cheap enough:
we already hold the lock.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
PATCH v14 6/9 efi/unaccepted: Avoid load_unaligned_zeropad() stepping into unaccepted memory
In reply to Kirill A. Shutemovload_unaligned_zeropad() can lead to unwanted loads across page boundaries. The unwanted loads are typically harmless. But, they might be made to totally unrelated or even unmapped memory. load_unaligned_zeropad() relies on exception fixup (#PF, #GP and now #VE) to recover from these unwanted loads.
But, this approach does not work for unaccepted memory. For TDX, a load from unaccepted memory will not lead to a recoverable exception within the guest. The guest will exit to the VMM where the only recourse is to terminate the guest.
There are two parts to fix this issue and comprehensively avoid access to unaccepted memory. Together these ensure that an extra "guard" page is accepted in addition to the memory that needs to be used.
1. Implicitly extend the range_contains_unaccepted_memory(start, end) checks up to end+unit_size if 'end' is aligned on a unit_size boundary.
2. Implicitly extend accept_memory(start, end) to end+unit_size if 'end' is aligned on a unit_size boundary.
Side note: This leads to something strange. Pages which were accepted
at boot, marked by the firmware as accepted and will never _need_ to be accepted might be on unaccepted_pages list This is a cue to ensure that the next page is accepted before 'page' can be used.
This is an actual, real-world problem which was discovered during TDX testing.
--- drivers/firmware/efi/unaccepted_memory.c | 35 ++++++++++++++++++++++++ 1 file changed, 35 insertions(+)
drivers/firmware/efi/unaccepted_memory.c +35 -0
@@ -46,6 +46,34 @@ void accept_memory(phys_addr_t start, phys_addr_t end) start -= unaccepted->phys_base; end -= unaccepted->phys_base; +/* +* load_unaligned_zeropad() can lead to unwanted loads across page +* boundaries. The unwanted loads are typically harmless. But, they +* might be made to totally unrelated or even unmapped memory. +* load_unaligned_zeropad() relies on exception fixup (#PF, #GP and now +* #VE) to recover from these unwanted loads. +* +* But, this approach does not work for unaccepted memory. For TDX, a +* load from unaccepted memory will not lead to a recoverable exception +* within the guest. The guest will exit to the VMM where the only +* recourse is to terminate the guest. +* +* There are two parts to fix this issue and comprehensively avoid +* access to unaccepted memory. Together these ensure that an extra +* "guard" page is accepted in addition to the memory that needs to be +* used: +* +* 1. Implicitly extend the range_contains_unaccepted_memory(start, end) +* checks up to end+unit_size if 'end' is aligned on a unit_size +* boundary. +* +* 2. Implicitly extend accept_memory(start, end) to end+unit_size if +* 'end' is aligned on a unit_size boundary. (immediately following +* this comment) +*/ +if (!(end % unit_size)) + end += unit_size; +/* Make sure not to overrun the bitmap */ if (end > unaccepted->size * unit_size *BITS_PER_BYTE ) end = unaccepted->size * unit_size *BITS_PER_BYTE ;
@@ -93,6 +121,13 @@ bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) start -= unaccepted->phys_base; end -= unaccepted->phys_base; +/* +* Also consider the unaccepted state of the *next* page. See fix #1 in +* the comment on load_unaligned_zeropad() in accept_memory(). +*/ +if (!(end % unit_size)) + end += unit_size; +/* Make sure not to overrun the bitmap */ if (end > unaccepted->size * unit_size *BITS_PER_BYTE ) end = unaccepted->size * unit_size *BITS_PER_BYTE ;
Signature · 1 line
2.39.3
[tip: x86/cc] efi/unaccepted: Avoid load_unaligned_zeropad() stepping into unaccepted memory
The following commit has been merged into the x86/cc branch of tip:
The following commit has been merged into the x86/cc branch of tip:
Commit-ID: c211c19e80d046441655e372c6ae15f29d358259 Gitweb: https://git.kernel.org/tip/c211c19e80d046441655e372c6ae15f29d358259 Author: Kirill A. Shutemov <[email protected]> AuthorDate: Tue, 06 Jun 2023 17:26:34 +03:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 06 Jun 2023 17:27:08 +02:00
efi/unaccepted: Avoid load_unaligned_zeropad() stepping into unaccepted memory
load_unaligned_zeropad() can lead to unwanted loads across page boundaries. The unwanted loads are typically harmless. But, they might be made to totally unrelated or even unmapped memory. load_unaligned_zeropad() relies on exception fixup (#PF, #GP and now #VE) to recover from these unwanted loads.
But, this approach does not work for unaccepted memory. For TDX, a load from unaccepted memory will not lead to a recoverable exception within the guest. The guest will exit to the VMM where the only recourse is to terminate the guest.
There are two parts to fix this issue and comprehensively avoid access to unaccepted memory. Together these ensure that an extra "guard" page is accepted in addition to the memory that needs to be used.
1. Implicitly extend the range_contains_unaccepted_memory(start, end) checks up to end+unit_size if 'end' is aligned on a unit_size boundary.
2. Implicitly extend accept_memory(start, end) to end+unit_size if 'end' is aligned on a unit_size boundary.
Side note: This leads to something strange. Pages which were accepted
at boot, marked by the firmware as accepted and will never _need_ to be accepted might be on unaccepted_pages list This is a cue to ensure that the next page is accepted before 'page' can be used.
This is an actual, real-world problem which was discovered during TDX testing.
--- drivers/firmware/efi/unaccepted_memory.c | 35 +++++++++++++++++++++++- 1 file changed, 35 insertions(+)
drivers/firmware/efi/unaccepted_memory.c +35 -0
@@ -46,6 +46,34 @@ void accept_memory(phys_addr_t start, phys_addr_t end) start -= unaccepted->phys_base; end -= unaccepted->phys_base; +/* +* load_unaligned_zeropad() can lead to unwanted loads across page +* boundaries. The unwanted loads are typically harmless. But, they +* might be made to totally unrelated or even unmapped memory. +* load_unaligned_zeropad() relies on exception fixup (#PF, #GP and now +* #VE) to recover from these unwanted loads. +* +* But, this approach does not work for unaccepted memory. For TDX, a +* load from unaccepted memory will not lead to a recoverable exception +* within the guest. The guest will exit to the VMM where the only +* recourse is to terminate the guest. +* +* There are two parts to fix this issue and comprehensively avoid +* access to unaccepted memory. Together these ensure that an extra +* "guard" page is accepted in addition to the memory that needs to be +* used: +* +* 1. Implicitly extend the range_contains_unaccepted_memory(start, end) +* checks up to end+unit_size if 'end' is aligned on a unit_size +* boundary. +* +* 2. Implicitly extend accept_memory(start, end) to end+unit_size if +* 'end' is aligned on a unit_size boundary. (immediately following +* this comment) +*/ +if (!(end % unit_size)) + end += unit_size; +/* Make sure not to overrun the bitmap */ if (end > unaccepted->size * unit_size *BITS_PER_BYTE ) end = unaccepted->size * unit_size *BITS_PER_BYTE ;
@@ -93,6 +121,13 @@ bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) start -= unaccepted->phys_base; end -= unaccepted->phys_base; +/* +* Also consider the unaccepted state of the *next* page. See fix #1 in +* the comment on load_unaligned_zeropad() in accept_memory(). +*/ +if (!(end % unit_size)) + end += unit_size; +/* Make sure not to overrun the bitmap */ if (end > unaccepted->size * unit_size *BITS_PER_BYTE ) end = unaccepted->size * unit_size *BITS_PER_BYTE ;
PATCH v14 7/9 x86/tdx: Make _tdx_hypercall() and __tdx_module_call() available in boot stub
In reply to Kirill A. ShutemovMemory acceptance requires a hypercall and one or multiple module calls.
Make helpers for the calls available in boot stub. It has to accept memory where kernel image and initrd are placed.
--- arch/x86/coco/tdx/tdx.c | 32 ------------------- arch/x86/include/asm/shared/tdx.h | 51 +++++++++++++++++++++++++++++++ arch/x86/include/asm/tdx.h | 19 ------------ 3 files changed, 51 insertions(+), 51 deletions(-)
arch/x86/coco/tdx/tdx.c +0 -32
@@ -14,20 +14,6 @@#include <asm/insn-eval.h> #include <asm/pgtable.h> -/* TDX module Call Leaf IDs */ -#define TDX_GET_INFO 1 -#define TDX_GET_VEINFO 3 -#define TDX_GET_REPORT 4 -#define TDX_ACCEPT_PAGE 6 -#define TDX_WR 8 - -/* TDCS fields. To be used by TDG.VM.WR and TDG.VM.RD module calls */ -#define TDCS_NOTIFY_ENABLES 0x9100000000000010 - -/* TDX hypercall Leaf IDs */ -#define TDVMCALL_MAP_GPA 0x10001 -#define TDVMCALL_REPORT_FATAL_ERROR 0x10003 -/* MMIO direction */ #define EPT_READ 0#define EPT_WRITE 1
@@ -51,24 +37,6 @@#define TDREPORT_SUBTYPE_0 0 -/* -* Wrapper for standard use of __tdx_hypercall with no output aside from -* return code. -*/ -static inline u64 _tdx_hypercall (u64 fn,u64 r12,u64 r13,u64 r14,u64 r15) -{ -struct tdx_hypercall_args args = { - .r10 =TDX_HYPERCALL_STANDARD , - .r11 = fn, - .r12 = r12, - .r13 = r13, - .r14 = r14, - .r15 = r15, - }; - -return __tdx_hypercall (&args); -} -/* Called from __tdx_hypercall() for unrecoverable failure */ noinstr void __tdx_hypercall_failed(void) {
arch/x86/include/asm/shared/tdx.h +51 -0
65 lines not on this page · raw message
arch/x86/include/asm/tdx.h +0 -19
@@ -20,21 +20,6 @@#ifndef __ASSEMBLY__ -/* -* Used to gather the output registers values of the TDCALL and SEAMCALL -* instructions when requesting services from the TDX module. -* -* This is a software only structure and not part of the TDX module/VMM ABI. -*/ -struct tdx_module_output { -u64 rcx; -u64 rdx; -u64 r8; -u64 r9; -u64 r10; -u64 r11; -}; -/* * Used by the #VE exception handler to gather the #VE exception * info from the TDX module. This is a software only structure
@@ -55,10 +40,6 @@ struct ve_info {void __init tdx_early_init (void ); -/* Used to communicate with the TDX module */ -u64 __tdx_module_call (u64 fn,u64 rcx,u64 rdx,u64 r8,u64 r9, -struct tdx_module_output *out); -void tdx_get_ve_info (struct ve_info *ve);bool tdx_handle_virt_exception (struct pt_regs *regs,struct ve_info *ve);
Signature · 1 line
2.39.3
[tip: x86/cc] x86/tdx: Make _tdx_hypercall() and __tdx_module_call() available in boot stub
The following commit has been merged into the x86/cc branch of tip:
The following commit has been merged into the x86/cc branch of tip:
Commit-ID: ff40b5769a50fab654a70575ff0f49853b799b0e Gitweb: https://git.kernel.org/tip/ff40b5769a50fab654a70575ff0f49853b799b0e Author: Kirill A. Shutemov <[email protected]> AuthorDate: Tue, 06 Jun 2023 17:26:35 +03:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 06 Jun 2023 17:31:23 +02:00
x86/tdx: Make _tdx_hypercall() and __tdx_module_call() available in boot stub
Memory acceptance requires a hypercall and one or multiple module calls.
Make helpers for the calls available in boot stub. It has to accept memory where kernel image and initrd are placed.
--- arch/x86/coco/tdx/tdx.c | 32 +------------------- arch/x86/include/asm/shared/tdx.h | 51 ++++++++++++++++++++++++++++++- arch/x86/include/asm/tdx.h | 19 +----------- 3 files changed, 51 insertions(+), 51 deletions(-)
arch/x86/coco/tdx/tdx.c +0 -32
@@ -14,20 +14,6 @@#include <asm/insn-eval.h> #include <asm/pgtable.h> -/* TDX module Call Leaf IDs */ -#define TDX_GET_INFO 1 -#define TDX_GET_VEINFO 3 -#define TDX_GET_REPORT 4 -#define TDX_ACCEPT_PAGE 6 -#define TDX_WR 8 - -/* TDCS fields. To be used by TDG.VM.WR and TDG.VM.RD module calls */ -#define TDCS_NOTIFY_ENABLES 0x9100000000000010 - -/* TDX hypercall Leaf IDs */ -#define TDVMCALL_MAP_GPA 0x10001 -#define TDVMCALL_REPORT_FATAL_ERROR 0x10003 -/* MMIO direction */ #define EPT_READ 0#define EPT_WRITE 1
@@ -51,24 +37,6 @@#define TDREPORT_SUBTYPE_0 0 -/* -* Wrapper for standard use of __tdx_hypercall with no output aside from -* return code. -*/ -static inline u64 _tdx_hypercall (u64 fn,u64 r12,u64 r13,u64 r14,u64 r15) -{ -struct tdx_hypercall_args args = { - .r10 =TDX_HYPERCALL_STANDARD , - .r11 = fn, - .r12 = r12, - .r13 = r13, - .r14 = r14, - .r15 = r15, - }; - -return __tdx_hypercall (&args); -} -/* Called from __tdx_hypercall() for unrecoverable failure */ noinstr void __tdx_hypercall_failed(void) {
arch/x86/include/asm/shared/tdx.h +51 -0
65 lines not on this page · raw message
arch/x86/include/asm/tdx.h +0 -19
@@ -21,21 +21,6 @@#ifndef __ASSEMBLY__/* -* Used to gather the output registers values of the TDCALL and SEAMCALL -* instructions when requesting services from the TDX module. -* -* This is a software only structure and not part of the TDX module/VMM ABI. -*/ -struct tdx_module_output { -u64 rcx; -u64 rdx; -u64 r8; -u64 r9; -u64 r10; -u64 r11; -}; - -/* * Used by the #VE exception handler to gather the #VE exception * info from the TDX module. This is a software only structure * and not part of the TDX module/VMM ABI.
@@ -55,10 +40,6 @@ struct ve_info {void __init tdx_early_init (void ); -/* Used to communicate with the TDX module */ -u64 __tdx_module_call (u64 fn,u64 rcx,u64 rdx,u64 r8,u64 r9, -struct tdx_module_output *out); -void tdx_get_ve_info (struct ve_info *ve);bool tdx_handle_virt_exception (struct pt_regs *regs,struct ve_info *ve);
PATCH v14 8/9 x86/tdx: Refactor try_accept_one()
In reply to Kirill A. ShutemovRework try_accept_one() to return accepted size instead of modifying 'start' inside the helper. It makes 'start' in-only argument and streamlines code on the caller side.
--- arch/x86/coco/tdx/tdx.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-)
arch/x86/coco/tdx/tdx.c +19 -19
@@ -713,18 +713,18 @@ static bool tdx_cache_flush_required(void)return true; } -static bool try_accept_one (phys_addr_t *start,unsigned long len, -enum pg_level pg_level) +static unsigned long try_accept_one (phys_addr_t start,unsigned long len, +enum pg_level pg_level) {unsigned long accept_size =page_level_size (pg_level);u64 tdcall_rcx;u8 page_size; -if (!IS_ALIGNED (*start, accept_size)) -return false; +if (!IS_ALIGNED (start, accept_size)) +return 0 ;if (len < accept_size) -return false; +return 0 ;/* * Pass the page physical address to the TDX module to accept the
@@ -743,15 +743,14 @@ static bool try_accept_one(phys_addr_t *start, unsigned long len, page_size =2 ;break ;default : -return false; +return 0 ; } - tdcall_rcx = *start | page_size; + tdcall_rcx = start | page_size;if (__tdx_module_call (TDX_ACCEPT_PAGE , tdcall_rcx,0 ,0 ,0 ,NULL )) -return false; +return 0 ; - *start += accept_size; -return true; +return accept_size; }/*
@@ -788,21 +787,22 @@ static bool tdx_enc_status_changed(unsigned long vaddr, int numpages, bool enc)*/ while (start < end) {unsigned long len = end - start; +unsigned long accept_size;/* * Try larger accepts first. It gives chance to VMM to keep -* 1G/2M SEPT entries where possible and speeds up process by -* cutting number of hypercalls (if successful). +* 1G/2M Secure EPT entries where possible and speeds up +* process by cutting number of hypercalls (if successful). */ -if (try_accept_one (&start, len,PG_LEVEL_1G )) -continue ; - -if (try_accept_one (&start, len,PG_LEVEL_2M )) -continue ; - -if (!try_accept_one (&start, len,PG_LEVEL_4K )) + accept_size =try_accept_one (start, len,PG_LEVEL_1G ); +if (!accept_size) + accept_size =try_accept_one (start, len,PG_LEVEL_2M ); +if (!accept_size) + accept_size =try_accept_one (start, len,PG_LEVEL_4K ); +if (!accept_size)return false; + start += accept_size; }return true;
Signature · 1 line
2.39.3
[tip: x86/cc] x86/tdx: Refactor try_accept_one()
The following commit has been merged into the x86/cc branch of tip:
The following commit has been merged into the x86/cc branch of tip:
Commit-ID: c2b353ae24d6fecddcc599529ad8319282494781 Gitweb: https://git.kernel.org/tip/c2b353ae24d6fecddcc599529ad8319282494781 Author: Kirill A. Shutemov <[email protected]> AuthorDate: Tue, 06 Jun 2023 17:26:36 +03:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 06 Jun 2023 17:38:50 +02:00
x86/tdx: Refactor try_accept_one()
Rework try_accept_one() to return accepted size instead of modifying 'start' inside the helper. It makes 'start' in-only argument and streamlines code on the caller side.
--- arch/x86/coco/tdx/tdx.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-)
arch/x86/coco/tdx/tdx.c +19 -19
@@ -713,18 +713,18 @@ static bool tdx_cache_flush_required(void)return true; } -static bool try_accept_one (phys_addr_t *start,unsigned long len, -enum pg_level pg_level) +static unsigned long try_accept_one (phys_addr_t start,unsigned long len, +enum pg_level pg_level) {unsigned long accept_size =page_level_size (pg_level);u64 tdcall_rcx;u8 page_size; -if (!IS_ALIGNED (*start, accept_size)) -return false; +if (!IS_ALIGNED (start, accept_size)) +return 0 ;if (len < accept_size) -return false; +return 0 ;/* * Pass the page physical address to the TDX module to accept the
@@ -743,15 +743,14 @@ static bool try_accept_one(phys_addr_t *start, unsigned long len, page_size =2 ;break ;default : -return false; +return 0 ; } - tdcall_rcx = *start | page_size; + tdcall_rcx = start | page_size;if (__tdx_module_call (TDX_ACCEPT_PAGE , tdcall_rcx,0 ,0 ,0 ,NULL )) -return false; +return 0 ; - *start += accept_size; -return true; +return accept_size; }/*
@@ -788,21 +787,22 @@ static bool tdx_enc_status_changed(unsigned long vaddr, int numpages, bool enc)*/ while (start < end) {unsigned long len = end - start; +unsigned long accept_size;/* * Try larger accepts first. It gives chance to VMM to keep -* 1G/2M SEPT entries where possible and speeds up process by -* cutting number of hypercalls (if successful). +* 1G/2M Secure EPT entries where possible and speeds up +* process by cutting number of hypercalls (if successful). */ -if (try_accept_one (&start, len,PG_LEVEL_1G )) -continue ; - -if (try_accept_one (&start, len,PG_LEVEL_2M )) -continue ; - -if (!try_accept_one (&start, len,PG_LEVEL_4K )) + accept_size =try_accept_one (start, len,PG_LEVEL_1G ); +if (!accept_size) + accept_size =try_accept_one (start, len,PG_LEVEL_2M ); +if (!accept_size) + accept_size =try_accept_one (start, len,PG_LEVEL_4K ); +if (!accept_size)return false; + start += accept_size; }return true;
PATCH v14 9/9 x86/tdx: Add unaccepted memory support
In reply to Kirill A. ShutemovHookup TDX-specific code to accept memory.
Accepting the memory is done with ACCEPT_PAGE module call on every page in the range. MAP_GPA hypercall is not required as the unaccepted memory is considered private already.
Extract the part of tdx_enc_status_changed() that does memory acceptance in a new helper. Move the helper tdx-shared.c. It is going to be used by both main kernel and decompressor.
--- arch/x86/Kconfig | 2 + arch/x86/boot/compressed/Makefile | 2 +- arch/x86/boot/compressed/error.c | 19 +++++++ arch/x86/boot/compressed/error.h | 1 + arch/x86/boot/compressed/mem.c | 35 +++++++++++- arch/x86/boot/compressed/tdx-shared.c | 2 + arch/x86/coco/tdx/Makefile | 2 +- arch/x86/coco/tdx/tdx-shared.c | 71 ++++++++++++++++++++++++ arch/x86/coco/tdx/tdx.c | 70 +---------------------- arch/x86/include/asm/shared/tdx.h | 2 + arch/x86/include/asm/unaccepted_memory.h | 24 ++++++++ 11 files changed, 160 insertions(+), 70 deletions(-) create mode 100644 arch/x86/boot/compressed/tdx-shared.c create mode 100644 arch/x86/coco/tdx/tdx-shared.c create mode 100644 arch/x86/include/asm/unaccepted_memory.h
arch/x86/Kconfig +2 -0
@@ -884,9 +884,11 @@ config INTEL_TDX_GUEST bool "Intel TDX (Trust Domain Extensions) - Guest Support" depends on X86_64 && CPU_SUP_INTEL depends on X86_X2APIC + depends on EFI_STUB select ARCH_HAS_CC_PLATFORM select X86_MEM_ENCRYPT select X86_MCE + select UNACCEPTED_MEMORY help Support running as a guest under Intel TDX. Without this support, the guest kernel can not boot or run under TDX.
arch/x86/boot/compressed/Makefile +1 -1
@@ -106,7 +106,7 @@ ifdef CONFIG_X86_64 endif vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o -vmlinux-objs-$(CONFIG_INTEL_TDX_GUEST) += $(obj)/tdx.o $(obj)/tdcall.o +vmlinux-objs-$(CONFIG_INTEL_TDX_GUEST) += $(obj)/tdx.o $(obj)/tdcall.o $(obj)/tdx-shared.o vmlinux-objs-$(CONFIG_UNACCEPTED_MEMORY) += $(obj)/mem.o vmlinux-objs-$(CONFIG_EFI) += $(obj)/efi.o
arch/x86/boot/compressed/error.c +19 -0
@@ -22,3 +22,22 @@ void error(char *m)while (1 ) asm("hlt" ); } + +/* EFI libstub provides vsnprintf() */ +#ifdef CONFIG_EFI_STUB +void panic (const char *fmt, ...) +{ +static char buf[1024 ]; +va_list args; +int len; + +va_start (args, fmt); + len =vsnprintf (buf,sizeof (buf), fmt, args); +va_end (args); + +if (len && buf[len -1 ] =='\n' ) + buf[len -1 ] ='\0' ; + +error (buf); +} +#endif
arch/x86/boot/compressed/error.h +1 -0
@@ -6,5 +6,6 @@void warn (char *m);void error (char *m)__noreturn ; +void panic (const char *fmt, ...)__noreturn __cold ;#endif /* BOOT_COMPRESSED_ERROR_H */
arch/x86/boot/compressed/mem.c +34 -1
@@ -2,11 +2,44 @@#include "error.h" #include "misc.h" +#include "tdx.h" +#include <asm/shared/tdx.h> + +/* +* accept_memory() and process_unaccepted_memory() called from EFI stub which +* runs before decompresser and its early_tdx_detect(). +* +* Enumerate TDX directly from the early users. +*/ +static bool early_is_tdx_guest (void ) +{ +static bool once; +static bool is_tdx; + +if (!IS_ENABLED (CONFIG_INTEL_TDX_GUEST )) +return false; + +if (!once) { +u32 eax, sig[3 ]; + +cpuid_count (TDX_CPUID_LEAF_ID ,0 , &eax, + &sig[0 ], &sig[2 ], &sig[1 ]); + is_tdx = !memcmp (TDX_IDENT , sig,sizeof (sig)); + once = true; + } + +return is_tdx; +}void arch_accept_memory (phys_addr_t start,phys_addr_t end) {/* Platform-specific memory-acceptance call goes here */ -error ("Cannot accept memory" ); +if (early_is_tdx_guest ()) { +if (!tdx_accept_memory (start, end)) +panic ("TDX: Failed to accept memory\n" ); + }else { +error ("Cannot accept memory: unknown platform\n" ); + } }bool init_unaccepted_memory (void )
arch/x86/boot/compressed/tdx-shared.c new file +2 -0
@@ -0,0 +1,2 @@ +#include "error.h" +#include "../../coco/tdx/tdx-shared.c"
arch/x86/coco/tdx/Makefile +1 -1
@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += tdx.o tdcall.o +obj-y += tdx.o tdx-shared.o tdcall.o
arch/x86/coco/tdx/tdx-shared.c new file +71 -0
71 lines not on this page · raw message
arch/x86/coco/tdx/tdx.c +3 -67
82 lines not on this page · raw message
arch/x86/include/asm/shared/tdx.h +2 -0
@@ -91,5 +91,7 @@ struct tdx_module_output {u64 __tdx_module_call (u64 fn,u64 rcx,u64 rdx,u64 r8,u64 r9,struct tdx_module_output *out); +bool tdx_accept_memory (phys_addr_t start,phys_addr_t end); +#endif /* !__ASSEMBLY__ */ #endif /* _ASM_X86_SHARED_TDX_H */
arch/x86/include/asm/unaccepted_memory.h new file +24 -0
@@ -0,0 +1,24 @@ +#ifndef _ASM_X86_UNACCEPTED_MEMORY_H +#define _ASM_X86_UNACCEPTED_MEMORY_H + +#include <linux/efi.h> +#include <asm/tdx.h> + +static inline void arch_accept_memory (phys_addr_t start,phys_addr_t end) +{ +/* Platform-specific memory-acceptance call goes here */ +if (cpu_feature_enabled (X86_FEATURE_TDX_GUEST )) { +if (!tdx_accept_memory (start, end)) +panic ("TDX: Failed to accept memory\n" ); + }else { +panic ("Cannot accept memory: unknown platform\n" ); + } +} + +static inline struct efi_unaccepted_memory *efi_get_unaccepted_table (void ) +{ +if (efi.unaccepted ==EFI_INVALID_TABLE_ADDR ) +return NULL ; +return __va (efi.unaccepted); +} +#endif
Signature · 1 line
2.39.3
[tip: x86/cc] x86/tdx: Add unaccepted memory support
The following commit has been merged into the x86/cc branch of tip:
The following commit has been merged into the x86/cc branch of tip:
Commit-ID: 75d090fd167acab4d7eda7e2b65729e877c0fd64 Gitweb: https://git.kernel.org/tip/75d090fd167acab4d7eda7e2b65729e877c0fd64 Author: Kirill A. Shutemov <[email protected]> AuthorDate: Tue, 06 Jun 2023 17:26:37 +03:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 06 Jun 2023 18:25:57 +02:00
x86/tdx: Add unaccepted memory support
Hookup TDX-specific code to accept memory.
Accepting the memory is done with ACCEPT_PAGE module call on every page in the range. MAP_GPA hypercall is not required as the unaccepted memory is considered private already.
Extract the part of tdx_enc_status_changed() that does memory acceptance in a new helper. Move the helper tdx-shared.c. It is going to be used by both main kernel and decompressor.
[ bp: Fix the INTEL_TDX_GUEST=y, KVM_GUEST=n build. ]
--- arch/x86/Kconfig | 2 +- arch/x86/boot/compressed/Makefile | 2 +- arch/x86/boot/compressed/error.c | 19 ++++++- arch/x86/boot/compressed/error.h | 1 +- arch/x86/boot/compressed/mem.c | 35 ++++++++++- arch/x86/boot/compressed/tdx-shared.c | 2 +- arch/x86/coco/tdx/Makefile | 2 +- arch/x86/coco/tdx/tdx-shared.c | 71 +++++++++++++++++++++++- arch/x86/coco/tdx/tdx.c | 70 +----------------------- arch/x86/include/asm/shared/tdx.h | 2 +- arch/x86/include/asm/tdx.h | 2 +- arch/x86/include/asm/unaccepted_memory.h | 24 ++++++++- 12 files changed, 162 insertions(+), 70 deletions(-) create mode 100644 arch/x86/boot/compressed/tdx-shared.c create mode 100644 arch/x86/coco/tdx/tdx-shared.c create mode 100644 arch/x86/include/asm/unaccepted_memory.h
arch/x86/Kconfig +2 -0
@@ -884,9 +884,11 @@ config INTEL_TDX_GUEST bool "Intel TDX (Trust Domain Extensions) - Guest Support" depends on X86_64 && CPU_SUP_INTEL depends on X86_X2APIC + depends on EFI_STUB select ARCH_HAS_CC_PLATFORM select X86_MEM_ENCRYPT select X86_MCE + select UNACCEPTED_MEMORY help Support running as a guest under Intel TDX. Without this support, the guest kernel can not boot or run under TDX.
arch/x86/boot/compressed/Makefile +1 -1
@@ -106,7 +106,7 @@ ifdef CONFIG_X86_64 endif vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o -vmlinux-objs-$(CONFIG_INTEL_TDX_GUEST) += $(obj)/tdx.o $(obj)/tdcall.o +vmlinux-objs-$(CONFIG_INTEL_TDX_GUEST) += $(obj)/tdx.o $(obj)/tdcall.o $(obj)/tdx-shared.o vmlinux-objs-$(CONFIG_UNACCEPTED_MEMORY) += $(obj)/mem.o vmlinux-objs-$(CONFIG_EFI) += $(obj)/efi.o
arch/x86/boot/compressed/error.c +19 -0
@@ -22,3 +22,22 @@ void error(char *m)while (1 ) asm("hlt" ); } + +/* EFI libstub provides vsnprintf() */ +#ifdef CONFIG_EFI_STUB +void panic (const char *fmt, ...) +{ +static char buf[1024 ]; +va_list args; +int len; + +va_start (args, fmt); + len =vsnprintf (buf,sizeof (buf), fmt, args); +va_end (args); + +if (len && buf[len -1 ] =='\n' ) + buf[len -1 ] ='\0' ; + +error (buf); +} +#endif
arch/x86/boot/compressed/error.h +1 -0
@@ -6,5 +6,6 @@void warn (char *m);void error (char *m)__noreturn ; +void panic (const char *fmt, ...)__noreturn __cold ;#endif /* BOOT_COMPRESSED_ERROR_H */
arch/x86/boot/compressed/mem.c +34 -1
@@ -2,11 +2,44 @@#include "error.h" #include "misc.h" +#include "tdx.h" +#include <asm/shared/tdx.h> + +/* +* accept_memory() and process_unaccepted_memory() called from EFI stub which +* runs before decompresser and its early_tdx_detect(). +* +* Enumerate TDX directly from the early users. +*/ +static bool early_is_tdx_guest (void ) +{ +static bool once; +static bool is_tdx; + +if (!IS_ENABLED (CONFIG_INTEL_TDX_GUEST )) +return false; + +if (!once) { +u32 eax, sig[3 ]; + +cpuid_count (TDX_CPUID_LEAF_ID ,0 , &eax, + &sig[0 ], &sig[2 ], &sig[1 ]); + is_tdx = !memcmp (TDX_IDENT , sig,sizeof (sig)); + once = true; + } + +return is_tdx; +}void arch_accept_memory (phys_addr_t start,phys_addr_t end) {/* Platform-specific memory-acceptance call goes here */ -error ("Cannot accept memory" ); +if (early_is_tdx_guest ()) { +if (!tdx_accept_memory (start, end)) +panic ("TDX: Failed to accept memory\n" ); + }else { +error ("Cannot accept memory: unknown platform\n" ); + } }bool init_unaccepted_memory (void )
arch/x86/boot/compressed/tdx-shared.c new file +2 -0
@@ -0,0 +1,2 @@ +#include "error.h" +#include "../../coco/tdx/tdx-shared.c"
arch/x86/coco/tdx/Makefile +1 -1
@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += tdx.o tdcall.o +obj-y += tdx.o tdx-shared.o tdcall.o
arch/x86/coco/tdx/tdx-shared.c new file +71 -0
71 lines not on this page · raw message
arch/x86/coco/tdx/tdx.c +3 -67
82 lines not on this page · raw message
arch/x86/include/asm/shared/tdx.h +2 -0
@@ -91,5 +91,7 @@ struct tdx_module_output {u64 __tdx_module_call (u64 fn,u64 rcx,u64 rdx,u64 r8,u64 r9,struct tdx_module_output *out); +bool tdx_accept_memory (phys_addr_t start,phys_addr_t end); +#endif /* !__ASSEMBLY__ */ #endif /* _ASM_X86_SHARED_TDX_H */
arch/x86/include/asm/tdx.h +2 -0
@@ -5,6 +5,8 @@#include <linux/init.h> #include <linux/bits.h> + +#include <asm/errno.h> #include <asm/ptrace.h> #include <asm/shared/tdx.h>
arch/x86/include/asm/unaccepted_memory.h new file +24 -0
@@ -0,0 +1,24 @@ +#ifndef _ASM_X86_UNACCEPTED_MEMORY_H +#define _ASM_X86_UNACCEPTED_MEMORY_H + +#include <linux/efi.h> +#include <asm/tdx.h> + +static inline void arch_accept_memory (phys_addr_t start,phys_addr_t end) +{ +/* Platform-specific memory-acceptance call goes here */ +if (cpu_feature_enabled (X86_FEATURE_TDX_GUEST )) { +if (!tdx_accept_memory (start, end)) +panic ("TDX: Failed to accept memory\n" ); + }else { +panic ("Cannot accept memory: unknown platform\n" ); + } +} + +static inline struct efi_unaccepted_memory *efi_get_unaccepted_table (void ) +{ +if (efi.unaccepted ==EFI_INVALID_TABLE_ADDR ) +return NULL ; +return __va (efi.unaccepted); +} +#endif
v9 cover Provide SEV-SNP support for unaccepted memory
In reply to Kirill A. ShutemovThis series adds SEV-SNP support for unaccepted memory to the patch series titled:
[PATCHv14 0/9] mm, x86/cc, efi: Implement support for unaccepted memory
Currently, when changing the state of a page under SNP, the page state change structure is kmalloc()'d. This leads to hangs during boot when accepting memory because the allocation can trigger the need to accept more memory. Additionally, the page state change operations are not optimized under Linux since it was expected that all memory has been validated already, resulting in poor performance when adding basic support for unaccepted memory.
This series consists of six patches:
- One pre-patch fix which can be taken regardless of this series.
- A pre-patch to switch from a kmalloc()'d page state change structure to a (smaller) stack-based page state change structure.
- A pre-patch to allow the use of the early boot GHCB in the core kernel path.
- A pre-patch to allow for use of 2M page state change requests and 2M page validation.
- SNP support for unaccepted memory.
- EFI support to inform EFI/OVMF to not accept all memory on exit boot services (from Dionna Glaze <[email protected]>)
The series is based off of and tested against Kirill Shutemov's tree:
https://github.com/intel/tdx.git guest-unaccepted-memory
---
Changes since v8:
- Rebase on updated unaccepted memory series
- Incorporate minor comment to simplify boolean variable assignment
Changes since v7:
- Rebase on updated unaccepted memory series
Changes since v6:
- Add missing Kconfig dependency on EFI_STUB.
- Include an EFI patch to tell OVMF to not accept all memory on the exit boot services call (from Dionna Glaze <[email protected]>)
Changes since v5:
- Replace the two fix patches with a single fix patch that uses a new function signature to address the problem instead of using casting.
- Adjust the #define WARN definition in arch/x86/kernel/sev-shared.c to allow use from arch/x86/boot/compressed path without moving them outside of the if statements.
Changes since v4:
- Two fixes for when an unsigned int used as the number of pages to process, it needs to be converted to an unsigned long before being used to calculate ending addresses, otherwise a value >= 0x100000 results in adding 0 in the calculation.
- Commit message and comment updates.
Changes since v3:
- Reworks the PSC process to greatly improve performance:
- Optimize the PSC process to use 2M pages when applicable.
- Optimize the page validation process to use 2M pages when applicable.
- Use the early GHCB in both the decompression phase and core kernel boot phase in order to minimize the use of the MSR protocol. The MSR protocol only allows for a single 4K page to be updated at a time.
- Move the ghcb_percpu_ready flag into the sev_config structure and rename it to ghcbs_initialized.
Changes since v2:
- Improve code comments in regards to when to use the per-CPU GHCB vs the MSR protocol and why a single global value is valid for both the BSP and APs.
- Add a comment related to the number of PSC entries and how it can impact the size of the struct and, therefore, stack usage.
- Add a WARN_ON_ONCE() for invoking vmgexit_psc() when per-CPU GHCBs haven't been created or registered, yet.
- Use the compiler support for clearing the PSC struct instead of issuing memset().
Changes since v1:
- Change from using a per-CPU PSC structure to a (smaller) stack PSC structure.
Dionna Glaze (1):
x86/efi: Safely enable unaccepted memory in UEFI
Tom Lendacky (5):
x86/sev: Fix calculation of end address based on number of pages x86/sev: Put PSC struct on the stack in prep for unaccepted memory
support
x86/sev: Allow for use of the early boot GHCB for PSC requests x86/sev: Use large PSC requests if applicable
x86/sev: Add SNP-specific unaccepted memory support
arch/x86/Kconfig | 2 + arch/x86/boot/compressed/mem.c | 3 + arch/x86/boot/compressed/sev.c | 54 ++++- arch/x86/boot/compressed/sev.h | 23 ++ arch/x86/include/asm/sev-common.h | 9 +- arch/x86/include/asm/sev.h | 23 +- arch/x86/include/asm/unaccepted_memory.h | 3 + arch/x86/kernel/sev-shared.c | 103 +++++++++ arch/x86/kernel/sev.c | 256 ++++++++++------------- drivers/firmware/efi/libstub/x86-stub.c | 36 ++++ include/linux/efi.h | 3 + 11 files changed, 356 insertions(+), 159 deletions(-) create mode 100644 arch/x86/boot/compressed/sev.h
Signature · 1 line
2.40.1
PATCH v9 1/6 x86/sev: Fix calculation of end address based on number of pages
When calculating an end address based on an unsigned int number of pages, any value greater than or equal to 0x100000 that is shift PAGE_SHIFT bits results in a 0 value, resulting in an invalid end address. Change the number of pages variable in various routines from an unsigned int to an unsigned long to calculate the end address correctly.
--- arch/x86/include/asm/sev.h | 16 ++++++++-------- arch/x86/kernel/sev.c | 14 +++++++------- 2 files changed, 15 insertions(+), 15 deletions(-)
arch/x86/include/asm/sev.h +8 -8
@@ -192,12 +192,12 @@ struct snp_guest_request_ioctl;void setup_ghcb (void );void __init early_snp_set_memory_private (unsigned long vaddr,unsigned long paddr, -unsigned int npages); +unsigned long npages);void __init early_snp_set_memory_shared (unsigned long vaddr,unsigned long paddr, -unsigned int npages); +unsigned long npages);void __init snp_prep_memory (unsigned long paddr,unsigned int sz,enum psc_op op); -void snp_set_memory_shared (unsigned long vaddr,unsigned int npages); -void snp_set_memory_private (unsigned long vaddr,unsigned int npages); +void snp_set_memory_shared (unsigned long vaddr,unsigned long npages); +void snp_set_memory_private (unsigned long vaddr,unsigned long npages);void snp_set_wakeup_secondary_cpu (void );bool snp_init (struct boot_params *bp);void __init __noreturn snp_abort (void );
@@ -212,12 +212,12 @@ static inline int pvalidate(unsigned long vaddr, bool rmp_psize, bool validate)static inline int rmpadjust (unsigned long vaddr,bool rmp_psize,unsigned long attrs) {return 0 ; }static inline void setup_ghcb (void ) { }static inline void __init -early_snp_set_memory_private (unsigned long vaddr,unsigned long paddr,unsigned int npages) { } +early_snp_set_memory_private (unsigned long vaddr,unsigned long paddr,unsigned long npages) { }static inline void __init -early_snp_set_memory_shared (unsigned long vaddr,unsigned long paddr,unsigned int npages) { } +early_snp_set_memory_shared (unsigned long vaddr,unsigned long paddr,unsigned long npages) { }static inline void __init snp_prep_memory (unsigned long paddr,unsigned int sz,enum psc_op op) { } -static inline void snp_set_memory_shared (unsigned long vaddr,unsigned int npages) { } -static inline void snp_set_memory_private (unsigned long vaddr,unsigned int npages) { } +static inline void snp_set_memory_shared (unsigned long vaddr,unsigned long npages) { } +static inline void snp_set_memory_private (unsigned long vaddr,unsigned long npages) { }static inline void snp_set_wakeup_secondary_cpu (void ) { }static inline bool snp_init (struct boot_params *bp) {return false; }static inline void snp_abort (void ) { }
arch/x86/kernel/sev.c +7 -7
@@ -645,7 +645,7 @@ static u64 __init get_jump_table_addr(void)return ret; } -static void pvalidate_pages (unsigned long vaddr,unsigned int npages,bool validate) +static void pvalidate_pages (unsigned long vaddr,unsigned long npages,bool validate) {unsigned long vaddr_end;int rc;
@@ -662,7 +662,7 @@ static void pvalidate_pages(unsigned long vaddr, unsigned int npages, bool valid } } -static void __init early_set_pages_state (unsigned long paddr,unsigned int npages,enum psc_op op) +static void __init early_set_pages_state (unsigned long paddr,unsigned long npages,enum psc_op op) {unsigned long paddr_end;u64 val;
@@ -701,7 +701,7 @@ static void __init early_set_pages_state(unsigned long paddr, unsigned int npage }void __init early_snp_set_memory_private (unsigned long vaddr,unsigned long paddr, -unsigned int npages) +unsigned long npages) {/* * This can be invoked in early boot while running identity mapped, so
@@ -723,7 +723,7 @@ void __init early_snp_set_memory_private(unsigned long vaddr, unsigned long padd }void __init early_snp_set_memory_shared (unsigned long vaddr,unsigned long paddr, -unsigned int npages) +unsigned long npages) {/* * This can be invoked in early boot while running identity mapped, so
@@ -879,7 +879,7 @@ static void __set_pages_state(struct snp_psc_desc *data, unsigned long vaddr,sev_es_terminate (SEV_TERM_SET_LINUX ,GHCB_TERM_PSC ); } -static void set_pages_state (unsigned long vaddr,unsigned int npages,int op) +static void set_pages_state (unsigned long vaddr,unsigned long npages,int op) {unsigned long vaddr_end, next_vaddr;struct snp_psc_desc *desc;
@@ -904,7 +904,7 @@ static void set_pages_state(unsigned long vaddr, unsigned int npages, int op)kfree (desc); } -void snp_set_memory_shared (unsigned long vaddr,unsigned int npages) +void snp_set_memory_shared (unsigned long vaddr,unsigned long npages) {if (!cc_platform_has (CC_ATTR_GUEST_SEV_SNP ))return ;
@@ -914,7 +914,7 @@ void snp_set_memory_shared(unsigned long vaddr, unsigned int npages)set_pages_state (vaddr, npages,SNP_PAGE_STATE_SHARED ); } -void snp_set_memory_private (unsigned long vaddr,unsigned int npages) +void snp_set_memory_private (unsigned long vaddr,unsigned long npages) {if (!cc_platform_has (CC_ATTR_GUEST_SEV_SNP ))return ;
Signature · 1 line
2.40.1
PATCH v9 2/6 x86/sev: Put PSC struct on the stack in prep for unaccepted memory support
In reply to Tom LendackyIn advance of providing support for unaccepted memory, switch from using kmalloc() for allocating the Page State Change (PSC) structure to using a local variable that lives on the stack. This is needed to avoid a possible recursive call into set_pages_state() if the kmalloc() call requires (more) memory to be accepted, which would result in a hang.
The current size of the PSC struct is 2,032 bytes. To make the struct more stack friendly, reduce the number of PSC entries from 253 down to 64, resulting in a size of 520 bytes. This is a nice compromise on struct size and total PSC requests while still allowing parallel PSC operations across vCPUs.
If the reduction in PSC entries results in any kind of performance issue (that is not seen at the moment), use of a larger static PSC struct, with fallback to the smaller stack version, can be investigated.
For more background info on this decision, see the subthread in the Link:
tag below.
--- arch/x86/include/asm/sev-common.h | 9 +++++++-- arch/x86/kernel/sev.c | 10 ++-------- 2 files changed, 9 insertions(+), 10 deletions(-)
arch/x86/include/asm/sev-common.h +7 -2
@@ -106,8 +106,13 @@ enum psc_op {#define GHCB_HV_FT_SNP BIT_ULL(0)#define GHCB_HV_FT_SNP_AP_CREATION BIT_ULL(1) -/* SNP Page State Change NAE event */ -#define VMGEXIT_PSC_MAX_ENTRY 253 +/* +* SNP Page State Change NAE event +* The VMGEXIT_PSC_MAX_ENTRY determines the size of the PSC structure, which +* is a local stack variable in set_pages_state(). Do not increase this value +* without evaluating the impact to stack usage. +*/ +#define VMGEXIT_PSC_MAX_ENTRY 64struct psc_hdr {u16 cur_entry;
arch/x86/kernel/sev.c +2 -8
@@ -882,11 +882,7 @@ static void __set_pages_state(struct snp_psc_desc *data, unsigned long vaddr,static void set_pages_state (unsigned long vaddr,unsigned long npages,int op) {unsigned long vaddr_end, next_vaddr; -struct snp_psc_desc *desc; - - desc =kmalloc (sizeof (*desc),GFP_KERNEL_ACCOUNT ); -if (!desc) -panic ("SNP: failed to allocate memory for PSC descriptor\n" ); +struct snp_psc_desc desc; vaddr = vaddr &PAGE_MASK ; vaddr_end = vaddr + (npages <<PAGE_SHIFT );
@@ -896,12 +892,10 @@ static void set_pages_state(unsigned long vaddr, unsigned long npages, int op) next_vaddr = min_t(unsigned long, vaddr_end, (VMGEXIT_PSC_MAX_ENTRY *PAGE_SIZE ) + vaddr); -__set_pages_state (desc, vaddr, next_vaddr, op); +__set_pages_state (&desc, vaddr, next_vaddr, op); vaddr = next_vaddr; } - -kfree (desc); }void snp_set_memory_shared (unsigned long vaddr,unsigned long npages)
Signature · 1 line
2.40.1
PATCH v9 3/6 x86/sev: Allow for use of the early boot GHCB for PSC requests
In reply to Tom LendackyUsing a GHCB for a page stage change (as opposed to the MSR protocol) allows for multiple pages to be processed in a single request. In prep for early PSC requests in support of unaccepted memory, update the invocation of vmgexit_psc() to be able to use the early boot GHCB and not just the per-CPU GHCB structure.
In order to use the proper GHCB (early boot vs per-CPU), set a flag that indicates when the per-CPU GHCBs are available and registered. For APs, the per-CPU GHCBs are created before they are started and registered upon startup, so this flag can be used globally for the BSP and APs instead of creating a per-CPU flag. This will allow for a significant reduction in the number of MSR protocol page state change requests when accepting memory.
--- arch/x86/kernel/sev.c | 61 +++++++++++++++++++++++++++---------------- 1 file changed, 38 insertions(+), 23 deletions(-)
arch/x86/kernel/sev.c +38 -23
@@ -119,7 +119,19 @@ static DEFINE_PER_CPU(struct sev_es_save_area *, sev_vmsa);struct sev_config {__u64 debug :1 , - __reserved :63 ; + +/* +* A flag used by __set_pages_state() that indicates when the +* per-CPU GHCB has been created and registered and thus can be +* used by the BSP instead of the early boot GHCB. +* +* For APs, the per-CPU GHCB is created before they are started +* and registered upon startup, so this flag can be used globally +* for the BSP and APs. +*/ + ghcbs_initialized :1 , + + __reserved :62 ; };static struct sev_config sev_cfg__read_mostly ;
@@ -662,7 +674,7 @@ static void pvalidate_pages(unsigned long vaddr, unsigned long npages, bool vali } } -static void __init early_set_pages_state (unsigned long paddr,unsigned long npages,enum psc_op op) +static void early_set_pages_state (unsigned long paddr,unsigned long npages,enum psc_op op) {unsigned long paddr_end;u64 val;
@@ -756,26 +768,13 @@ void __init snp_prep_memory(unsigned long paddr, unsigned int sz, enum psc_op opWARN (1 ,"invalid memory op %d\n" , op); } -static int vmgexit_psc (struct snp_psc_desc *desc) +static int vmgexit_psc (struct ghcb *ghcb,struct snp_psc_desc *desc) {int cur_entry, end_entry, ret =0 ;struct snp_psc_desc *data; -struct ghcb_state state;struct es_em_ctxt ctxt; -unsigned long flags; -struct ghcb *ghcb; -/* -* __sev_get_ghcb() needs to run with IRQs disabled because it is using -* a per-CPU GHCB. -*/ -local_irq_save (flags); - - ghcb =__sev_get_ghcb (&state); -if (!ghcb) { - ret =1 ; - goto out_unlock; - } +vc_ghcb_invalidate (ghcb);/* Copy the input desc into GHCB shared buffer */ data = (struct snp_psc_desc *)ghcb->shared_buffer;
@@ -832,20 +831,18 @@ static int vmgexit_psc(struct snp_psc_desc *desc) } out: -__sev_put_ghcb (&state); - -out_unlock: -local_irq_restore (flags); -return ret; }static void __set_pages_state (struct snp_psc_desc *data,unsigned long vaddr,unsigned long vaddr_end,int op) { +struct ghcb_state state;struct psc_hdr *hdr;struct psc_entry *e; +unsigned long flags;unsigned long pfn; +struct ghcb *ghcb;int i; hdr = &data->hdr;
@@ -875,8 +872,20 @@ static void __set_pages_state(struct snp_psc_desc *data, unsigned long vaddr, i++; } -if (vmgexit_psc (data)) +local_irq_save (flags); + +if (sev_cfg.ghcbs_initialized) + ghcb =__sev_get_ghcb (&state); +else + ghcb = boot_ghcb; + +if (!ghcb ||vmgexit_psc (ghcb, data))sev_es_terminate (SEV_TERM_SET_LINUX ,GHCB_TERM_PSC ); + +if (sev_cfg.ghcbs_initialized) +__sev_put_ghcb (&state); + +local_irq_restore (flags); }static void set_pages_state (unsigned long vaddr,unsigned long npages,int op)
@@ -884,6 +893,10 @@ static void set_pages_state(unsigned long vaddr, unsigned long npages, int op)unsigned long vaddr_end, next_vaddr;struct snp_psc_desc desc; +/* Use the MSR protocol when a GHCB is not available. */ +if (!boot_ghcb) +return early_set_pages_state (__pa (vaddr), npages, op); + vaddr = vaddr &PAGE_MASK ; vaddr_end = vaddr + (npages <<PAGE_SHIFT );
@@ -1261,6 +1274,8 @@ void setup_ghcb(void)if (cc_platform_has (CC_ATTR_GUEST_SEV_SNP ))snp_register_per_cpu_ghcb (); + sev_cfg.ghcbs_initialized = true; +return ; }
Signature · 1 line
2.40.1
PATCH v9 4/6 x86/sev: Use large PSC requests if applicable
In reply to Tom LendackyIn advance of providing support for unaccepted memory, request 2M Page State Change (PSC) requests when the address range allows for it. By using a 2M page size, more PSC operations can be handled in a single request to the hypervisor. The hypervisor will determine if it can accommodate the larger request by checking the mapping in the nested page table. If mapped as a large page, then the 2M page request can be performed, otherwise the 2M page request will be broken down into 512 4K page requests. This is still more efficient than having the guest perform multiple PSC requests in order to process the 512 4K pages.
In conjunction with the 2M PSC requests, attempt to perform the associated PVALIDATE instruction of the page using the 2M page size. If PVALIDATE fails with a size mismatch, then fallback to validating 512 4K pages. To do this, page validation is modified to work with the PSC structure and not just a virtual address range.
--- arch/x86/include/asm/sev.h | 4 ++ arch/x86/kernel/sev.c | 125 ++++++++++++++++++++++++------------- 2 files changed, 84 insertions(+), 45 deletions(-)
arch/x86/include/asm/sev.h +4 -0
@@ -80,11 +80,15 @@ extern void vc_no_ghcb(void);extern void vc_boot_ghcb (void );extern bool handle_vc_boot_ghcb (struct pt_regs *regs); +/* PVALIDATE return codes */ +#define PVALIDATE_FAIL_SIZEMISMATCH 6 +/* Software defined (when rFlags.CF = 1) */ #define PVALIDATE_FAIL_NOUPDATE 255/* RMP page size */ #define RMP_PG_SIZE_4K 0 +#define RMP_PG_SIZE_2M 1#define RMPADJUST_VMSA_PAGE_BIT BIT(16)
arch/x86/kernel/sev.c +80 -45
228 lines not on this page · raw message
Signature · 1 line
2.40.1
PATCH v9 5/6 x86/sev: Add SNP-specific unaccepted memory support
In reply to Tom LendackyAdd SNP-specific hooks to the unaccepted memory support in the boot path (__accept_memory()) and the core kernel (accept_memory()) in order to support booting SNP guests when unaccepted memory is present. Without this support, SNP guests will fail to boot and/or panic() when unaccepted memory is present in the EFI memory map.
The process of accepting memory under SNP involves invoking the hypervisor to perform a page state change for the page to private memory and then issuing a PVALIDATE instruction to accept the page.
Since the boot path and the core kernel paths perform similar operations, move the pvalidate_pages() and vmgexit_psc() functions into sev-shared.c to avoid code duplication.
Create the new header file arch/x86/boot/compressed/sev.h because adding the function declaration to any of the existing SEV related header files pulls in too many other header files, causing the build to fail.
--- arch/x86/Kconfig | 2 + arch/x86/boot/compressed/mem.c | 3 + arch/x86/boot/compressed/sev.c | 54 ++++++++++- arch/x86/boot/compressed/sev.h | 23 +++++ arch/x86/include/asm/sev.h | 3 + arch/x86/include/asm/unaccepted_memory.h | 3 + arch/x86/kernel/sev-shared.c | 103 +++++++++++++++++++++ arch/x86/kernel/sev.c | 112 +++-------------------- 8 files changed, 204 insertions(+), 99 deletions(-) create mode 100644 arch/x86/boot/compressed/sev.h
arch/x86/Kconfig +2 -0
@@ -1543,11 +1543,13 @@ config X86_MEM_ENCRYPT config AMD_MEM_ENCRYPT bool "AMD Secure Memory Encryption (SME) support" depends on X86_64 && CPU_SUP_AMD + depends on EFI_STUB select DMA_COHERENT_POOL select ARCH_USE_MEMREMAP_PROT select INSTRUCTION_DECODER select ARCH_HAS_CC_PLATFORM select X86_MEM_ENCRYPT + select UNACCEPTED_MEMORY help Say yes to enable support for the encryption of system memory. This requires an AMD processor that supports Secure Memory
arch/x86/boot/compressed/mem.c +3 -0
@@ -3,6 +3,7 @@#include "error.h" #include "misc.h" #include "tdx.h" +#include "sev.h" #include <asm/shared/tdx.h> /*
@@ -37,6 +38,8 @@ void arch_accept_memory(phys_addr_t start, phys_addr_t end)if (early_is_tdx_guest ()) {if (!tdx_accept_memory (start, end))panic ("TDX: Failed to accept memory\n" ); + }else if (sev_snp_enabled ()) { +snp_accept_memory (start, end); }else {error ("Cannot accept memory: unknown platform\n" ); }
arch/x86/boot/compressed/sev.c +53 -1
66 lines not on this page · raw message
arch/x86/boot/compressed/sev.h new file +23 -0
@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* AMD SEV header for early boot related functions. +* +* Author: Tom Lendacky <[email protected]> +*/ + +#ifndef BOOT_COMPRESSED_SEV_H +#define BOOT_COMPRESSED_SEV_H + +#ifdef CONFIG_AMD_MEM_ENCRYPT + +bool sev_snp_enabled (void ); +void snp_accept_memory (phys_addr_t start,phys_addr_t end); + +#else + +static inline bool sev_snp_enabled (void ) {return false; } +static inline void snp_accept_memory (phys_addr_t start,phys_addr_t end) { } + +#endif + +#endif
arch/x86/include/asm/sev.h +3 -0
@@ -206,6 +206,7 @@ void snp_set_wakeup_secondary_cpu(void);bool snp_init (struct boot_params *bp);void __init __noreturn snp_abort (void );int snp_issue_guest_request (u64 exit_code,struct snp_req_data *input,struct snp_guest_request_ioctl *rio); +void snp_accept_memory (phys_addr_t start,phys_addr_t end);#else static inline void sev_es_ist_enter (struct pt_regs *regs) { }static inline void sev_es_ist_exit (void ) { }
@@ -229,6 +230,8 @@ static inline int snp_issue_guest_request(u64 exit_code, struct snp_req_data *in {return -ENOTTY ; } + +static inline void snp_accept_memory (phys_addr_t start,phys_addr_t end) { }#endif #endif
arch/x86/include/asm/unaccepted_memory.h +3 -0
@@ -3,6 +3,7 @@#include <linux/efi.h> #include <asm/tdx.h> +#include <asm/sev.h> static inline void arch_accept_memory (phys_addr_t start,phys_addr_t end) {
@@ -10,6 +11,8 @@ static inline void arch_accept_memory(phys_addr_t start, phys_addr_t end)if (cpu_feature_enabled (X86_FEATURE_TDX_GUEST )) {if (!tdx_accept_memory (start, end))panic ("TDX: Failed to accept memory\n" ); + }else if (cc_platform_has (CC_ATTR_GUEST_SEV_SNP )) { +snp_accept_memory (start, end); }else {panic ("Cannot accept memory: unknown platform\n" ); }
arch/x86/kernel/sev-shared.c +103 -0
112 lines not on this page · raw message
arch/x86/kernel/sev.c +14 -98
130 lines not on this page · raw message
Signature · 1 line
2.40.1
Re: [PATCH v9 5/6] x86/sev: Add SNP-specific unaccepted memory support
Quoted 8 lines
arch/x86/Kconfig +0
-0@@ -1543,11 +1543,13 @@ config X86_MEM_ENCRYPT config AMD_MEM_ENCRYPT bool "AMD Secure Memory Encryption (SME) support" depends on X86_64 && CPU_SUP_AMD
arch/x86/Kconfig config X86_MEM_ENCRYPT · line 1555
Quoted 10 lines
arch/x86/Kconfig +2
-0+ depends on EFI_STUB select DMA_COHERENT_POOL select ARCH_USE_MEMREMAP_PROT select INSTRUCTION_DECODER select ARCH_HAS_CC_PLATFORM select X86_MEM_ENCRYPT + select UNACCEPTED_MEMORY help Say yes to enable support for the encryption of system memory. This requires an AMD processor that supports Secure Memory
Unfortunately this makes AMD_MEM_ENCRYPT depend on EFI just to unconditionally enable UNACCEPTED_MEMORY. It seems like an easy target to make that optional, e.g. with a separate configuration item:
---
config AMD_UNACCEPTED_MEMORY
def_bool y
depends on AMD_MEM_ENCRYPT && EFI_STUB
select UNACCEPTED_MEMORY---
Using that we can successfully build and run SNP VMs without UEFI/OVMF (which we already did with earlier Linux versions).
From a quick look at
[PATCHv14 9/9] x86/tdx: Add unaccepted memory support
it actually seems very similar for INTEL_TDX_GUEST.
Ideally UNACCEPTED_MEMORY would not assume EFI either, but the implementation actually clearly does.
Re: [PATCH v9 5/6] x86/sev: Add SNP-specific unaccepted memory support
Quoted 21 lines · Christopher Schramm
Quoted 10 lines
arch/x86/Kconfig +0
-0@@ -1543,11 +1543,13 @@ config X86_MEM_ENCRYPTconfig
AMD_MEM_ENCRYPT
bool "AMD Secure Memory Encryption (SME) support" depends onX86_64&&CPU_SUP_AMD
+ depends onEFI_STUB
selectDMA_COHERENT_POOLQuoted 8 lines
select
ARCH_USE_MEMREMAP_PROT
selectINSTRUCTION_DECODER
selectARCH_HAS_CC_PLATFORM
selectX86_MEM_ENCRYPT
+ selectUNACCEPTED_MEMORY
help
Say yes to enable support for the encryption of system memory. This requires an AMD processor that supports Secure MemoryUnfortunately this makes
AMD_MEM_ENCRYPTdepend on EFI just to unconditionally enableUNACCEPTED_MEMORY. It seems like an easy target to make that optional, e.g. with a separate configuration item:
Quoted 8 lines
---
configAMD_UNACCEPTED_MEMORY
def_booly
depends onAMD_MEM_ENCRYPT&&EFI_STUB
selectUNACCEPTED_MEMORY
---Using that we can successfully build and run SNP VMs without UEFI/OVMF (which we already did with earlier Linux versions).
This seems reasonable to me.
I would recommend naming it AMD_MEM_ENCRYPT_UNACCEPTED_MEMORY to keep the association and add a prompt along the lines of "AMD Secure Encrypted Virtualization (SEV) unaccepted memory support" (I don't really see this used with bare-metal, but who knows.)
I think there will need to be changes elsewhere in the kernel to support this, though. A quick build test with just AMD_UNACCEPTED_MEMORY not being set broke the build when EFI_STUB was still set to y.
Quoted 8 lines
From a quick look at
[PATCHv14 9/9] x86/tdx: Add unaccepted memory support
it actually seems very similar for
INTEL_TDX_GUEST.Ideally
UNACCEPTED_MEMORYwould not assume EFI either, but the implementation actually clearly does.
@Kirill, is this something you are interested in having as well?
Thanks,
Tom
Re: [PATCH v9 5/6] x86/sev: Add SNP-specific unaccepted memory support
Quoted 10 lines · Tom Lendacky
Quoted 8 lines
From a quick look at
[PATCHv14 9/9] x86/tdx: Add unaccepted memory support
it actually seems very similar for
INTEL_TDX_GUEST.Ideally
UNACCEPTED_MEMORYwould not assume EFI either, but the implementation actually clearly does.@Kirill, is this something you are interested in having as well?
Unaccepted memory is an EFI feature I don't see how UNACCEPTED_MEMORY can be untied from EFI. If there's other (non-EFI) environment that has similar concept, sure we can try to generalize it beyond EFI.
TDX guest is only runs with EFI firmware so far, so depending onf EFI and EFI_STUB is fine for TDX>
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCH v9 5/6] x86/sev: Add SNP-specific unaccepted memory support
Quoted 7 lines · Kirill A. Shutemov
Quoted 6 lines · Tom Lendacky
Quoted 6 lines
From a quick look at
[PATCHv14 9/9] x86/tdx: Add unaccepted memory support
it actually seems very similar for
INTEL_TDX_GUEST.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Ideally
UNACCEPTED_MEMORYwould not assume EFI either, but the implementation actually clearly does.@Kirill, is this something you are interested in having as well?
Unaccepted memory is an EFI feature I don't see how
UNACCEPTED_MEMORYcan be untied from EFI. If there's other (non-EFI) environment that has similar concept, sure we can try to generalize it beyond EFI.
Sorry, didn't mean to include the EFI related statement there. Agreed that unaccepted memory is only an EFI feature right now.
Quoted 3 lines
TDX guest is only runs with EFI firmware so far, so depending onf EFI and
EFI_STUBis fine for TDX>
Right, SEV initially only ran on EFI firmware, but others have managed to get it working in other environments. So I was just wondering if you would also want to split the UNACCEPTED_MEMORY out for TDX similar to what was suggested for SEV.
Thanks,
Tom
Quoted 1 line
PATCH v9 6/6 x86/efi: Safely enable unaccepted memory in UEFI
In reply to Tom LendackyFrom: Dionna Glaze <[email protected]>
The UEFI v2.9 specification includes a new memory type to be used in environments where the OS must accept memory that is provided from its host. Before the introduction of this memory type, all memory was accepted eagerly in the firmware. In order for the firmware to safely stop accepting memory on the OS's behalf, the OS must affirmatively indicate support to the firmware. This is only a problem for AMD SEV-SNP, since Linux has had support for it since 5.19. The other technology that can make use of unaccepted memory, Intel TDX, does not yet have Linux support, so it can strictly require unaccepted memory support as a dependency of CONFIG_TDX and not require communication with the firmware.
Enabling unaccepted memory requires calling a 0-argument enablement protocol before ExitBootServices. This call is only made if the kernel is compiled with UNACCEPTED_MEMORY=y
This protocol will be removed after the end of life of the first LTS that includes it, in order to give firmware implementations an expiration date for it. When the protocol is removed, firmware will strictly infer that a SEV-SNP VM is running an OS that supports the unaccepted memory type. At the earliest convenience, when unaccepted memory support is added to Linux, SEV-SNP may take strict dependence in it. After the firmware removes support for the protocol, this patch should be reverted.
[tl: address some checkscript warnings]
--- drivers/firmware/efi/libstub/x86-stub.c | 36 +++++++++++++++++++++++++ include/linux/efi.h | 3 +++ 2 files changed, 39 insertions(+)
drivers/firmware/efi/libstub/x86-stub.c +36 -0
@@ -26,6 +26,17 @@ const efi_dxe_services_table_t *efi_dxe_table; u32 image_offset __section(".data");static efi_loaded_image_t *image =NULL ; +typedef union sev_memory_acceptance_protocol sev_memory_acceptance_protocol_t ; +union sev_memory_acceptance_protocol { +struct { + efi_status_t (__efiapi * allow_unaccepted_memory)( + sev_memory_acceptance_protocol_t *); + }; +struct { +u32 allow_unaccepted_memory; + } mixed_mode; +}; +static efi_status_t preserve_pci_rom_image (efi_pci_io_protocol_t *pci,struct pci_setup_rom **__rom) {
@@ -310,6 +321,29 @@ setup_memory_protection(unsigned long image_base, unsigned long image_size)#endif } +static void setup_unaccepted_memory (void ) +{ +efi_guid_t mem_acceptance_proto =OVMF_SEV_MEMORY_ACCEPTANCE_PROTOCOL_GUID ; +sev_memory_acceptance_protocol_t *proto; +efi_status_t status; + +if (!IS_ENABLED (CONFIG_UNACCEPTED_MEMORY )) +return ; + +/* +* Enable unaccepted memory before calling exit boot services in order +* for the UEFI to not accept all memory on EBS. +*/ + status =efi_bs_call (locate_protocol, &mem_acceptance_proto,NULL , + (void **)&proto); +if (status !=EFI_SUCCESS ) +return ; + + status =efi_call_proto (proto, allow_unaccepted_memory); +if (status !=EFI_SUCCESS ) +efi_err ("Memory acceptance protocol failed\n" ); +} +static const efi_char16_t apple[] =L"Apple" ; static void setup_quirks(struct boot_params *boot_params,
@@ -908,6 +942,8 @@ asmlinkage unsigned long efi_main(efi_handle_t handle,setup_quirks (boot_params, bzimage_addr, buffer_end - buffer_start); +setup_unaccepted_memory (); + status =exit_boot (boot_params, handle);if (status !=EFI_SUCCESS ) {efi_err ("exit_boot() failed!\n" );
include/linux/efi.h +3 -0
@@ -437,6 +437,9 @@ void efi_native_runtime_setup(void);#define DELLEMC_EFI_RCI2_TABLE_GUID EFI_GUID(0x2d9f28a2, 0xa886, 0x456a, 0x97, 0xa8, 0xf1, 0x1e, 0xf2, 0x4f, 0xf4, 0x55)#define AMD_SEV_MEM_ENCRYPT_GUID EFI_GUID(0x0cf29b71, 0x9e51, 0x433a, 0xa3, 0xb7, 0x81, 0xf3, 0xab, 0x16, 0xb8, 0x75) +/* OVMF protocol GUIDs */ +#define OVMF_SEV_MEMORY_ACCEPTANCE_PROTOCOL_GUID EFI_GUID(0xc5a010fe, 0x38a7, 0x4531, 0x8a, 0x4a, 0x05, 0x00, 0xd2, 0xfd, 0x16, 0x49) + typedef struct { efi_guid_t guid; u64 table;
Signature · 1 line
2.40.1
Re: [PATCH v9 6/6] x86/efi: Safely enable unaccepted memory in UEFI
Quoted 28 lines · Tom Lendacky
From: Dionna Glaze <[email protected]>
The UEFI v2.9 specification includes a new memory type to be used in environments where the OS must accept memory that is provided from its host. Before the introduction of this memory type, all memory was accepted eagerly in the firmware. In order for the firmware to safely stop accepting memory on the OS's behalf, the OS must affirmatively indicate support to the firmware. This is only a problem for AMD SEV-SNP, since Linux has had support for it since 5.19. The other technology that can make use of unaccepted memory, Intel TDX, does not yet have Linux support, so it can strictly require unaccepted memory support as a dependency of
CONFIG_TDXand not require communication with the firmware.Enabling unaccepted memory requires calling a 0-argument enablement protocol before ExitBootServices. This call is only made if the kernel is compiled with UNACCEPTED_MEMORY=y
This protocol will be removed after the end of life of the first LTS that includes it, in order to give firmware implementations an expiration date for it. When the protocol is removed, firmware will strictly infer that a SEV-SNP VM is running an OS that supports the unaccepted memory type. At the earliest convenience, when unaccepted memory support is added to Linux, SEV-SNP may take strict dependence in it. After the firmware removes support for the protocol, this patch should be reverted.
Quoted 4 lines
[tl: address some checkscript warnings]
Signed-off-by Dionna Glaze <[email protected]>Signed-off-by Tom Lendacky <[email protected]>
Quoted 83 lines
--- drivers/firmware/efi/libstub/x86-stub.c | 36 +++++++++++++++++++++++++ include/linux/efi.h | 3 +++ 2 files changed, 39 insertions(+)drivers/firmware/efi/libstub/x86-stub.c +36
-0@@ -26,6 +26,17 @@ const efi_dxe_services_table_t *efi_dxe_table; u32 image_offset __section(".data");static efi_loaded_image_t *image =NULL ; +typedef union sev_memory_acceptance_protocol sev_memory_acceptance_protocol_t ; +union sev_memory_acceptance_protocol { +struct { + efi_status_t (__efiapi * allow_unaccepted_memory)( + sev_memory_acceptance_protocol_t *); + }; +struct { +u32 allow_unaccepted_memory; + } mixed_mode; +}; +static efi_status_t preserve_pci_rom_image (efi_pci_io_protocol_t *pci,struct pci_setup_rom **__rom) {@@ -310,6 +321,29 @@ setup_memory_protection(unsigned long image_base, unsigned long image_size)#endif } +static void setup_unaccepted_memory (void ) +{ +efi_guid_t mem_acceptance_proto =OVMF_SEV_MEMORY_ACCEPTANCE_PROTOCOL_GUID ; +sev_memory_acceptance_protocol_t *proto; +efi_status_t status; + +if (!IS_ENABLED (CONFIG_UNACCEPTED_MEMORY )) +return ; + +/* +* Enable unaccepted memory before calling exit boot services in order +* for the UEFI to not accept all memory on EBS. +*/ + status =efi_bs_call (locate_protocol, &mem_acceptance_proto,NULL , + (void **)&proto); +if (status !=EFI_SUCCESS ) +return ; + + status =efi_call_proto (proto, allow_unaccepted_memory); +if (status !=EFI_SUCCESS ) +efi_err ("Memory acceptance protocol failed\n" ); +} +static const efi_char16_t apple[] =L"Apple" ; static void setup_quirks(struct boot_params *boot_params,@@ -908,6 +942,8 @@ asmlinkage unsigned long efi_main(efi_handle_t handle,setup_quirks (boot_params, bzimage_addr, buffer_end - buffer_start); +setup_unaccepted_memory (); + status =exit_boot (boot_params, handle);if (status !=EFI_SUCCESS ) {efi_err ("exit_boot() failed!\n" );include/linux/efi.h +3
-1@@ -437,6 +437,9 @@ void efi_native_runtime_setup(void);#define DELLEMC_EFI_RCI2_TABLE_GUID EFI_GUID(0x2d9f28a2, 0xa886, 0x456a, 0x97, 0xa8, 0xf1, 0x1e, 0xf2, 0x4f, 0xf4, 0x55)#define AMD_SEV_MEM_ENCRYPT_GUID EFI_GUID(0x0cf29b71, 0x9e51, 0x433a, 0xa3, 0xb7, 0x81, 0xf3, 0xab, 0x16, 0xb8, 0x75) +/* OVMF protocol GUIDs */ +#define OVMF_SEV_MEMORY_ACCEPTANCE_PROTOCOL_GUID EFI_GUID(0xc5a010fe, 0x38a7, 0x4531, 0x8a, 0x4a, 0x05, 0x00, 0xd2, 0xfd, 0x16, 0x49) + typedef struct { efi_guid_t guid; u64 table; --2.40.1
Borislav Petkov
maintainerQuoted 5 lines · Kirill A. Shutemov
v14:
- Fix error handling inarch_accept_memory()(Tom);
- Address Borislav's feedback:
+ code restructure;
+ added/adjusted comments;
In file included from arch/x86/coco/tdx/tdx-shared.c:1:./arch/x86/include/asm/tdx.h: In function ‘tdx_kvm_hypercall’:./arch/x86/include/asm/tdx.h:70:17: error: ‘ENODEV’ undeclared (first use in this function)
70 | return -ENODEV;
| ^~~~~~./arch/x86/include/asm/tdx.h:70:17: note: each undeclared identifier is reported only once for each function it appears in make[4]: *** [scripts/Makefile.build:252: arch/x86/coco/tdx/tdx-shared.o] Error 1
make[3]: *** [scripts/Makefile.build:494: arch/x86/coco/tdx] Error 2
make[2]: *** [scripts/Makefile.build:494: arch/x86/coco] Error 2
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [scripts/Makefile.build:494: arch/x86] Error 2
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:2026: .] Error 2
Not enough build tests ran?
$ grep INTEL_TDX_GUEST .config CONFIG_INTEL_TDX_GUEST=y $ grep KVM_GUEST .config $
Why does that tdx_kvm_hypercall() thing even depend on CONFIG_KVM_GUEST?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteBorislav Petkov
maintainerQuoted 1 line · Borislav Petkov
Why does that
tdx_kvm_hypercall()thing even depend onCONFIG_KVM_GUEST?
Amending with this, obviously:
---
arch/x86/include/asm/tdx.h +2 -0
@@ -5,6 +5,8 @@#include <linux/init.h> #include <linux/bits.h> + +#include <asm/errno.h> #include <asm/ptrace.h> #include <asm/shared/tdx.h>
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 21 lines · Borislav Petkov
Quoted 5 lines · Kirill A. Shutemov
v14:
- Fix error handling inarch_accept_memory()(Tom);
- Address Borislav's feedback:
+ code restructure;
+ added/adjusted comments;In file included from
arch/x86/coco/tdx/tdx-shared.c:1:./arch/x86/include/asm/tdx.h: In function ‘tdx_kvm_hypercall’:./arch/x86/include/asm/tdx.h:70:17: error: ‘ENODEV’ undeclared (first use in this function)70 | return -ENODEV; | ^~~~~~
./arch/x86/include/asm/tdx.h:70:17: note: each undeclared identifier is reported only once for each function it appears in make[4]: *** [scripts/Makefile.build:252:arch/x86/coco/tdx/tdx-shared.o] Error 1
make[3]: *** [scripts/Makefile.build:494:arch/x86/coco/tdx] Error 2
make[2]: *** [scripts/Makefile.build:494:arch/x86/coco] Error 2
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [scripts/Makefile.build:494:arch/x86] Error 2
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:2026: .] Error 2
Quoted 1 line
Not enough build tests ran?
Hm. I've got a lot of reports from 0day, but never this one.
Kai has posted patch that fixes it already:
https://lore.kernel.org/all/[email protected]/
Quoted 6 lines
$ grep INTEL_TDX_GUEST .config CONFIG_INTEL_TDX_GUEST=y $ grep KVM_GUEST .config $Why does that
tdx_kvm_hypercall()thing even depend onCONFIG_KVM_GUEST?
Because nobody uses it otherwise. For instance, Hyper-V guest will no need this KVM glue.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov