mm, x86/cc: Implement support for unaccepted memory
Subsystems (16)
v7 cover mm, x86/cc: Implement support for unaccepted memory
UEFI Specification version 2.9 introduces the concept of memory
acceptance: some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, requiring memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific for the Virtual Machine platform.
Accepting memory is costly and it makes VMM allocate memory for the accepted guest physical address range. It's better to postpone memory acceptance until memory is needed. It lowers boot time and reduces memory overhead.
The kernel needs to know what memory has been accepted. Firmware communicates this information via memory map: a new memory type -- EFI_UNACCEPTED_MEMORY -- indicates such memory.
Range-based tracking works fine for firmware, but it gets bulky for the kernel: e820 has to be modified on every page acceptance. It leads to table fragmentation, but there's a limited number of entries in the e820 table
Another option is to mark such memory as usable in e820 and track if the range has been accepted in a bitmap. One bit in the bitmap represents 2MiB in the address space: one 4k page is enough to track 64GiB or physical address space.
In the worst-case scenario -- a huge hole in the middle of the address space -- It needs 256MiB to handle 4PiB of the address space.
Any unaccepted memory that is not aligned to 2M gets accepted upfront.
The approach lowers boot time substantially. Boot to shell is ~2.5x faster for 4G TDX VM and ~4x faster for 64G.
TDX-specific code isolated from the core of unaccepted memory support. It supposed to help to plug-in different implementation of unaccepted memory such as SEV-SNP.
The tree can be found here:
https://github.com/intel/tdx.git guest-unaccepted-memory
v7:
- Rework meminfo counter to use PageUnaccepted() and move to generic code;
- Fix range_contains_unaccepted_memory() on machines without unaccepted memory;
- Add Reviewed-by from David;
v6:
- Fix load_unaligned_zeropad() on machine with unaccepted memory;
- Clear PageUnaccepted() on merged pages, leaving it only on head;
- Clarify error handling in allocate_e820();
- Fix build with CONFIG_UNACCEPTED_MEMORY=y, but without TDX;
- Disable kexec at boottime instead of build conflict;
- Rebased to tip/master;
- Spelling fixes;
- Add Reviewed-by from Mike and David;
v5:
- Updates comments and commit messages;
+ Explain options for unaccepted memory handling;
- Expose amount of unaccepted memory in /proc/meminfo
- Adjust check in page_expected_state();
- Fix error code handling in allocate_e820();
- Centralize __pa()/__va() definitions in the boot stub;
- Avoid includes from the main kernel in the boot stub;
- Use an existing hole in boot_param for unaccepted_memory, instead of adding to the end of the structure;
- Extract allocate_unaccepted_memory() form allocate_e820();
- Complain if there's unaccepted memory, but kernel does not support it;
- Fix vmstat counter;
- Split up few preparatory patches;
- Random readability adjustments;
v4:
- PageBuddyUnaccepted() -> PageUnaccepted;
- Use separate page_type, not shared with offline;
- Rework interface between core-mm and arch code;
- Adjust commit messages;
- Ack from Mike;
Kirill A. Shutemov (14):
x86/boot: Centralize __pa()/__va() definitions
mm: Add support for unaccepted memory
mm: Report unaccepted memory in meminfo
efi/x86: Get full memory map in allocate_e820()
x86/boot: Add infrastructure required for unaccepted memory support efi/x86: Implement support for unaccepted memory x86/boot/compressed: Handle unaccepted memory
x86/mm: Reserve unaccepted memory bitmap
x86/mm: Provide helpers for unaccepted memory
x86/mm: Avoid load_unaligned_zeropad() stepping into unaccepted memory
x86: Disable kexec if system has unaccepted memory x86/tdx: Make _tdx_hypercall() and __tdx_module_call() available in
boot stub
x86/tdx: Refactor try_accept_one()
x86/tdx: Add unaccepted memory support
Documentation/x86/zero-page.rst | 1 + arch/x86/Kconfig | 1 + arch/x86/boot/bitops.h | 40 ++++++++ arch/x86/boot/compressed/Makefile | 1 + arch/x86/boot/compressed/align.h | 14 +++ arch/x86/boot/compressed/bitmap.c | 43 ++++++++ arch/x86/boot/compressed/bitmap.h | 49 +++++++++ arch/x86/boot/compressed/bits.h | 36 +++++++ arch/x86/boot/compressed/compiler.h | 9 ++ arch/x86/boot/compressed/efi.h | 1 + arch/x86/boot/compressed/find.c | 54 ++++++++++ arch/x86/boot/compressed/find.h | 80 +++++++++++++++ arch/x86/boot/compressed/ident_map_64.c | 8 -- arch/x86/boot/compressed/kaslr.c | 35 ++++--- arch/x86/boot/compressed/math.h | 37 +++++++ arch/x86/boot/compressed/mem.c | 111 ++++++++++++++++++++ arch/x86/boot/compressed/minmax.h | 61 +++++++++++ arch/x86/boot/compressed/misc.c | 6 ++ arch/x86/boot/compressed/misc.h | 15 +++ arch/x86/boot/compressed/pgtable_types.h | 25 +++++ arch/x86/boot/compressed/sev.c | 2 - arch/x86/boot/compressed/tdx.c | 78 ++++++++++++++ arch/x86/coco/tdx/tdx.c | 94 ++++++++--------- arch/x86/include/asm/page.h | 3 + arch/x86/include/asm/shared/tdx.h | 47 +++++++++ arch/x86/include/asm/tdx.h | 19 ---- arch/x86/include/asm/unaccepted_memory.h | 16 +++ arch/x86/include/uapi/asm/bootparam.h | 2 +- arch/x86/kernel/e820.c | 10 ++ arch/x86/mm/Makefile | 2 + arch/x86/mm/unaccepted_memory.c | 123 +++++++++++++++++++++++ drivers/base/node.c | 7 ++ drivers/firmware/efi/Kconfig | 14 +++ drivers/firmware/efi/efi.c | 1 + drivers/firmware/efi/libstub/x86-stub.c | 103 ++++++++++++++++--- fs/proc/meminfo.c | 5 + include/linux/efi.h | 3 +- include/linux/mmzone.h | 1 + include/linux/page-flags.h | 31 ++++++ mm/internal.h | 12 +++ mm/memblock.c | 9 ++ mm/page_alloc.c | 96 +++++++++++++++++- mm/vmstat.c | 1 + 43 files changed, 1191 insertions(+), 115 deletions(-) create mode 100644 arch/x86/boot/compressed/align.h create mode 100644 arch/x86/boot/compressed/bitmap.c create mode 100644 arch/x86/boot/compressed/bitmap.h create mode 100644 arch/x86/boot/compressed/bits.h create mode 100644 arch/x86/boot/compressed/compiler.h create mode 100644 arch/x86/boot/compressed/find.c create mode 100644 arch/x86/boot/compressed/find.h create mode 100644 arch/x86/boot/compressed/math.h create mode 100644 arch/x86/boot/compressed/mem.c create mode 100644 arch/x86/boot/compressed/minmax.h create mode 100644 arch/x86/boot/compressed/pgtable_types.h create mode 100644 arch/x86/include/asm/unaccepted_memory.h create mode 100644 arch/x86/mm/unaccepted_memory.c
Signature · 1 line
2.35.1
PATCH v7 1/14 x86/boot: Centralize __pa()/__va() definitions
Replace multiple __pa()/__va() definitions with a single one in misc.h.
--- arch/x86/boot/compressed/ident_map_64.c | 8 -------- arch/x86/boot/compressed/misc.h | 9 +++++++++ arch/x86/boot/compressed/sev.c | 2 -- 3 files changed, 9 insertions(+), 10 deletions(-)
arch/x86/boot/compressed/ident_map_64.c +0 -8
@@ -8,14 +8,6 @@* Copyright (C) 2016 Kees Cook */ -/* -* Since we're dealing with identity mappings, physical and virtual -* addresses are the same, so override these defines which are ultimately -* used by the headers in misc.h. -*/ -#define __pa (x) ((unsigned long)(x)) -#define __va (x) ((void *)((unsigned long)(x))) -/* No PAGE_TABLE_ISOLATION support needed either: */ #undef CONFIG_PAGE_TABLE_ISOLATION
arch/x86/boot/compressed/misc.h +9 -0
@@ -19,6 +19,15 @@/* cpu_feature_enabled() cannot be used this early */ #define USE_EARLY_PGTABLE_L5 +/* +* Boot stub deals with identity mappings, physical and virtual addresses are +* the same, so override these defines. +* +* <asm/page.h> will not define them if they are already defined. +*/ +#define __pa (x) ((unsigned long)(x)) +#define __va (x) ((void *)((unsigned long)(x))) +#include <linux/linkage.h> #include <linux/screen_info.h> #include <linux/elf.h>
arch/x86/boot/compressed/sev.c +0 -2
@@ -104,9 +104,7 @@ static enum es_result vc_read_mem(struct es_em_ctxt *ctxt, }#undef __init -#undef __pa#define __init -#define __pa (x) ((unsigned long)(x))#define __BOOT_COMPRESSED
Signature · 1 line
2.35.1
Re: [PATCHv7 01/14] x86/boot: Centralize __pa()/__va() definitions
Quoted 10 lines · Kirill A. Shutemov
Replace multiple __pa()/__va() definitions with a single one in
misc.h.Signed-off-by Kirill A. Shutemov <[email protected]>Reviewed-by David Hildenbrand <[email protected]>Reviewed-by Mike Rapoport <[email protected]>--- arch/x86/boot/compressed/ident_map_64.c | 8 -------- arch/x86/boot/compressed/misc.h | 9 +++++++++ arch/x86/boot/compressed/sev.c | 2 -- 3 files changed, 9 insertions(+), 10 deletions(-)
PATCH v7 2/14 mm: Add support for unaccepted memory
In reply to Kirill A. ShutemovUEFI Specification version 2.9 introduces the concept of memory acceptance. Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, require memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific to the Virtual Machine platform.
There are several ways kernel can deal with unaccepted memory:
1. Accept all the memory during the boot. It is easy to implement and it doesn't have runtime cost once the system is booted. The downside is very long boot time.
Accept can be parallelized to multiple CPUs to keep it manageable (i.e. via DEFERRED_STRUCT_PAGE_INIT), but it tends to saturate memory bandwidth and does not scale beyond the point.
2. Accept a block of memory on the first use. It requires more infrastructure and changes in page allocator to make it work, but it provides good boot time.
On-demand memory accept means latency spikes every time kernel steps onto a new memory block. The spikes will go away once workload data set size gets stabilized or all memory gets accepted.
3. Accept all memory in background. Introduce a thread (or multiple) that gets memory accepted proactively. It will minimize time the system experience latency spikes on memory allocation while keeping low boot time.
This approach cannot function on its own. It is an extension of #2:
background memory acceptance requires functional scheduler, but the page allocator may need to tap into unaccepted memory before that.
The downside of the approach is that these threads also steal CPU cycles and memory bandwidth from the user's workload and may hurt user experience.
Implement #2 for now. It is a reasonable default. Some workloads may want to use #1 or #3 and they can be implemented later based on user's demands.
Support of unaccepted memory requires a few changes in core-mm code:
- memblock has to accept memory on allocation;
- page allocator has to accept memory on the first allocation of the
page;
Memblock change is trivial.
The page allocator is modified to accept pages on the first allocation. The new page type (encoded in the _mapcount) -- PageUnaccepted() -- is used to indicate that the page requires acceptance.
Architecture has to provide two helpers if it wants to support unaccepted memory:
- accept_memory() makes a range of physical addresses accepted.
- range_contains_unaccepted_memory() checks anything within the range of physical addresses requires acceptance.
--- include/linux/page-flags.h | 31 +++++++++++++ mm/internal.h | 12 +++++ mm/memblock.c | 9 ++++ mm/page_alloc.c | 89 +++++++++++++++++++++++++++++++++++++- 4 files changed, 139 insertions(+), 2 deletions(-)
include/linux/page-flags.h +31 -0
@@ -942,6 +942,14 @@ static inline bool is_page_hwpoison(struct page *page)#define PG_offline 0x00000100#define PG_table 0x00000200#define PG_guard 0x00000400 +#define PG_unaccepted 0x00000800 + +/* +* Page types allowed at page_expected_state() +* +* PageUnaccepted() will get cleared in post_alloc_hook(). +*/ +#define PAGE_TYPES_EXPECTED PG_unaccepted#define PageType (page, flag) \ ((page->page_type & (PAGE_TYPE_BASE | flag)) == PAGE_TYPE_BASE)
@@ -967,6 +975,18 @@ static __always_inline void __ClearPage##uname(struct page *page) \ page->page_type |= PG_##lname; \ } +#define PAGE_TYPE_OPS_FALSE (uname) \ +static __always_inline int Page##uname(struct page *page) \ +{ \ + return false; \ +} \ +static __always_inline void __SetPage##uname(struct page *page) \ +{ \ +} \ +static __always_inline void __ClearPage##uname(struct page *page) \ +{ \ +} +/* * PageBuddy() indicates that the page is free and in the buddy system * (see mm/page_alloc.c).
@@ -997,6 +1017,17 @@ PAGE_TYPE_OPS(Buddy, buddy)*/ PAGE_TYPE_OPS (Offline, offline) +/* +* PageUnaccepted() indicates that the page has to be "accepted" before it can +* be read or written. The page allocator must call accept_page() before +* touching the page or returning it to the caller. + */ +#ifdef CONFIG_UNACCEPTED_MEMORY +PAGE_TYPE_OPS(Unaccepted, unaccepted) +#else +PAGE_TYPE_OPS_FALSE(Unaccepted) +#endif +extern void page_offline_freeze (void );extern void page_offline_thaw (void );extern void page_offline_begin (void );
mm/internal.h +12 -0
@@ -861,4 +861,16 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags); DECLARE_PER_CPU(struct per_cpu_nodestat, boot_nodestats); +#ifndef CONFIG_UNACCEPTED_MEMORY +static inline bool range_contains_unaccepted_memory (phys_addr_t start, +phys_addr_t end) +{ +return false; +} + +static inline void accept_memory (phys_addr_t start,phys_addr_t end) +{ +} +#endif +#endif /* __MM_INTERNAL_H */
mm/memblock.c +9 -0
@@ -1405,6 +1405,15 @@ phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size,*/ kmemleak_alloc_phys (found, size,0 ,0 ); +/* +* Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, +* require memory to be accepted before it can be used by the +* guest. +* +* Accept the memory of the allocated buffer. +*/ +accept_memory (found, found + size); +return found; }
mm/page_alloc.c +87 -2
155 lines not on this page · raw message
Signature · 1 line
2.35.1
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 331 lines · Kirill A. Shutemov
UEFI Specification version 2.9 introduces the concept of memory acceptance. Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, require memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific to the Virtual Machine platform.
There are several ways kernel can deal with unaccepted memory:
1. Accept all the memory during the boot. It is easy to implement and it doesn't have runtime cost once the system is booted. The downside is very long boot time.
Accept can be parallelized to multiple CPUs to keep it manageable (i.e. via
DEFERRED_STRUCT_PAGE_INIT), but it tends to saturate memory bandwidth and does not scale beyond the point.2. Accept a block of memory on the first use. It requires more infrastructure and changes in page allocator to make it work, but it provides good boot time.
On-demand memory accept means latency spikes every time kernel steps onto a new memory block. The spikes will go away once workload data set size gets stabilized or all memory gets accepted.
3. Accept all memory in background. Introduce a thread (or multiple) that gets memory accepted proactively. It will minimize time the system experience latency spikes on memory allocation while keeping low boot time.
This approach cannot function on its own. It is an extension of #2:
background memory acceptance requires functional scheduler, but the page allocator may need to tap into unaccepted memory before that.The downside of the approach is that these threads also steal CPU cycles and memory bandwidth from the user's workload and may hurt user experience.
Implement #2 for now. It is a reasonable default. Some workloads may want to use #1 or #3 and they can be implemented later based on user's demands.
Support of unaccepted memory requires a few changes in core-mm code:
- memblock has to accept memory on allocation;
- page allocator has to accept memory on the first allocation of the
page;Memblock change is trivial.
The page allocator is modified to accept pages on the first allocation. The new page type (encoded in the
_mapcount) --PageUnaccepted()-- is used to indicate that the page requires acceptance.Architecture has to provide two helpers if it wants to support unaccepted memory:
-
accept_memory()makes a range of physical addresses accepted.-
range_contains_unaccepted_memory()checks anything within the range of physical addresses requires acceptance.Signed-off-by Kirill A. Shutemov <[email protected]>Reviewed-by David Hildenbrand <[email protected]>--- include/linux/page-flags.h | 31 +++++++++++++ mm/internal.h | 12 +++++ mm/memblock.c | 9 ++++ mm/page_alloc.c | 89 +++++++++++++++++++++++++++++++++++++- 4 files changed, 139 insertions(+), 2 deletions(-)include/linux/page-flags.h +31
-0@@ -942,6 +942,14 @@ static inline bool is_page_hwpoison(struct page *page)#define PG_offline 0x00000100#define PG_table 0x00000200#define PG_guard 0x00000400 +#define PG_unaccepted 0x00000800 + +/* +* Page types allowed at page_expected_state() +* +* PageUnaccepted() will get cleared in post_alloc_hook(). +*/ +#define PAGE_TYPES_EXPECTED PG_unaccepted#define PageType (page, flag) \ ((page->page_type & (PAGE_TYPE_BASE | flag)) == PAGE_TYPE_BASE)@@ -967,6 +975,18 @@ static __always_inline void __ClearPage##uname(struct page *page) \ page->page_type |= PG_##lname; \ } +#define PAGE_TYPE_OPS_FALSE (uname) \ +static __always_inline int Page##uname(struct page *page) \ +{ \ + return false; \ +} \ +static __always_inline void __SetPage##uname(struct page *page) \ +{ \ +} \ +static __always_inline void __ClearPage##uname(struct page *page) \ +{ \ +} +/* * PageBuddy() indicates that the page is free and in the buddy system * (see mm/page_alloc.c). @@ -997,6 +1017,17 @@ PAGE_TYPE_OPS(Buddy, buddy)*/ PAGE_TYPE_OPS (Offline, offline) +/* +* PageUnaccepted() indicates that the page has to be "accepted" before it can +* be read or written. The page allocator must call accept_page() before +* touching the page or returning it to the caller. + */ +#ifdef CONFIG_UNACCEPTED_MEMORY +PAGE_TYPE_OPS(Unaccepted, unaccepted) +#else +PAGE_TYPE_OPS_FALSE(Unaccepted) +#endif +extern void page_offline_freeze (void );extern void page_offline_thaw (void );extern void page_offline_begin (void );mm/internal.h +12
-0@@ -861,4 +861,16 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags); DECLARE_PER_CPU(struct per_cpu_nodestat, boot_nodestats); +#ifndef CONFIG_UNACCEPTED_MEMORY +static inline bool range_contains_unaccepted_memory (phys_addr_t start, +phys_addr_t end) +{ +return false; +} + +static inline void accept_memory (phys_addr_t start,phys_addr_t end) +{ +} +#endif +#endif /* __MM_INTERNAL_H */ mm/memblock.c +9
-0@@ -1405,6 +1405,15 @@ phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size,*/ kmemleak_alloc_phys (found, size,0 ,0 ); +/* +* Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, +* require memory to be accepted before it can be used by the +* guest. +* +* Accept the memory of the allocated buffer. +*/ +accept_memory (found, found + size); +return found; }mm/page_alloc.c +85
-2@@ -122,6 +122,12 @@ typedef int __bitwise fpi_t;*/ #define FPI_SKIP_KASAN_POISON ((__force fpi_t)BIT(2)) +/* +* Check if the page needs to be marked as PageUnaccepted(). +* Used for the new pages added to the buddy allocator for the first time. +*/ +#define FPI_UNACCEPTED_SLOWPATH ((__force fpi_t)BIT(3)) +/* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ static DEFINE_MUTEX (pcp_batch_high_lock);#define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8)@@ -993,6 +999,35 @@ buddy_merge_likely(unsigned long pfn, unsigned long buddy_pfn, NULL) != NULL; } +/* +* Page acceptance can be very slow. Do not call under critical locks. +*/ +static void accept_page (struct page *page,unsigned int order) +{ +phys_addr_t start =page_to_phys (page); +int i; + +if (!PageUnaccepted (page)) +return ; + +accept_memory (start, start + (PAGE_SIZE << order)); +__ClearPageUnaccepted (page); + +/* Assert that there is no PageUnaccepted() on tail pages */ +if (IS_ENABLED (CONFIG_DEBUG_VM )) { +for (i =1 ; i < (1 << order); i++) +VM_BUG_ON_PAGE (PageUnaccepted (page + i), page + i); + } +} + +static bool page_contains_unaccepted (struct page *page,unsigned int order) +{ +phys_addr_t start =page_to_phys (page); +phys_addr_t end = start + (PAGE_SIZE << order); + +return range_contains_unaccepted_memory (start, end); +} +/* * Freeing function for a buddy system allocator. * @@ -1027,6 +1062,7 @@ static inline void __free_one_page(struct page *page,unsigned long combined_pfn;struct page *buddy;bool to_tail; +bool page_needs_acceptance = false;VM_BUG_ON (!zone_is_initialized (zone));VM_BUG_ON_PAGE (page->flags &PAGE_FLAGS_CHECK_AT_PREP , page);@@ -1038,6 +1074,11 @@ static inline void __free_one_page(struct page *page,VM_BUG_ON_PAGE (pfn & ((1 << order) -1 ), page);VM_BUG_ON_PAGE (bad_range (zone, page), page); +if (PageUnaccepted (page)) { + page_needs_acceptance = true; +__ClearPageUnaccepted (page); + } +while (order <MAX_ORDER -1 ) {if (compaction_capture (capc, page, order, migratetype)) { __mod_zone_freepage_state(zone, -(1 << order),@@ -1072,6 +1113,13 @@ static inline void __free_one_page(struct page *page,clear_page_guard (zone, buddy, order, migratetype); elsedel_page_from_free_list (buddy ,zone ,order ); + +/* Mark page unaccepted if any of merged pages were unaccepted */ +if (PageUnaccepted (buddy)) { + page_needs_acceptance = true; +__ClearPageUnaccepted (buddy); + } + combined_pfn = buddy_pfn & pfn; page = page + (combined_pfn - pfn); pfn = combined_pfn;@@ -1081,6 +1129,23 @@ static inline void __free_one_page(struct page *page, done_merging:set_buddy_order (page, order); +/* +* The page gets marked as PageUnaccepted() if any of merged-in pages +* is PageUnaccepted(). +* +* New pages, just being added to buddy allocator, do not have +* PageUnaccepted() set. FPI_UNACCEPTED_SLOWPATH indicates that the +* page is new and page_contains_unaccepted() check is required to +* determinate if acceptance is required. +* +* Avoid calling page_contains_unaccepted() if it is known that the page +* needs acceptance. It can be costly. +*/ +if (!page_needs_acceptance && (fpi_flags &FPI_UNACCEPTED_SLOWPATH )) + page_needs_acceptance =page_contains_unaccepted (page, order); +if (page_needs_acceptance) +__SetPageUnaccepted (page); +if (fpi_flags &FPI_TO_TAIL ) to_tail = true;else if (is_shuffle_order (order))@@ -1164,7 +1229,13 @@ int split_free_page(struct page *free_page,static inline bool page_expected_state (struct page *page,unsigned long check_flags) { -if (unlikely (atomic_read (&page->_mapcount) !=-1 )) +/* +* The page must not be mapped to userspace and must not have +* a PageType other than listed in PAGE_TYPES_EXPECTED. +* +* Note, bit cleared means the page type is set. +*/ +if (unlikely ((atomic_read (&page->_mapcount) |PAGE_TYPES_EXPECTED ) !=-1 ))return false; if (unlikely((unsigned long)page->mapping |@@ -1669,7 +1740,9 @@ void __free_pages_core(struct page *page, unsigned int order)* Bypass PCP and place fresh pages right to the tail, primarily * relevant for memory onlining. */ -__free_pages_ok (page, order,FPI_TO_TAIL |FPI_SKIP_KASAN_POISON ); +__free_pages_ok (page, order, +FPI_TO_TAIL |FPI_SKIP_KASAN_POISON | +FPI_UNACCEPTED_SLOWPATH ); }#ifdef CONFIG_NUMA @@ -1822,6 +1895,9 @@ static void __init deferred_free_range(unsigned long pfn,return ; } +/* Accept chunks smaller than page-block upfront */ +accept_memory (PFN_PHYS (pfn),PFN_PHYS (pfn + nr_pages)); +for (i =0 ; i < nr_pages; i++, page++, pfn++) {if ((pfn & (pageblock_nr_pages -1 )) ==0 )set_pageblock_migratetype (page,MIGRATE_MOVABLE );@@ -2281,6 +2357,13 @@ static inline void expand(struct zone *zone, struct page *page,if (set_page_guard (zone, &page[size], high, migratetype))continue ; +/* +* Transfer PageUnaccepted() to the newly split pages so +* they can be accepted after dropping the zone lock. +*/ +if (PageUnaccepted (page)) +__SetPageUnaccepted (&page[size]); +add_to_free_list (&page[size], zone, high, migratetype);set_buddy_order (&page[size], high); }
mm/page_alloc.c inline void post_alloc_hook(struct page *page, unsigned int order, · line 2501
Quoted 8 lines
mm/page_alloc.c +2
-0@@ -2411,6 +2494,8 @@ inline void post_alloc_hook(struct page *page, unsigned int order,*/ kernel_unpoison_pages (page,1 << order); +accept_page (page, order); +/* * As memory initialization might be integrated into KASAN, * KASAN unpoisoning and memory initializion code must be
Reviewed previous version(v6) of this patch. Seems no change in this version except '!PageUnaccepted' check addition in function 'accept_page' and rename of function 'page_contains_unaccepted'.
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
In reply to Kirill A. ShutemovQuoted 79 lines · Kirill A. Shutemov
UEFI Specification version 2.9 introduces the concept of memory acceptance. Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, require memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific to the Virtual Machine platform.
There are several ways kernel can deal with unaccepted memory:
1. Accept all the memory during the boot. It is easy to implement and it doesn't have runtime cost once the system is booted. The downside is very long boot time.
Accept can be parallelized to multiple CPUs to keep it manageable (i.e. via
DEFERRED_STRUCT_PAGE_INIT), but it tends to saturate memory bandwidth and does not scale beyond the point.2. Accept a block of memory on the first use. It requires more infrastructure and changes in page allocator to make it work, but it provides good boot time.
On-demand memory accept means latency spikes every time kernel steps onto a new memory block. The spikes will go away once workload data set size gets stabilized or all memory gets accepted.
3. Accept all memory in background. Introduce a thread (or multiple) that gets memory accepted proactively. It will minimize time the system experience latency spikes on memory allocation while keeping low boot time.
This approach cannot function on its own. It is an extension of #2:
background memory acceptance requires functional scheduler, but the page allocator may need to tap into unaccepted memory before that.The downside of the approach is that these threads also steal CPU cycles and memory bandwidth from the user's workload and may hurt user experience.
Implement #2 for now. It is a reasonable default. Some workloads may want to use #1 or #3 and they can be implemented later based on user's demands.
Support of unaccepted memory requires a few changes in core-mm code:
- memblock has to accept memory on allocation;
- page allocator has to accept memory on the first allocation of the
page;Memblock change is trivial.
The page allocator is modified to accept pages on the first allocation. The new page type (encoded in the
_mapcount) --PageUnaccepted()-- is used to indicate that the page requires acceptance.Architecture has to provide two helpers if it wants to support unaccepted memory:
-
accept_memory()makes a range of physical addresses accepted.-
range_contains_unaccepted_memory()checks anything within the range of physical addresses requires acceptance.Signed-off-by Kirill A. Shutemov <[email protected]>Reviewed-by David Hildenbrand <[email protected]>--- include/linux/page-flags.h | 31 +++++++++++++ mm/internal.h | 12 +++++ mm/memblock.c | 9 ++++ mm/page_alloc.c | 89 +++++++++++++++++++++++++++++++++++++- 4 files changed, 139 insertions(+), 2 deletions(-)mm/memblock.c +0
-0@@ -1405,6 +1405,15 @@ phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size,*/
mm/memblock.c phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size, · line 1415
Quoted 10 lines
mm/memblock.c +8
-0kmemleak_alloc_phys (found, size,0 ,0 ); +/* +* Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, +* require memory to be accepted before it can be used by the +* guest. +* +* Accept the memory of the allocated buffer. +*/ +accept_memory (found, found + size);
The SNP support will kmalloc a descriptor that can be used to supply the range to the hypervisor using the GHCB/VMGEXIT. But kmalloc won't work when called this early, so we are likely to need an early_accept_memory or some kind of flag to know whether this is an early call or not in order to use a static descriptor in the file.
Thanks,
Tom
Quoted 4 lines
mm/memblock.c +1
-0+return found; }
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 84 lines · Tom Lendacky
Quoted 83 lines · Kirill A. Shutemov
UEFI Specification version 2.9 introduces the concept of memory acceptance. Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, require memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific to the Virtual Machine platform.
There are several ways kernel can deal with unaccepted memory:
1. Accept all the memory during the boot. It is easy to implement and it doesn't have runtime cost once the system is booted. The downside is very long boot time.
Accept can be parallelized to multiple CPUs to keep it manageable (i.e. via
DEFERRED_STRUCT_PAGE_INIT), but it tends to saturate memory bandwidth and does not scale beyond the point.2. Accept a block of memory on the first use. It requires more infrastructure and changes in page allocator to make it work, but it provides good boot time.
On-demand memory accept means latency spikes every time kernel steps onto a new memory block. The spikes will go away once workload data set size gets stabilized or all memory gets accepted.
3. Accept all memory in background. Introduce a thread (or multiple) that gets memory accepted proactively. It will minimize time the system experience latency spikes on memory allocation while keeping low boot time.
This approach cannot function on its own. It is an extension of #2:
background memory acceptance requires functional scheduler, but the page allocator may need to tap into unaccepted memory before that.The downside of the approach is that these threads also steal CPU cycles and memory bandwidth from the user's workload and may hurt user experience.
Implement #2 for now. It is a reasonable default. Some workloads may want to use #1 or #3 and they can be implemented later based on user's demands.
Support of unaccepted memory requires a few changes in core-mm code:
- memblock has to accept memory on allocation;
- page allocator has to accept memory on the first allocation of the page;
Memblock change is trivial.
The page allocator is modified to accept pages on the first allocation. The new page type (encoded in the
_mapcount) --PageUnaccepted()-- is used to indicate that the page requires acceptance.Architecture has to provide two helpers if it wants to support unaccepted memory:
-
accept_memory()makes a range of physical addresses accepted.-
range_contains_unaccepted_memory()checks anything within the range of physical addresses requires acceptance.Signed-off-by Kirill A. Shutemov <[email protected]>Reviewed-by David Hildenbrand <[email protected]>---
include/linux/page-flags.h| 31 +++++++++++++ mm/internal.h | 12 +++++
mm/memblock.c | 9 ++++
mm/page_alloc.c | 89 +++++++++++++++++++++++++++++++++++++-4 files changed, 139 insertions(+), 2 deletions(-)mm/memblock.c +2
-0@@ -1405,6 +1405,15 @@ phys_addr_t __initmemblock_alloc_range_nid(phys_addr_t size, */ kmemleak_alloc_phys(found, size, 0, 0); + /* +* Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP,
Quoted 8 lines
mm/memblock.c phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size, · line 1415
Quoted 6 lines
mm/memblock.c +6
-0+* require memory to be accepted before it can be used by the +* guest. +* +* Accept the memory of the allocated buffer. +*/ + accept_memory(found, found + size);The SNP support will kmalloc a descriptor that can be used to supply the range to the hypervisor using the GHCB/VMGEXIT. But kmalloc won't work when called this early, so we are likely to need an
early_accept_memoryor some kind of flag to know whether this is an early call or not in order to use a static descriptor in the file.
Then again, the accept_memory() call from mm/page_alloc.c can also be early... maybe I can check system_state and make a determination as to which SNP page state change mechanism to use (GHCB vs MSR protocol). It might not be performance optimal, though, as the MSR protocol is one 4K page at a time.
Thanks,
Tom
Quoted 7 lines
Thanks,
TomQuoted 3 lines
mm/memblock.c +1
-0+ return found; }
Borislav Petkov
maintainerRe: [PATCHv7 02/14] mm: Add support for unaccepted memory
In reply to Kirill A. ShutemovQuoted 3 lines · Kirill A. Shutemov
On-demand memory accept means latency spikes every time kernel steps onto a new memory block. The spikes will go away once workload data set size gets stabilized or all memory gets accepted.
What does that mean?
If we're accepting 2M pages and considering referential locality, how are those "spikes" even noticeable?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 8 lines · Borislav Petkov
Quoted 3 lines · Kirill A. Shutemov
On-demand memory accept means latency spikes every time kernel steps onto a new memory block. The spikes will go away once workload data set size gets stabilized or all memory gets accepted.
What does that mean?
If we're accepting 2M pages and considering referential locality, how are those "spikes" even noticeable?
Acceptance is slow and the heavy lifting is done inside the TDX module.
It involves flushing old aliases out of the caches and initializing the
memory integrity metadata for every cacheline. This implementation does acceptance in 2MB chunks while holding a global lock.
So, those (effective) 2MB clflush+memset's (plus a few thousand cycles for the hypercall/tdcall transitions) can't happen in parallel. They are serialized and must wait on each other. If you have a few hundred CPUs all trying to allocate memory (say, doing the first kernel compile after a reboot), this is going to be very, very painful for a while.
That said, I think this is the right place to _start_. There is going to need to be some kind of follow-on solution (likely background acceptance of some kind). But, even with that solution, *this* code is still needed to handle the degenerate case where the background accepter can't keep up with foreground memory needs.
Borislav Petkov
maintainerRe: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 4 lines · Dave Hansen
Acceptance is slow and the heavy lifting is done inside the TDX module.
It involves flushing old aliases out of the caches and initializing the
memory integrity metadata for every cacheline. This implementation does acceptance in 2MB chunks while holding a global lock.
Oh, fun.
Quoted 2 lines
So, those (effective) 2MB clflush+memset's (plus a few thousand cycles for the hypercall/tdcall transitions)
So this sounds strange - page validation on AMD - judging by the pseudocode of the PVALIDATE insn - does a bunch of sanity checks on the gVA of the page and then installs it into the RMP and also "PVALIDATE performs the same segmentation and paging checks as a 1-byte read. PVALIDATE does not invalidate TLB caches."
But that still sounds a lot less work than what the TDX module needs to do...
Quoted 2 lines
can't happen in parallel. They are serialized and must wait on each other.
Ofc, the Intel version of the RMP table needs to be protected. :-)
Quoted 9 lines
If you have a few hundred CPUs all trying to allocate memory (say, doing the first kernel compile after a reboot), this is going to be very, very painful for a while.
That said, I think this is the right place to _start_. There is going to need to be some kind of follow-on solution (likely background acceptance of some kind). But, even with that solution, *this* code is still needed to handle the degenerate case where the background accepter can't keep up with foreground memory needs.
I'm still catering to the view that it should be a two-tier thing: you validate during boot a certain amount - say 4G - a size for which the boot delay is acceptable and you do the rest on-demand along with a background accepter.
That should give you the best of both worlds...
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 4 lines · Borislav Petkov
Quoted 2 lines · Dave Hansen
So, those (effective) 2MB clflush+memset's (plus a few thousand cycles for the hypercall/tdcall transitions)
Quoted 8 lines
So this sounds strange - page validation on AMD - judging by the pseudocode of the PVALIDATE insn - does a bunch of sanity checks on the gVA of the page and then installs it into the RMP and also "PVALIDATE performs the same segmentation and paging checks as a 1-byte read. PVALIDATE does not invalidate TLB caches."
But that still sounds a lot less work than what the TDX module needs to do...
Sure does... *Something* has to manage the cache coherency so that old physical aliases of the converted memory don't write back and clobber new data. But, maybe the hardware is doing that now.
Quoted 10 lines
Quoted 9 lines
If you have a few hundred CPUs all trying to allocate memory (say, doing the first kernel compile after a reboot), this is going to be very, very painful for a while.
That said, I think this is the right place to _start_. There is going to need to be some kind of follow-on solution (likely background acceptance of some kind). But, even with that solution, *this* code is still needed to handle the degenerate case where the background accepter can't keep up with foreground memory needs.
Quoted 6 lines
I'm still catering to the view that it should be a two-tier thing: you validate during boot a certain amount - say 4G - a size for which the boot delay is acceptable and you do the rest on-demand along with a background accepter.
That should give you the best of both worlds...
Yeah, that two-tier system is the way it's happening today from what I understand. This whole conversation is about how to handle the >4GB memory.
Borislav Petkov
maintainerRe: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 3 lines · Dave Hansen
Sure does... *Something* has to manage the cache coherency so that old physical aliases of the converted memory don't write back and clobber new data. But, maybe the hardware is doing that now.
Let's hope.
Quoted 3 lines
Yeah, that two-tier system is the way it's happening today from what I understand. This whole conversation is about how to handle the >4GB memory.
Would it be possible to pre-accept a bunch of mem - think "pre-fault" - from userspace?
I.e., I'm thinking some huge process is going to start in the VM, VM userspace goes and causes a chunk of memory to be pre-accepted and then the process starts and runs more-or-less smoothly as the majority of its memory has already been "prepared".
Or does that not make any sense from mm perspective?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 14 lines · Borislav Petkov
Quoted 3 lines · Dave Hansen
Sure does... *Something* has to manage the cache coherency so that old physical aliases of the converted memory don't write back and clobber new data. But, maybe the hardware is doing that now.
Let's hope.
Quoted 3 lines
Yeah, that two-tier system is the way it's happening today from what I understand. This whole conversation is about how to handle the >4GB memory.
Would it be possible to pre-accept a bunch of mem - think "pre-fault" - from userspace?
Quoted 7 lines
I.e., I'm thinking some huge process is going to start in the VM, VM userspace goes and causes a chunk of memory to be pre-accepted and then the process starts and runs more-or-less smoothly as the majority of its memory has already been "prepared".
Or does that not make any sense from mm perspective?
The less core-MM code to handle unaccepted memory the better. Meaning, that any kind of additional pre-acceptance (in addition to what we have here) needs good justification.
Signature · 3 lines
Thanks, David / dhildenb
Borislav Petkov
maintainerRe: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 3 lines · David Hildenbrand
The less core-MM code to handle unaccepted memory the better. Meaning, that any kind of additional pre-acceptance (in addition to what we have here) needs good justification.
I actually mean to have a user interface to have the core code pre-accept. I.e., interface to what will be already there anyway.
Or is that frowned upon too?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 8 lines · Borislav Petkov
Quoted 3 lines · David Hildenbrand
The less core-MM code to handle unaccepted memory the better. Meaning, that any kind of additional pre-acceptance (in addition to what we have here) needs good justification.
I actually mean to have a user interface to have the core code pre-accept. I.e., interface to what will be already there anyway.
Ah, got it. The "issue" with ordinary prefaulting in user space is that you'll also end up zeroing the memory. But yeah, it should just accept a bunch of memory.
Quoted 1 line
Or is that frowned upon too?
I was assuming you'd want some additional interface that only accept memory without actually temporarily allocating it.
Signature · 3 lines
Thanks, David / dhildenb
Mike Rapoport
maintainerRe: [PATCHv7 02/14] mm: Add support for unaccepted memory
In reply to Borislav PetkovQuoted 11 lines · Borislav Petkov
Quoted 3 lines · Dave Hansen
Sure does... *Something* has to manage the cache coherency so that old physical aliases of the converted memory don't write back and clobber new data. But, maybe the hardware is doing that now.
Let's hope.
Quoted 3 lines
Yeah, that two-tier system is the way it's happening today from what I understand. This whole conversation is about how to handle the >4GB memory.
Quoted 7 lines
Would it be possible to pre-accept a bunch of mem - think "pre-fault" - from userspace?
I.e., I'm thinking some huge process is going to start in the VM, VM userspace goes and causes a chunk of memory to be pre-accepted and then the process starts and runs more-or-less smoothly as the majority of its memory has already been "prepared".
An application in the VM can do mlock() or mmap(..., MAP_POPULATE, ...) and this will essentially force acceptance of that memory.
But there's no sysctl or something for that.
Quoted 7 lines
Or does that not make any sense from mm perspective?
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Signature · 2 lines
Sincerely yours, Mike.
Borislav Petkov
maintainerRe: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 2 lines · Mike Rapoport
An application in the VM can do
mlock()ormmap(..., MAP_POPULATE, ...)and this will essentially force acceptance of that memory.
Ah, cool, that's what I meant.
Quoted 1 line
But there's no sysctl or something for that.
Yeah, no need.
I was simply wondering whether one can relocate the acceptance work to the moment prior to starting the process so that it can run smoothly once started and doesn't cause spikes due to on-demand acceptance.
At least not too many.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCHv7 02/14] mm: Add support for unaccepted memory
In reply to Kirill A. ShutemovQuoted 57 lines · Kirill A. Shutemov
UEFI Specification version 2.9 introduces the concept of memory acceptance. Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, require memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific to the Virtual Machine platform.
There are several ways kernel can deal with unaccepted memory:
1. Accept all the memory during the boot. It is easy to implement and it doesn't have runtime cost once the system is booted. The downside is very long boot time.
Accept can be parallelized to multiple CPUs to keep it manageable (i.e. via
DEFERRED_STRUCT_PAGE_INIT), but it tends to saturate memory bandwidth and does not scale beyond the point.2. Accept a block of memory on the first use. It requires more infrastructure and changes in page allocator to make it work, but it provides good boot time.
On-demand memory accept means latency spikes every time kernel steps onto a new memory block. The spikes will go away once workload data set size gets stabilized or all memory gets accepted.
3. Accept all memory in background. Introduce a thread (or multiple) that gets memory accepted proactively. It will minimize time the system experience latency spikes on memory allocation while keeping low boot time.
This approach cannot function on its own. It is an extension of #2:
background memory acceptance requires functional scheduler, but the page allocator may need to tap into unaccepted memory before that.The downside of the approach is that these threads also steal CPU cycles and memory bandwidth from the user's workload and may hurt user experience.
Implement #2 for now. It is a reasonable default. Some workloads may want to use #1 or #3 and they can be implemented later based on user's demands.
Support of unaccepted memory requires a few changes in core-mm code:
- memblock has to accept memory on allocation;
- page allocator has to accept memory on the first allocation of the
page;Memblock change is trivial.
The page allocator is modified to accept pages on the first allocation. The new page type (encoded in the
_mapcount) --PageUnaccepted()-- is used to indicate that the page requires acceptance.Architecture has to provide two helpers if it wants to support unaccepted memory:
Quoted 8 lines
-
accept_memory()makes a range of physical addresses accepted.-
range_contains_unaccepted_memory()checks anything within the range of physical addresses requires acceptance.Signed-off-by Kirill A. Shutemov <[email protected]>Reviewed-by David Hildenbrand <[email protected]>
Hmm I realize it's not ideal to raise this at v7, and maybe it was discussed before, but it's really not great how this affects the core page allocator paths. Wouldn't it be possible to only release pages to page allocator when accepted, and otherwise use some new per-zone variables together with the bitmap to track how much exactly is where to accept? Then it could be hooked in get_page_from_freelist() similarly to CONFIG_DEFERRED_STRUCT_PAGE_INIT - if we fail zone_watermark_fast() and there are unaccepted pages in the zone, accept them and continue. With a static key to flip in case we eventually accept everything. Because this is really similar scenario to the deferred init and that one was solved in a way that adds minimal overhead.
Quoted 274 lines
--- include/linux/page-flags.h | 31 +++++++++++++ mm/internal.h | 12 +++++ mm/memblock.c | 9 ++++ mm/page_alloc.c | 89 +++++++++++++++++++++++++++++++++++++- 4 files changed, 139 insertions(+), 2 deletions(-)include/linux/page-flags.h +31
-0@@ -942,6 +942,14 @@ static inline bool is_page_hwpoison(struct page *page)#define PG_offline 0x00000100#define PG_table 0x00000200#define PG_guard 0x00000400 +#define PG_unaccepted 0x00000800 + +/* +* Page types allowed at page_expected_state() +* +* PageUnaccepted() will get cleared in post_alloc_hook(). +*/ +#define PAGE_TYPES_EXPECTED PG_unaccepted#define PageType (page, flag) \ ((page->page_type & (PAGE_TYPE_BASE | flag)) == PAGE_TYPE_BASE)@@ -967,6 +975,18 @@ static __always_inline void __ClearPage##uname(struct page *page) \ page->page_type |= PG_##lname; \ } +#define PAGE_TYPE_OPS_FALSE (uname) \ +static __always_inline int Page##uname(struct page *page) \ +{ \ + return false; \ +} \ +static __always_inline void __SetPage##uname(struct page *page) \ +{ \ +} \ +static __always_inline void __ClearPage##uname(struct page *page) \ +{ \ +} +/* * PageBuddy() indicates that the page is free and in the buddy system * (see mm/page_alloc.c). @@ -997,6 +1017,17 @@ PAGE_TYPE_OPS(Buddy, buddy)*/ PAGE_TYPE_OPS (Offline, offline) +/* +* PageUnaccepted() indicates that the page has to be "accepted" before it can +* be read or written. The page allocator must call accept_page() before +* touching the page or returning it to the caller. + */ +#ifdef CONFIG_UNACCEPTED_MEMORY +PAGE_TYPE_OPS(Unaccepted, unaccepted) +#else +PAGE_TYPE_OPS_FALSE(Unaccepted) +#endif +extern void page_offline_freeze (void );extern void page_offline_thaw (void );extern void page_offline_begin (void );mm/internal.h +12
-0@@ -861,4 +861,16 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags); DECLARE_PER_CPU(struct per_cpu_nodestat, boot_nodestats); +#ifndef CONFIG_UNACCEPTED_MEMORY +static inline bool range_contains_unaccepted_memory (phys_addr_t start, +phys_addr_t end) +{ +return false; +} + +static inline void accept_memory (phys_addr_t start,phys_addr_t end) +{ +} +#endif +#endif /* __MM_INTERNAL_H */ mm/memblock.c +9
-0@@ -1405,6 +1405,15 @@ phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size,*/ kmemleak_alloc_phys (found, size,0 ,0 ); +/* +* Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, +* require memory to be accepted before it can be used by the +* guest. +* +* Accept the memory of the allocated buffer. +*/ +accept_memory (found, found + size); +return found; }mm/page_alloc.c +87
-2@@ -122,6 +122,12 @@ typedef int __bitwise fpi_t;*/ #define FPI_SKIP_KASAN_POISON ((__force fpi_t)BIT(2)) +/* +* Check if the page needs to be marked as PageUnaccepted(). +* Used for the new pages added to the buddy allocator for the first time. +*/ +#define FPI_UNACCEPTED_SLOWPATH ((__force fpi_t)BIT(3)) +/* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ static DEFINE_MUTEX (pcp_batch_high_lock);#define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8)@@ -993,6 +999,35 @@ buddy_merge_likely(unsigned long pfn, unsigned long buddy_pfn, NULL) != NULL; } +/* +* Page acceptance can be very slow. Do not call under critical locks. +*/ +static void accept_page (struct page *page,unsigned int order) +{ +phys_addr_t start =page_to_phys (page); +int i; + +if (!PageUnaccepted (page)) +return ; + +accept_memory (start, start + (PAGE_SIZE << order)); +__ClearPageUnaccepted (page); + +/* Assert that there is no PageUnaccepted() on tail pages */ +if (IS_ENABLED (CONFIG_DEBUG_VM )) { +for (i =1 ; i < (1 << order); i++) +VM_BUG_ON_PAGE (PageUnaccepted (page + i), page + i); + } +} + +static bool page_contains_unaccepted (struct page *page,unsigned int order) +{ +phys_addr_t start =page_to_phys (page); +phys_addr_t end = start + (PAGE_SIZE << order); + +return range_contains_unaccepted_memory (start, end); +} +/* * Freeing function for a buddy system allocator. * @@ -1027,6 +1062,7 @@ static inline void __free_one_page(struct page *page,unsigned long combined_pfn;struct page *buddy;bool to_tail; +bool page_needs_acceptance = false;VM_BUG_ON (!zone_is_initialized (zone));VM_BUG_ON_PAGE (page->flags &PAGE_FLAGS_CHECK_AT_PREP , page);@@ -1038,6 +1074,11 @@ static inline void __free_one_page(struct page *page,VM_BUG_ON_PAGE (pfn & ((1 << order) -1 ), page);VM_BUG_ON_PAGE (bad_range (zone, page), page); +if (PageUnaccepted (page)) { + page_needs_acceptance = true; +__ClearPageUnaccepted (page); + } +while (order <MAX_ORDER -1 ) {if (compaction_capture (capc, page, order, migratetype)) { __mod_zone_freepage_state(zone, -(1 << order),@@ -1072,6 +1113,13 @@ static inline void __free_one_page(struct page *page,clear_page_guard (zone, buddy, order, migratetype); elsedel_page_from_free_list (buddy ,zone ,order ); + +/* Mark page unaccepted if any of merged pages were unaccepted */ +if (PageUnaccepted (buddy)) { + page_needs_acceptance = true; +__ClearPageUnaccepted (buddy); + } + combined_pfn = buddy_pfn & pfn; page = page + (combined_pfn - pfn); pfn = combined_pfn;@@ -1081,6 +1129,23 @@ static inline void __free_one_page(struct page *page, done_merging:set_buddy_order (page, order); +/* +* The page gets marked as PageUnaccepted() if any of merged-in pages +* is PageUnaccepted(). +* +* New pages, just being added to buddy allocator, do not have +* PageUnaccepted() set. FPI_UNACCEPTED_SLOWPATH indicates that the +* page is new and page_contains_unaccepted() check is required to +* determinate if acceptance is required. +* +* Avoid calling page_contains_unaccepted() if it is known that the page +* needs acceptance. It can be costly. +*/ +if (!page_needs_acceptance && (fpi_flags &FPI_UNACCEPTED_SLOWPATH )) + page_needs_acceptance =page_contains_unaccepted (page, order); +if (page_needs_acceptance) +__SetPageUnaccepted (page); +if (fpi_flags &FPI_TO_TAIL ) to_tail = true;else if (is_shuffle_order (order))@@ -1164,7 +1229,13 @@ int split_free_page(struct page *free_page,static inline bool page_expected_state (struct page *page,unsigned long check_flags) { -if (unlikely (atomic_read (&page->_mapcount) !=-1 )) +/* +* The page must not be mapped to userspace and must not have +* a PageType other than listed in PAGE_TYPES_EXPECTED. +* +* Note, bit cleared means the page type is set. +*/ +if (unlikely ((atomic_read (&page->_mapcount) |PAGE_TYPES_EXPECTED ) !=-1 ))return false; if (unlikely((unsigned long)page->mapping |@@ -1669,7 +1740,9 @@ void __free_pages_core(struct page *page, unsigned int order)* Bypass PCP and place fresh pages right to the tail, primarily * relevant for memory onlining. */ -__free_pages_ok (page, order,FPI_TO_TAIL |FPI_SKIP_KASAN_POISON ); +__free_pages_ok (page, order, +FPI_TO_TAIL |FPI_SKIP_KASAN_POISON | +FPI_UNACCEPTED_SLOWPATH ); }#ifdef CONFIG_NUMA @@ -1822,6 +1895,9 @@ static void __init deferred_free_range(unsigned long pfn,return ; } +/* Accept chunks smaller than page-block upfront */ +accept_memory (PFN_PHYS (pfn),PFN_PHYS (pfn + nr_pages)); +for (i =0 ; i < nr_pages; i++, page++, pfn++) {if ((pfn & (pageblock_nr_pages -1 )) ==0 )set_pageblock_migratetype (page,MIGRATE_MOVABLE );@@ -2281,6 +2357,13 @@ static inline void expand(struct zone *zone, struct page *page,if (set_page_guard (zone, &page[size], high, migratetype))continue ; +/* +* Transfer PageUnaccepted() to the newly split pages so +* they can be accepted after dropping the zone lock. +*/ +if (PageUnaccepted (page)) +__SetPageUnaccepted (&page[size]); +add_to_free_list (&page[size], zone, high, migratetype);set_buddy_order (&page[size], high); }@@ -2411,6 +2494,8 @@ inline void post_alloc_hook(struct page *page, unsigned int order,*/ kernel_unpoison_pages (page,1 << order); +accept_page (page, order); +/* * As memory initialization might be integrated into KASAN, * KASAN unpoisoning and memory initializion code must be
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 67 lines · Vlastimil Babka
Quoted 57 lines · Kirill A. Shutemov
UEFI Specification version 2.9 introduces the concept of memory acceptance. Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, require memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific to the Virtual Machine platform.
There are several ways kernel can deal with unaccepted memory:
1. Accept all the memory during the boot. It is easy to implement and it doesn't have runtime cost once the system is booted. The downside is very long boot time.
Accept can be parallelized to multiple CPUs to keep it manageable (i.e. via
DEFERRED_STRUCT_PAGE_INIT), but it tends to saturate memory bandwidth and does not scale beyond the point.2. Accept a block of memory on the first use. It requires more infrastructure and changes in page allocator to make it work, but it provides good boot time.
On-demand memory accept means latency spikes every time kernel steps onto a new memory block. The spikes will go away once workload data set size gets stabilized or all memory gets accepted.
3. Accept all memory in background. Introduce a thread (or multiple) that gets memory accepted proactively. It will minimize time the system experience latency spikes on memory allocation while keeping low boot time.
This approach cannot function on its own. It is an extension of #2:
background memory acceptance requires functional scheduler, but the page allocator may need to tap into unaccepted memory before that.The downside of the approach is that these threads also steal CPU cycles and memory bandwidth from the user's workload and may hurt user experience.
Implement #2 for now. It is a reasonable default. Some workloads may want to use #1 or #3 and they can be implemented later based on user's demands.
Support of unaccepted memory requires a few changes in core-mm code:
- memblock has to accept memory on allocation;
- page allocator has to accept memory on the first allocation of the
page;Memblock change is trivial.
The page allocator is modified to accept pages on the first allocation. The new page type (encoded in the
_mapcount) --PageUnaccepted()-- is used to indicate that the page requires acceptance.Architecture has to provide two helpers if it wants to support unaccepted memory:
Quoted 8 lines
-
accept_memory()makes a range of physical addresses accepted.-
range_contains_unaccepted_memory()checks anything within the range of physical addresses requires acceptance.Signed-off-by Kirill A. Shutemov <[email protected]>Reviewed-by David Hildenbrand <[email protected]>
Quoted 10 lines
Hmm I realize it's not ideal to raise this at v7, and maybe it was discussed before, but it's really not great how this affects the core page allocator paths. Wouldn't it be possible to only release pages to page allocator when accepted, and otherwise use some new per-zone variables together with the bitmap to track how much exactly is where to accept? Then it could be hooked in
get_page_from_freelist()similarly toCONFIG_DEFERRED_STRUCT_PAGE_INIT- if we failzone_watermark_fast()and there are unaccepted pages in the zone, accept them and continue. With a static key to flip in case we eventually accept everything. Because this is really similar scenario to the deferred init and that one was solved in a way that adds minimal overhead.
I kind of like just having the memory stats being correct (e.g., free memory) and acceptance being an internal detail to be triggered when allocating pages -- just like the arch_alloc_page() callback.
I'm sure we could optimize for the !unaccepted memory via static keys also in this version with some checks at the right places if we find this to hurt performance?
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 79 lines · David Hildenbrand
Quoted 67 lines · Vlastimil Babka
Quoted 57 lines · Kirill A. Shutemov
UEFI Specification version 2.9 introduces the concept of memory acceptance. Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, require memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific to the Virtual Machine platform.
There are several ways kernel can deal with unaccepted memory:
1. Accept all the memory during the boot. It is easy to implement and it doesn't have runtime cost once the system is booted. The downside is very long boot time.
Accept can be parallelized to multiple CPUs to keep it manageable (i.e. via
DEFERRED_STRUCT_PAGE_INIT), but it tends to saturate memory bandwidth and does not scale beyond the point.2. Accept a block of memory on the first use. It requires more infrastructure and changes in page allocator to make it work, but it provides good boot time.
On-demand memory accept means latency spikes every time kernel steps onto a new memory block. The spikes will go away once workload data set size gets stabilized or all memory gets accepted.
3. Accept all memory in background. Introduce a thread (or multiple) that gets memory accepted proactively. It will minimize time the system experience latency spikes on memory allocation while keeping low boot time.
This approach cannot function on its own. It is an extension of #2:
background memory acceptance requires functional scheduler, but the page allocator may need to tap into unaccepted memory before that.The downside of the approach is that these threads also steal CPU cycles and memory bandwidth from the user's workload and may hurt user experience.
Implement #2 for now. It is a reasonable default. Some workloads may want to use #1 or #3 and they can be implemented later based on user's demands.
Support of unaccepted memory requires a few changes in core-mm code:
- memblock has to accept memory on allocation;
- page allocator has to accept memory on the first allocation of the
page;Memblock change is trivial.
The page allocator is modified to accept pages on the first allocation. The new page type (encoded in the
_mapcount) --PageUnaccepted()-- is used to indicate that the page requires acceptance.Architecture has to provide two helpers if it wants to support unaccepted memory:
Quoted 8 lines
-
accept_memory()makes a range of physical addresses accepted.-
range_contains_unaccepted_memory()checks anything within the range of physical addresses requires acceptance.Signed-off-by Kirill A. Shutemov <[email protected]>Reviewed-by David Hildenbrand <[email protected]>Quoted 10 lines
Hmm I realize it's not ideal to raise this at v7, and maybe it was discussed before, but it's really not great how this affects the core page allocator paths. Wouldn't it be possible to only release pages to page allocator when accepted, and otherwise use some new per-zone variables together with the bitmap to track how much exactly is where to accept? Then it could be hooked in
get_page_from_freelist()similarly toCONFIG_DEFERRED_STRUCT_PAGE_INIT- if we failzone_watermark_fast()and there are unaccepted pages in the zone, accept them and continue. With a static key to flip in case we eventually accept everything. Because this is really similar scenario to the deferred init and that one was solved in a way that adds minimal overhead.
Quoted 3 lines
I kind of like just having the memory stats being correct (e.g., free memory) and acceptance being an internal detail to be triggered when allocating pages -- just like the
arch_alloc_page()callback.
Hm, good point about the stats. Could be tweaked perhaps so it appears correct on the outside, but might be tricky.
Quoted 3 lines
I'm sure we could optimize for the !unaccepted memory via static keys also in this version with some checks at the right places if we find this to hurt performance?
It would be great if we would at least somehow hit the necessary code only when dealing with a >=pageblock size block. The bitmap approach and accepting everything smaller uprofront actually seems rather compatible. Yet in the current patch we e.g. check PageUnaccepted(buddy) on every buddy size while merging.
A list that sits besides the existing free_area, contains only >=pageblock order sizes of unaccepted pages (no migratetype distinguished) and we tap into it approximately before __rmqueue_fallback()? There would be some trickery around releasing zone-lock for doing accept_memory(), but should be manageable.
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 97 lines · Vlastimil Babka
Quoted 79 lines · David Hildenbrand
Quoted 67 lines · Vlastimil Babka
Quoted 57 lines · Kirill A. Shutemov
UEFI Specification version 2.9 introduces the concept of memory acceptance. Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, require memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific to the Virtual Machine platform.
There are several ways kernel can deal with unaccepted memory:
1. Accept all the memory during the boot. It is easy to implement and it doesn't have runtime cost once the system is booted. The downside is very long boot time.
Accept can be parallelized to multiple CPUs to keep it manageable (i.e. via
DEFERRED_STRUCT_PAGE_INIT), but it tends to saturate memory bandwidth and does not scale beyond the point.2. Accept a block of memory on the first use. It requires more infrastructure and changes in page allocator to make it work, but it provides good boot time.
On-demand memory accept means latency spikes every time kernel steps onto a new memory block. The spikes will go away once workload data set size gets stabilized or all memory gets accepted.
3. Accept all memory in background. Introduce a thread (or multiple) that gets memory accepted proactively. It will minimize time the system experience latency spikes on memory allocation while keeping low boot time.
This approach cannot function on its own. It is an extension of #2:
background memory acceptance requires functional scheduler, but the page allocator may need to tap into unaccepted memory before that.The downside of the approach is that these threads also steal CPU cycles and memory bandwidth from the user's workload and may hurt user experience.
Implement #2 for now. It is a reasonable default. Some workloads may want to use #1 or #3 and they can be implemented later based on user's demands.
Support of unaccepted memory requires a few changes in core-mm code:
- memblock has to accept memory on allocation;
- page allocator has to accept memory on the first allocation of the
page;Memblock change is trivial.
The page allocator is modified to accept pages on the first allocation. The new page type (encoded in the
_mapcount) --PageUnaccepted()-- is used to indicate that the page requires acceptance.Architecture has to provide two helpers if it wants to support unaccepted memory:
Quoted 8 lines
-
accept_memory()makes a range of physical addresses accepted.-
range_contains_unaccepted_memory()checks anything within the range of physical addresses requires acceptance.Signed-off-by Kirill A. Shutemov <[email protected]>Reviewed-by David Hildenbrand <[email protected]>Quoted 10 lines
Hmm I realize it's not ideal to raise this at v7, and maybe it was discussed before, but it's really not great how this affects the core page allocator paths. Wouldn't it be possible to only release pages to page allocator when accepted, and otherwise use some new per-zone variables together with the bitmap to track how much exactly is where to accept? Then it could be hooked in
get_page_from_freelist()similarly toCONFIG_DEFERRED_STRUCT_PAGE_INIT- if we failzone_watermark_fast()and there are unaccepted pages in the zone, accept them and continue. With a static key to flip in case we eventually accept everything. Because this is really similar scenario to the deferred init and that one was solved in a way that adds minimal overhead.Quoted 3 lines
I kind of like just having the memory stats being correct (e.g., free memory) and acceptance being an internal detail to be triggered when allocating pages -- just like the
arch_alloc_page()callback.Hm, good point about the stats. Could be tweaked perhaps so it appears correct on the outside, but might be tricky.
Quoted 3 lines
I'm sure we could optimize for the !unaccepted memory via static keys also in this version with some checks at the right places if we find this to hurt performance?
It would be great if we would at least somehow hit the necessary code only when dealing with a >=pageblock size block. The bitmap approach and accepting everything smaller uprofront actually seems rather compatible. Yet in the current patch we e.g. check PageUnaccepted(buddy) on every buddy size while merging.
Quoted 6 lines
A list that sits besides the existing
free_area, contains only >=pageblock order sizes of unaccepted pages (no migratetype distinguished) and we tap into it approximately before__rmqueue_fallback()? There would be some trickery around releasing zone-lock for doingaccept_memory(), but should be manageable.
Just curious, do we have a microbenchmark that is able to reveal the impact of such code changes before we start worrying?
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 8 lines · David Hildenbrand
Quoted 6 lines
A list that sits besides the existing
free_area, contains only >=pageblock order sizes of unaccepted pages (no migratetype distinguished) and we tap into it approximately before__rmqueue_fallback()? There would be some trickery around releasing zone-lock for doingaccept_memory(), but should be manageable.Just curious, do we have a microbenchmark that is able to reveal the impact of such code changes before we start worrying?
Nope. I went looking to see if I could find any impact. I think Kirill did too. Too bad that effort didn't make it into the changelog yet.
The merging check at least is just checking a field in a cache-hot 'struct page'. The common case is probably three instructions:
load to a register
check the bit
jump if not set
It adds a wee bit of icache pressure, but it's also the kind of thing that should be a piece of cake for the branch predictors.
That dynamic check could easily be wrapped by a static branch. But, that first requires more code to go dig in the nooks and crannies of the page allocator to make sure *ALL* pages are accepted.
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
In reply to Vlastimil BabkaQuoted 8 lines · Vlastimil Babka
Quoted 3 lines
I'm sure we could optimize for the !unaccepted memory via static keys also in this version with some checks at the right places if we find this to hurt performance?
It would be great if we would at least somehow hit the necessary code only when dealing with a >=pageblock size block. The bitmap approach and accepting everything smaller uprofront actually seems rather compatible. Yet in the current patch we e.g. check PageUnaccepted(buddy) on every buddy size while merging.
Needing to check PageUnaccepted() during the merge is fallout from moving the acceptance to post_alloc_hook(). I _think_ an earlier version of this did page acceptance under the zone lock, closer to where the page comes off the 2M/4M lists.
But, page acceptance is horribly slow, so I asked Kirill to move it out from under the zone lock. Doing it in post_alloc_hook() (after the zone lock is dropped) makes a lot of sense since we do zeroing in there and zeroing is also nice and slow.
But, post_alloc_hook() is long after the 2M page has been split and that means that we have to deal with potentially unaccepted pages during merges.
I think there are three basic options:
1. This patch: Do acceptance after the zone lock is dropped and deal with mixed-acceptance merges
2. Do acceptance under the zone lock as pages come off the 2M/4M lists, but before the page is split.
3. Pull the page off the 2M/4M lists, drop the zone lock, accept it, then put it back.
I'm not sure any of those other options are better.
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 23 lines · Dave Hansen
Quoted 8 lines · Vlastimil Babka
Quoted 3 lines
I'm sure we could optimize for the !unaccepted memory via static keys also in this version with some checks at the right places if we find this to hurt performance?
It would be great if we would at least somehow hit the necessary code only when dealing with a >=pageblock size block. The bitmap approach and accepting everything smaller uprofront actually seems rather compatible. Yet in the current patch we e.g. check PageUnaccepted(buddy) on every buddy size while merging.
Needing to check
PageUnaccepted()during the merge is fallout from moving the acceptance topost_alloc_hook(). I _think_ an earlier version of this did page acceptance under the zone lock, closer to where the page comes off the 2M/4M lists.But, page acceptance is horribly slow, so I asked Kirill to move it out from under the zone lock. Doing it in
post_alloc_hook()(after the zone lock is dropped) makes a lot of sense since we do zeroing in there and zeroing is also nice and slow.But,
post_alloc_hook()is long after the 2M page has been split and that means that we have to deal with potentially unaccepted pages during merges.
Quoted 6 lines
I think there are three basic options:
1. This patch: Do acceptance after the zone lock is dropped and deal with mixed-acceptance merges
2. Do acceptance under the zone lock as pages come off the 2M/4M lists, but before the page is split.
Rather not, as acceptance can be slow and we shouldn't hog the zone lock while doing it.
Quoted 2 lines
3. Pull the page off the 2M/4M lists, drop the zone lock, accept it, then put it back.
Worth trying, IMHO. Perhaps easier to manage if the lists are distinct from the normal ones, as I suggested.
Quoted 1 line
I'm not sure any of those other options are better.
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 4 lines · Vlastimil Babka
Quoted 2 lines
3. Pull the page off the 2M/4M lists, drop the zone lock, accept it, then put it back.
Worth trying, IMHO. Perhaps easier to manage if the lists are distinct from the normal ones, as I suggested.
I was playing with another series recently where I did this, momentarily taking pages off some of the high-order lists and dropping the zone lock.
Kirill, if you go looking at this, just make sure that you don't let this happen to too much memory at once. You might end up yanking memory out of the allocator that's not reflected in NR_FREE_PAGES.
You might, for instance want to make sure that only a small number of threads can have pulled memory off the free lists at once. Something *logically* like this:
// Limit to two threads at once:atomic_t nr_accepting_threads = ATOMIC_INIT(2);
page = del_page_from_free_list();
if (!PageAccepted(page)) {
if (atomic_dec_and_test(&nr_accepting_threads)) {
// already at the thread limit
add_page_from_free_list(page, ...);
spin_unlock_irq(&zone->lock);
// wait for a slot...
spin_lock_irq(&zone->lock);
goto retry;
} else {
spin_unlock_irq(&zone->lock);
accept_page(page);
spin_lock_irq(&zone->lock);
add_page_from_free_list(page, ...);
// do merging if it was a 2M page
}
}It's a little nasty because the whole thing is not a sleepable context.
I also know that the merging code needs some refactoring if you want to
do merging with 2M pages here. It might all get easier if you move all the page allocator stuff to only work at the 4M granularity.
In any case, I'm not trying to dissuade anyone from listening to the other reviewer feedback. Just trying to save you a few cycles on a similar problem I was looking at recently.
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
In reply to Vlastimil BabkaQuoted 67 lines · Vlastimil Babka
Quoted 57 lines · Kirill A. Shutemov
UEFI Specification version 2.9 introduces the concept of memory acceptance. Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, require memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific to the Virtual Machine platform.
There are several ways kernel can deal with unaccepted memory:
1. Accept all the memory during the boot. It is easy to implement and it doesn't have runtime cost once the system is booted. The downside is very long boot time.
Accept can be parallelized to multiple CPUs to keep it manageable (i.e. via
DEFERRED_STRUCT_PAGE_INIT), but it tends to saturate memory bandwidth and does not scale beyond the point.2. Accept a block of memory on the first use. It requires more infrastructure and changes in page allocator to make it work, but it provides good boot time.
On-demand memory accept means latency spikes every time kernel steps onto a new memory block. The spikes will go away once workload data set size gets stabilized or all memory gets accepted.
3. Accept all memory in background. Introduce a thread (or multiple) that gets memory accepted proactively. It will minimize time the system experience latency spikes on memory allocation while keeping low boot time.
This approach cannot function on its own. It is an extension of #2:
background memory acceptance requires functional scheduler, but the page allocator may need to tap into unaccepted memory before that.The downside of the approach is that these threads also steal CPU cycles and memory bandwidth from the user's workload and may hurt user experience.
Implement #2 for now. It is a reasonable default. Some workloads may want to use #1 or #3 and they can be implemented later based on user's demands.
Support of unaccepted memory requires a few changes in core-mm code:
- memblock has to accept memory on allocation;
- page allocator has to accept memory on the first allocation of the
page;Memblock change is trivial.
The page allocator is modified to accept pages on the first allocation. The new page type (encoded in the
_mapcount) --PageUnaccepted()-- is used to indicate that the page requires acceptance.Architecture has to provide two helpers if it wants to support unaccepted memory:
Quoted 8 lines
-
accept_memory()makes a range of physical addresses accepted.-
range_contains_unaccepted_memory()checks anything within the range of physical addresses requires acceptance.Signed-off-by Kirill A. Shutemov <[email protected]>Reviewed-by David Hildenbrand <[email protected]>
Quoted 11 lines
Hmm I realize it's not ideal to raise this at v7, and maybe it was discussed before, but it's really not great how this affects the core page allocator paths. Wouldn't it be possible to only release pages to page allocator when accepted, and otherwise use some new per-zone variables together with the bitmap to track how much exactly is where to accept? Then it could be hooked in
get_page_from_freelist()similarly toCONFIG_DEFERRED_STRUCT_PAGE_INIT- if we failzone_watermark_fast()and there are unaccepted pages in the zone, accept them and continue. With a static key to flip in case we eventually accept everything. Because this is really similar scenario to the deferred init and that one was solved in a way that adds minimal overhead.
I think it might be more straight-forward to always accept pages in the size of the pageblock. Smaller ranges should matter because they have been accepted in deferred_free_range. In expand, if PageUnaccepted is set on a pageblock-sized page, take it off the list, drop the zone->lock leaving IRQs disabled, accept the memory and reacquire the lock to split the page into the required order.
IRQs being left disabled is unfortunate but even if the acceptance is slow, it's presumably not so slow to cause major problems. This would would reduce and probably eliminate the need to do the assert check in accept_page. It might also simplify __free_one_page if it's known that a pageblock range of pages are either all accepted or unaccepted.
Lastly, the default behaviour should probably be "accept all memory at boot" and use Kconfig to allow acceptable be deferred or overridden by command line. There are at least two reasons for this. Even though this is a virtual machine, there still may be latency sensitive applications running early in boot using pinned vcpu->pcpu and no memory overcommit. The unpredictable performance of the application early in boot may be unacceptable and unavoidable. It might take a long time but it could eventually generate bug reports about "unpredictable performance early in boot" that will be hard to track down unless accept_memory is observed using perf at the right time. Even when that does happen, there will need to be an option to turn it off if the unpredictable performance cannot be tolerated. Second, any benchmarking done early in boot is likely to be disrupted making the series a potential bisection magnet that masks a performance bug elsewhere in the merge window.
Signature · 2 lines
Mel Gorman SUSE Labs
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 11 lines
The unpredictable performance of the application early in boot may be unacceptable and unavoidable. It might take a long time but it could eventually generate bug reports about "unpredictable performance early in boot" that will be hard to track down unless
accept_memoryis observed using perf at the right time. Even when that does happen, there will need to be an option to turn it off if the unpredictable performance cannot be tolerated. Second, any benchmarking done early in boot is likely to be disrupted making the series a potential bisection magnet that masks a performance bug elsewhere in the merge window.
I'm doing some boot performance tests now before I run some workload memory acceptance latency tests.
Note that this testing is on AMD SEV-SNP, so this patch series on top of the AMD guest patches v12, plus a
patch Brijesh Singh wrote to define __accept_memory for SEV-SNP
https://github.com/AMDESE/linux/commit/ecae2582666d50ce1e633975d703d2f904183ece
I was getting pretty consistent boot times, only going up slightly as the memory size increased, but at 256GB, the VM crashes because it touches some unaccepted memory without first accepting it. 255GB boots fine.
The stack track is in mm/page_alloc.c. I've done a little investigation, but I can't account for why there's a hard cutoff of correctness at 256GB
[ 0.065563] RIP: 0010:memmap_init_range+0x108/0x173 [ 0.066309] Code: 77 16 f6 42 10 02 74 10 48 03 42 08 48 c1 e8 0c 48 89 c3 e9 3a ff ff ff 48 89 df 48 c1 e7 06 48 03 3d d9 a2 66 ff 48 8d 47 08 <c7> 47 34 01 00 00 00 48 c7 47 38 00 00 00 00 c7 47 30 ff ff ff ff [ 0.069108] RSP: 0000:ffffffffad603dc8 EFLAGS: 00010082 ORIG_RAX: 0000000000000404 [ 0.070193] RAX: ffffdba740000048 RBX: 0000000000000001 RCX: 0000000000000000 [ 0.071170] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffdba740000040 [ 0.072224] RBP: 0000000000000000 R08: 0000000000001000 R09: 0000000000000000 [ 0.073283] R10: 0000000000000001 R11: ffffffffad645c60 R12: 0000000000000000 [ 0.074304] R13: 00000000000000a0 R14: 0000000000000000 R15: 0000000000000000 [ 0.075285] FS: 0000000000000000(0000) GS:ffffffffadd6c000(0000) knlGS:0000000000000000 [ 0.076365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.077194] CR2: ffffdba740000074 CR3: 0008001ee3a0c000 CR4: 00000000000606b0 [ 0.078209] Call Trace: [ 0.078524] <TASK> [ 0.078887] ? free_area_init+0x5c1/0x66c [ 0.079417] ? zone_sizes_init+0x52/0x6c [ 0.079934] ? setup_arch+0xa55/0xb6d [ 0.080417] ? start_kernel+0x64/0x65a [ 0.080897] ? secondary_startup_64_no_verify+0xd6/0xdb [ 0.081620] </TASK>
Quoted 4 lines
--
Mel Gorman
SUSE Labs
Signature · 1 line
-Dionna Glaze, PhD (she/her)
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 12 lines · Dionna Amalie Glaze
Quoted 11 lines
The unpredictable performance of the application early in boot may be unacceptable and unavoidable. It might take a long time but it could eventually generate bug reports about "unpredictable performance early in boot" that will be hard to track down unless
accept_memoryis observed using perf at the right time. Even when that does happen, there will need to be an option to turn it off if the unpredictable performance cannot be tolerated. Second, any benchmarking done early in boot is likely to be disrupted making the series a potential bisection magnet that masks a performance bug elsewhere in the merge window.
Quoted 6 lines
I'm doing some boot performance tests now before I run some workload memory acceptance latency tests.
Note that this testing is on AMD SEV-SNP, so this patch series on top of the AMD guest patches v12, plus a
patch Brijesh Singh wrote to define__accept_memoryfor SEV-SNP
https://github.com/AMDESE/linux/commit/ecae2582666d50ce1e633975d703d2f904183ece
Note that there is a bug in Brijesh's version of the patch and it will almost exclusively use the MSR protocol. Please try the version of the patch that I recently sent up based on the current unaccepted memory tree from Kirill.
https://lore.kernel.org/lkml/[email protected]/
Thanks,
Tom
Quoted 42 lines
I was getting pretty consistent boot times, only going up slightly as the memory size increased, but at 256GB, the VM crashes because it touches some unaccepted memory without first accepting it. 255GB boots fine.
The stack track is in
mm/page_alloc.c. I've done a little investigation, but I can't account for why there's a hard cutoff of correctness at 256GB[ 0.065563] RIP: 0010:memmap_init_range+0x108/0x173 [ 0.066309] Code: 77 16 f6 42 10 02 74 10 48 03 42 08 48 c1 e8 0c 48 89 c3 e9 3a ff ff ff 48 89 df 48 c1 e7 06 48 03 3d d9 a2 66 ff 48 8d 47 08 <c7> 47 34 01 00 00 00 48 c7 47 38 00 00 00 00 c7 47 30 ff ff ff ff [ 0.069108] RSP: 0000:ffffffffad603dc8 EFLAGS: 00010082 ORIG_RAX: 0000000000000404 [ 0.070193] RAX: ffffdba740000048 RBX: 0000000000000001 RCX: 0000000000000000 [ 0.071170] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffdba740000040 [ 0.072224] RBP: 0000000000000000 R08: 0000000000001000 R09: 0000000000000000 [ 0.073283] R10: 0000000000000001 R11: ffffffffad645c60 R12: 0000000000000000 [ 0.074304] R13: 00000000000000a0 R14: 0000000000000000 R15: 0000000000000000 [ 0.075285] FS: 0000000000000000(0000) GS:ffffffffadd6c000(0000) knlGS:0000000000000000 [ 0.076365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.077194] CR2: ffffdba740000074 CR3: 0008001ee3a0c000 CR4: 00000000000606b0 [ 0.078209] Call Trace: [ 0.078524] <TASK> [ 0.078887] ? free_area_init+0x5c1/0x66c [ 0.079417] ? zone_sizes_init+0x52/0x6c [ 0.079934] ? setup_arch+0xa55/0xb6d [ 0.080417] ? start_kernel+0x64/0x65a [ 0.080897] ? secondary_startup_64_no_verify+0xd6/0xdb [ 0.081620] </TASK>Quoted 4 lines
--
Mel Gorman
SUSE Labs
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 29 lines
Quoted 20 lines
The stack track is in
mm/page_alloc.c. I've done a little investigation, but I can't account for why there's a hard cutoff of correctness at 256GB[ 0.065563] RIP: 0010:memmap_init_range+0x108/0x173 [ 0.066309] Code: 77 16 f6 42 10 02 74 10 48 03 42 08 48 c1 e8 0c 48 89 c3 e9 3a ff ff ff 48 89 df 48 c1 e7 06 48 03 3d d9 a2 66 ff 48 8d 47 08 <c7> 47 34 01 00 00 00 48 c7 47 38 00 00 00 00 c7 47 30 ff ff ff ff [ 0.069108] RSP: 0000:ffffffffad603dc8 EFLAGS: 00010082 ORIG_RAX: 0000000000000404 [ 0.070193] RAX: ffffdba740000048 RBX: 0000000000000001 RCX: 0000000000000000 [ 0.071170] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffdba740000040 [ 0.072224] RBP: 0000000000000000 R08: 0000000000001000 R09: 0000000000000000 [ 0.073283] R10: 0000000000000001 R11: ffffffffad645c60 R12: 0000000000000000 [ 0.074304] R13: 00000000000000a0 R14: 0000000000000000 R15: 0000000000000000 [ 0.075285] FS: 0000000000000000(0000) GS:ffffffffadd6c000(0000) knlGS:0000000000000000 [ 0.076365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.077194] CR2: ffffdba740000074 CR3: 0008001ee3a0c000 CR4: 00000000000606b0Quoted 8 lines
[ 0.078209] Call Trace: [ 0.078524] <TASK> [ 0.078887] ? free_area_init+0x5c1/0x66c [ 0.079417] ? zone_sizes_init+0x52/0x6c [ 0.079934] ? setup_arch+0xa55/0xb6d [ 0.080417] ? start_kernel+0x64/0x65a [ 0.080897] ? secondary_startup_64_no_verify+0xd6/0xdb [ 0.081620] </TASK>
Quoted 5 lines
Note that there is a bug in Brijesh's version of the patch and it will almost exclusively use the MSR protocol. Please try the version of the patch that I recently sent up based on the current unaccepted memory tree from Kirill.
I've now tested this patch set with Tom's new patch set, and it appears to be that the problem with 256GB is more likely to be due to this unaccepted memory patch set rather than something AMD-specific.
Kirill, do you see any problems with 256GB on TDX? It seems there is some unaccepted memory getting touched in memmap_init_range when the VM's memory size is at least 256GB
Signature · 1 line
-Dionna Glaze, PhD (she/her)
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 33 lines · Dionna Amalie Glaze
Quoted 28 lines
Quoted 20 lines
The stack track is in
mm/page_alloc.c. I've done a little investigation, but I can't account for why there's a hard cutoff of correctness at 256GB[ 0.065563] RIP: 0010:memmap_init_range+0x108/0x173 [ 0.066309] Code: 77 16 f6 42 10 02 74 10 48 03 42 08 48 c1 e8 0c 48 89 c3 e9 3a ff ff ff 48 89 df 48 c1 e7 06 48 03 3d d9 a2 66 ff 48 8d 47 08 <c7> 47 34 01 00 00 00 48 c7 47 38 00 00 00 00 c7 47 30 ff ff ff ff [ 0.069108] RSP: 0000:ffffffffad603dc8 EFLAGS: 00010082 ORIG_RAX: 0000000000000404 [ 0.070193] RAX: ffffdba740000048 RBX: 0000000000000001 RCX: 0000000000000000 [ 0.071170] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffdba740000040 [ 0.072224] RBP: 0000000000000000 R08: 0000000000001000 R09: 0000000000000000 [ 0.073283] R10: 0000000000000001 R11: ffffffffad645c60 R12: 0000000000000000 [ 0.074304] R13: 00000000000000a0 R14: 0000000000000000 R15: 0000000000000000 [ 0.075285] FS: 0000000000000000(0000) GS:ffffffffadd6c000(0000) knlGS:0000000000000000 [ 0.076365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.077194] CR2: ffffdba740000074 CR3: 0008001ee3a0c000 CR4: 00000000000606b0Quoted 8 lines
[ 0.078209] Call Trace: [ 0.078524] <TASK> [ 0.078887] ? free_area_init+0x5c1/0x66c [ 0.079417] ? zone_sizes_init+0x52/0x6c [ 0.079934] ? setup_arch+0xa55/0xb6d [ 0.080417] ? start_kernel+0x64/0x65a [ 0.080897] ? secondary_startup_64_no_verify+0xd6/0xdb [ 0.081620] </TASK>Quoted 5 lines
Note that there is a bug in Brijesh's version of the patch and it will almost exclusively use the MSR protocol. Please try the version of the patch that I recently sent up based on the current unaccepted memory tree from Kirill.
Quoted 7 lines
I've now tested this patch set with Tom's new patch set, and it appears to be that the problem with 256GB is more likely to be due to this unaccepted memory patch set rather than something AMD-specific.
Kirill, do you see any problems with 256GB on TDX? It seems there is some unaccepted memory getting touched in
memmap_init_rangewhen the VM's memory size is at least 256GB
It really helps this kind of stuff if you can post the *actual* error. I assume this was a page fault, so there should have been some other stuff before the RIP:...
Another thing that's really nice is to do the disassembly of the "Code:" or share disassembly of memmap_init_range. Even nicer would be to give an faddr2line of the RIP value and track down which C code was actually at fault.
It's *possible* to look into these things from what you posted, but it's just slow. I'm sure Kirill will appreciate the help.
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 5 lines
It really helps this kind of stuff if you can post the *actual* error. I assume this was a page fault, so there should have been some other stuff before the RIP:...
I posted the error on August 15th. I was bumping in my last post since I confirmed with Tom Lendacky that it wasn't AMD's patches at fault.
Here's a new dump below that matches the disassembly:
[ 0.043137] Faking a node at [mem 0x0000000000000000-0x000000403fffffff] [ 0.044018] NODE_DATA(0) allocated [mem 0x403fffc000-0x403fffffff] [ 0.044922] Zone ranges: [ 0.045250] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.046039] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.046828] Normal [mem 0x0000000100000000-0x000000403fffffff] [ 0.047657] Movable zone start for each node [ 0.048201] Early memory node ranges [ 0.048674] node 0: [mem 0x0000000000001000-0x000000000009ffff] [ 0.049474] node 0: [mem 0x0000000000100000-0x000000000080cfff [ 0.050274] node 0: [mem 0x000000000080f000-0x00000000beceefff] [ 0.051074] node 0: [mem 0x00000000befff000-0x00000000bfbb0fff] [ 0.051874] node 0: [mem 0x00000000bfbb2000-0x00000000bffdbfff] [ 0.052674] node 0: [mem 0x0000000100000000-0x000000403fffffff] [ 0.053530] Initmem setup node 0 [mem 0x0000000000001000-0x000000403fffffff] PANIC: Unsupported exit-code 0x404 in early #VC exception (IP: 0xfffffffface0cdd0) [ 0.056667] CPU: 0 PID: 0 Comm: swapper Not tainted 5.17.0-rc6-173762-gffb12b02c6d7-dirty #1 [ 0.057744] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 [ 0.058920] RIP: 0010:memmap_init_range+0x11d/0x188 [ 0.059686] Code: 77 16 f6 42 10 02 74 10 48 03 42 08 48 c1 e8 0c 48 89 c3 e9 3a ff ff ff 48 89 df 48 c1 e7 06 48 03 3d a4 1e 65 ff 48 8d 47 08 <c7> 47 34 01 00 00 00 48 c7 47 38 00 00 00 00 c7 47 30 ff ff ff ff [ 0.062121] RSP: 0000:ffffffffac603dc0 EFLAGS: 00010082 ORIG_RAX: 0000000000000404 [ 0.063087] RAX: ffffda1ac0000048 RBX: 0000000000000001 RCX: 0000000000000000 [ 0.063998] RDX: 0300000000000000 RSI: 0000000000000000 RDI: ffffda1ac000004 [ 0.064944] RBP: 0000000000000000 R08: 0000000000001000 R09: 0000000000000000 [ 0.065873] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000 [ 0.066782] R13: 00000000000000a0 R14: 0000000000000000 R15: 0000000000000000 [ 0.067695] FS: 0000000000000000(0000) GS:ffffffffacd88000(0000) knlGS:0000000000000000 [ 0.068727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.069488] CR2: ffffda1ac0000074 CR3: 00080020b680c000 CR4: 00000000000606b0 [ 0.070397] Call Trace: [ 0.070710] <TASK> [ 0.070976] ? free_area_init+0x724/0x7d4 [ 0.071486] ? zone_sizes_init+0x52/0x6c [ 0.071986] ? setup_arch+0xa55/0xb77 [ 0.072453] ? start_kernel+0x64/0x65f [ 0.072931] ? secondary_startup_64_no_verify+0xd6/0xdb [ 0.073598] </TASK>
Note this is a crash in SEV-SNP, but I assume we'd get the same #VE in TDX.
Quoted 2 lines
Another thing that's really nice is to do the disassembly of the "Code:" or share disassembly of
memmap_init_range.
0000000000000172 <memmap_init_range>:
172: 41 56 push %r14 174: 89 f0 mov %esi,%eax 176: 45 89 ce mov %r9d,%r14d 179: 41 55 push %r13 17b: 4c 8d 2c 39 lea (%rcx,%rdi,1),%r13 17f: 41 54 push %r12 181: 49 89 d4 mov %rdx,%r12 184: 49 8d 55 ff lea -0x1(%r13),%rdx 188: 48 3b 15 00 00 00 00 cmp 0x0(%rip),%rdx # 18f <memmap_init_range+0x1d> 18f: 55 push %rbp 190: 53 push %rbx 191: 48 89 cb mov %rcx,%rbx 194: 76 07 jbe 19d <memmap_init_range+0x2b> 196: 48 89 15 00 00 00 00 mov %rdx,0x0(%rip) # 19d <memmap_init_range+0x2b> 19d: 4c 89 e5 mov %r12,%rbp 1a0: ba 03 00 00 00 mov $0x3,%edx 1a5: 48 c1 e0 3a shl $0x3a,%rax 1a9: 48 c1 e5 38 shl $0x38,%rbp 1ad: 48 c1 e2 38 shl $0x38,%rdx 1b1: 48 21 d5 and %rdx,%rbp 1b4: 48 09 c5 or %rax,%rbp 1b7: 49 39 dd cmp %rbx,%r13 1ba: 0f 86 31 01 00 00 jbe 2f1 <memmap_init_range+0x17f> 1c0: 45 85 f6 test %r14d,%r14d 1c3: 0f 85 b4 00 00 00 jne 27d <memmap_init_range+0x10b> 1c9: 49 83 fc 03 cmp $0x3,%r12 1cd: 0f 94 c1 sete %cl 1d0: 22 0d 00 00 00 00 and 0x0(%rip),%cl # 1d6 <memmap_init_range+0x64> 1d6: 0f 84 a1 00 00 00 je 27d <memmap_init_range+0x10b> 1dc: 48 8b 15 00 00 00 00 mov 0x0(%rip),%rdx # 1e3 <memmap_init_range+0x71> 1e3: 48 85 d2 test %rdx,%rdx 1e6: 74 10 je 1f8 <memmap_init_range+0x86> 1e8: 48 8b 42 08 mov 0x8(%rdx),%rax 1ec: 48 03 02 add (%rdx),%rax 1ef: 48 c1 e8 0c shr $0xc,%rax 1f3: 48 39 d8 cmp %rbx,%rax 1f6: 77 55 ja 24d <memmap_init_range+0xdb> 1f8: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # 1ff <memmap_init_range+0x8d> 1ff: 4c 6b 05 00 00 00 00 imul $0x18,0x0(%rip),%r8 # 207 <memmap_init_range+0x95> 206: 18 207: 31 f6 xor %esi,%esi 209: 48 89 05 00 00 00 00 mov %rax,0x0(%rip) # 210 <memmap_init_range+0x9e> 210: 49 01 c0 add %rax,%r8 213: 48 89 c7 mov %rax,%rdi 216: 4c 39 c0 cmp %r8,%rax 219: 73 26 jae 241 <memmap_init_range+0xcf> 21b: 48 8b 57 08 mov 0x8(%rdi),%rdx 21f: 48 03 17 add (%rdi),%rdx 222: 48 83 c0 18 add $0x18,%rax 226: 48 c1 ea 0c shr $0xc,%rdx 22a: 48 39 da cmp %rbx,%rdx 22d: 76 0e jbe 23d <memmap_init_range+0xcb> 22f: 40 84 f6 test %sil,%sil 232: 74 19 je 24d <memmap_init_range+0xdb> 234: 48 89 3d 00 00 00 00 mov %rdi,0x0(%rip) # 23b <memmap_init_range+0xc9> 23b: eb 10 jmp 24d <memmap_init_range+0xdb> 23d: 89 ce mov %ecx,%esi 23f: eb d2 jmp 213 <memmap_init_range+0xa1> 241: 40 84 f6 test %sil,%sil 244: 74 07 je 24d <memmap_init_range+0xdb> 246: 48 89 05 00 00 00 00 mov %rax,0x0(%rip) # 24d <memmap_init_range+0xdb> 24d: 48 8b 15 00 00 00 00 mov 0x0(%rip),%rdx # 254 <memmap_init_range+0xe2> 254: 48 8b 02 mov (%rdx),%rax 257: 48 8d 88 ff 0f 00 00 lea 0xfff(%rax),%rcx 25e: 48 c1 e9 0c shr $0xc,%rcx 262: 48 39 d9 cmp %rbx,%rcx 265: 77 16 ja 27d <memmap_init_range+0x10b> 267: f6 42 10 02 testb $0x2,0x10(%rdx) 26b: 74 10 je 27d <memmap_init_range+0x10b> 26d: 48 03 42 08 add 0x8(%rdx),%rax 271: 48 c1 e8 0c shr $0xc,%rax 275: 48 89 c3 mov %rax,%rbx 278: e9 3a ff ff ff jmp 1b7 <memmap_init_range+0x45> 27d: 48 89 df mov %rbx,%rdi 280: 48 c1 e7 06 shl $0x6,%rdi 284: 48 03 3d 00 00 00 00 add 0x0(%rip),%rdi # 28b <memmap_init_range+0x119> 28b: 48 8d 47 08 lea 0x8(%rdi),%rax 28f: c7 47 34 01 00 00 00 movl $0x1,0x34(%rdi) # Here's where the crash RIP is. 296: 48 c7 47 38 00 00 00 movq $0x0,0x38(%rdi) 29d: 00 29e: c7 47 30 ff ff ff ff movl $0xffffffff,0x30(%rdi) 2a5: 48 c7 47 28 00 00 00 movq $0x0,0x28(%rdi) 2ac: 00 2ad: 48 c7 47 20 00 00 00 movq $0x0,0x20(%rdi) 2b4: 00 2b5: 48 c7 47 18 00 00 00 movq $0x0,0x18(%rdi) 2bc: 00 2bd: 48 89 2f mov %rbp,(%rdi) 2c0: 48 89 47 08 mov %rax,0x8(%rdi) 2c4: 48 89 47 10 mov %rax,0x10(%rdi) 2c8: 41 83 fe 01 cmp $0x1,%r14d 2cc: 75 05 jne 2d3 <memmap_init_range+0x161> 2ce: 48 0f ba 2f 0c btsq $0xc,(%rdi) 2d3: f7 c3 ff 01 00 00 test $0x1ff,%ebx 2d9: 75 0e jne 2e9 <memmap_init_range+0x177> 2db: 8b 74 24 38 mov 0x38(%rsp),%esi 2df: e8 00 00 00 00 call 2e4 <memmap_init_range+0x172> 2e4: e8 00 00 00 00 call 2e9 <memmap_init_range+0x177> 2e9: 48 ff c3 inc %rbx 2ec: e9 c6 fe ff ff jmp 1b7 <memmap_init_range+0x45> 2f1: 5b pop %rbx 2f2: 5d pop %rbp 2f3: 41 5c pop %r12 2f5: 41 5d pop %r13 2f7: 41 5e pop %r14 2f9: c3 ret
Quoted 3 lines
Even nicer would be to give
an faddr2line of the RIP value and track down which C code was actually at fault.
arch_atomic_setarch/x86/include/asm/atomic.h:41
of INIT_LIST_HEAD in __init_single_page, called from memmap_init_range. --
-Dionna Glaze, PhD (she/her)
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 182 lines · Dionna Amalie Glaze
Quoted 5 lines
It really helps this kind of stuff if you can post the *actual* error. I assume this was a page fault, so there should have been some other stuff before the RIP:...
I posted the error on August 15th. I was bumping in my last post since I confirmed with Tom Lendacky that it wasn't AMD's patches at fault.
Here's a new dump below that matches the disassembly:[ 0.043137] Faking a node at [mem 0x0000000000000000-0x000000403fffffff] [ 0.044018] NODE_DATA(0) allocated [mem 0x403fffc000-0x403fffffff] [ 0.044922] Zone ranges: [ 0.045250] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.046039] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.046828] Normal [mem 0x0000000100000000-0x000000403fffffff] [ 0.047657] Movable zone start for each node [ 0.048201] Early memory node ranges [ 0.048674] node 0: [mem 0x0000000000001000-0x000000000009ffff] [ 0.049474] node 0: [mem 0x0000000000100000-0x000000000080cfff [ 0.050274] node 0: [mem 0x000000000080f000-0x00000000beceefff] [ 0.051074] node 0: [mem 0x00000000befff000-0x00000000bfbb0fff] [ 0.051874] node 0: [mem 0x00000000bfbb2000-0x00000000bffdbfff] [ 0.052674] node 0: [mem 0x0000000100000000-0x000000403fffffff] [ 0.053530] Initmem setup node 0 [mem 0x0000000000001000-0x000000403fffffff] PANIC: Unsupported exit-code 0x404 in early #VC exception (IP: 0xfffffffface0cdd0) [ 0.056667] CPU: 0 PID: 0 Comm: swapper Not tainted 5.17.0-rc6-173762-gffb12b02c6d7-dirty #1 [ 0.057744] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 [ 0.058920] RIP: 0010:memmap_init_range+0x11d/0x188 [ 0.059686] Code: 77 16 f6 42 10 02 74 10 48 03 42 08 48 c1 e8 0c 48 89 c3 e9 3a ff ff ff 48 89 df 48 c1 e7 06 48 03 3d a4 1e 65 ff 48 8d 47 08 <c7> 47 34 01 00 00 00 48 c7 47 38 00 00 00 00 c7 47 30 ff ff ff ff [ 0.062121] RSP: 0000:ffffffffac603dc0 EFLAGS: 00010082 ORIG_RAX: 0000000000000404 [ 0.063087] RAX: ffffda1ac0000048 RBX: 0000000000000001 RCX: 0000000000000000 [ 0.063998] RDX: 0300000000000000 RSI: 0000000000000000 RDI: ffffda1ac000004 [ 0.064944] RBP: 0000000000000000 R08: 0000000000001000 R09: 0000000000000000 [ 0.065873] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000 [ 0.066782] R13: 00000000000000a0 R14: 0000000000000000 R15: 0000000000000000 [ 0.067695] FS: 0000000000000000(0000) GS:ffffffffacd88000(0000) knlGS:0000000000000000 [ 0.068727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.069488] CR2: ffffda1ac0000074 CR3: 00080020b680c000 CR4: 00000000000606b0 [ 0.070397] Call Trace: [ 0.070710] <TASK> [ 0.070976] ? free_area_init+0x724/0x7d4 [ 0.071486] ? zone_sizes_init+0x52/0x6c [ 0.071986] ? setup_arch+0xa55/0xb77 [ 0.072453] ? start_kernel+0x64/0x65f [ 0.072931] ? secondary_startup_64_no_verify+0xd6/0xdb [ 0.073598] </TASK>Note this is a crash in SEV-SNP, but I assume we'd get the same #VE in TDX.
Quoted 2 lines
Another thing that's really nice is to do the disassembly of the "Code:" or share disassembly of
memmap_init_range.0000000000000172 <
memmap_init_range>:172: 41 56 push %r14 174: 89 f0 mov %esi,%eax 176: 45 89 ce mov %r9d,%r14d 179: 41 55 push %r13 17b: 4c 8d 2c 39 lea (%rcx,%rdi,1),%r13 17f: 41 54 push %r12 181: 49 89 d4 mov %rdx,%r12 184: 49 8d 55 ff lea -0x1(%r13),%rdx 188: 48 3b 15 00 00 00 00 cmp 0x0(%rip),%rdx # 18f <memmap_init_range+0x1d> 18f: 55 push %rbp 190: 53 push %rbx 191: 48 89 cb mov %rcx,%rbx 194: 76 07 jbe 19d <memmap_init_range+0x2b> 196: 48 89 15 00 00 00 00 mov %rdx,0x0(%rip) # 19d <memmap_init_range+0x2b> 19d: 4c 89 e5 mov %r12,%rbp 1a0: ba 03 00 00 00 mov $0x3,%edx 1a5: 48 c1 e0 3a shl $0x3a,%rax 1a9: 48 c1 e5 38 shl $0x38,%rbp 1ad: 48 c1 e2 38 shl $0x38,%rdx 1b1: 48 21 d5 and %rdx,%rbp 1b4: 48 09 c5 or %rax,%rbp 1b7: 49 39 dd cmp %rbx,%r13 1ba: 0f 86 31 01 00 00 jbe 2f1 <memmap_init_range+0x17f> 1c0: 45 85 f6 test %r14d,%r14d 1c3: 0f 85 b4 00 00 00 jne 27d <memmap_init_range+0x10b> 1c9: 49 83 fc 03 cmp $0x3,%r12 1cd: 0f 94 c1 sete %cl 1d0: 22 0d 00 00 00 00 and 0x0(%rip),%cl # 1d6 <memmap_init_range+0x64> 1d6: 0f 84 a1 00 00 00 je 27d <memmap_init_range+0x10b> 1dc: 48 8b 15 00 00 00 00 mov 0x0(%rip),%rdx # 1e3 <memmap_init_range+0x71> 1e3: 48 85 d2 test %rdx,%rdx 1e6: 74 10 je 1f8 <memmap_init_range+0x86> 1e8: 48 8b 42 08 mov 0x8(%rdx),%rax 1ec: 48 03 02 add (%rdx),%rax 1ef: 48 c1 e8 0c shr $0xc,%rax 1f3: 48 39 d8 cmp %rbx,%rax 1f6: 77 55 ja 24d <memmap_init_range+0xdb> 1f8: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # 1ff <memmap_init_range+0x8d> 1ff: 4c 6b 05 00 00 00 00 imul $0x18,0x0(%rip),%r8 # 207 <memmap_init_range+0x95> 206: 18 207: 31 f6 xor %esi,%esi 209: 48 89 05 00 00 00 00 mov %rax,0x0(%rip) # 210 <memmap_init_range+0x9e> 210: 49 01 c0 add %rax,%r8 213: 48 89 c7 mov %rax,%rdi 216: 4c 39 c0 cmp %r8,%rax 219: 73 26 jae 241 <memmap_init_range+0xcf> 21b: 48 8b 57 08 mov 0x8(%rdi),%rdx 21f: 48 03 17 add (%rdi),%rdx 222: 48 83 c0 18 add $0x18,%rax 226: 48 c1 ea 0c shr $0xc,%rdx 22a: 48 39 da cmp %rbx,%rdx 22d: 76 0e jbe 23d <memmap_init_range+0xcb> 22f: 40 84 f6 test %sil,%sil 232: 74 19 je 24d <memmap_init_range+0xdb> 234: 48 89 3d 00 00 00 00 mov %rdi,0x0(%rip) # 23b <memmap_init_range+0xc9> 23b: eb 10 jmp 24d <memmap_init_range+0xdb> 23d: 89 ce mov %ecx,%esi 23f: eb d2 jmp 213 <memmap_init_range+0xa1> 241: 40 84 f6 test %sil,%sil 244: 74 07 je 24d <memmap_init_range+0xdb> 246: 48 89 05 00 00 00 00 mov %rax,0x0(%rip) # 24d <memmap_init_range+0xdb> 24d: 48 8b 15 00 00 00 00 mov 0x0(%rip),%rdx # 254 <memmap_init_range+0xe2> 254: 48 8b 02 mov (%rdx),%rax 257: 48 8d 88 ff 0f 00 00 lea 0xfff(%rax),%rcx 25e: 48 c1 e9 0c shr $0xc,%rcx 262: 48 39 d9 cmp %rbx,%rcx 265: 77 16 ja 27d <memmap_init_range+0x10b> 267: f6 42 10 02 testb $0x2,0x10(%rdx) 26b: 74 10 je 27d <memmap_init_range+0x10b> 26d: 48 03 42 08 add 0x8(%rdx),%rax 271: 48 c1 e8 0c shr $0xc,%rax 275: 48 89 c3 mov %rax,%rbx 278: e9 3a ff ff ff jmp 1b7 <memmap_init_range+0x45> 27d: 48 89 df mov %rbx,%rdi 280: 48 c1 e7 06 shl $0x6,%rdi 284: 48 03 3d 00 00 00 00 add 0x0(%rip),%rdi # 28b <memmap_init_range+0x119> 28b: 48 8d 47 08 lea 0x8(%rdi),%rax 28f: c7 47 34 01 00 00 00 movl $0x1,0x34(%rdi) # Here's where the crash RIP is. 296: 48 c7 47 38 00 00 00 movq $0x0,0x38(%rdi) 29d: 00 29e: c7 47 30 ff ff ff ff movl $0xffffffff,0x30(%rdi) 2a5: 48 c7 47 28 00 00 00 movq $0x0,0x28(%rdi) 2ac: 00 2ad: 48 c7 47 20 00 00 00 movq $0x0,0x20(%rdi) 2b4: 00 2b5: 48 c7 47 18 00 00 00 movq $0x0,0x18(%rdi) 2bc: 00 2bd: 48 89 2f mov %rbp,(%rdi) 2c0: 48 89 47 08 mov %rax,0x8(%rdi) 2c4: 48 89 47 10 mov %rax,0x10(%rdi) 2c8: 41 83 fe 01 cmp $0x1,%r14d 2cc: 75 05 jne 2d3 <memmap_init_range+0x161> 2ce: 48 0f ba 2f 0c btsq $0xc,(%rdi) 2d3: f7 c3 ff 01 00 00 test $0x1ff,%ebx 2d9: 75 0e jne 2e9 <memmap_init_range+0x177> 2db: 8b 74 24 38 mov 0x38(%rsp),%esi 2df: e8 00 00 00 00 call 2e4 <memmap_init_range+0x172> 2e4: e8 00 00 00 00 call 2e9 <memmap_init_range+0x177> 2e9: 48 ff c3 inc %rbx 2ec: e9 c6 fe ff ff jmp 1b7 <memmap_init_range+0x45> 2f1: 5b pop %rbx 2f2: 5d pop %rbp 2f3: 41 5c pop %r12 2f5: 41 5d pop %r13 2f7: 41 5e pop %r14 2f9: c3 ret
Quoted 8 lines
Quoted 3 lines
Even nicer would be to give
an faddr2line of the RIP value and track down which C code was actually at fault.
arch_atomic_setarch/x86/include/asm/atomic.h:41of
INIT_LIST_HEADin__init_single_page, called frommemmap_init_range.
Looks like the first access to the memory map fails, although I think it's not in INIT_LIST_HEAD() but rather in init_page_count().
I'd start with making sure that page_alloc::memmap_alloc() actually returns accepted memory. If you build kernel with CONFIG_DEBUG_VM=y the memory map will poisoned in this function, so my guess is it'd crash there.
Quoted 2 lines
--
-Dionna Glaze, PhD (she/her)
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 8 lines
Looks like the first access to the memory map fails, although I think it's not in
INIT_LIST_HEAD()but rather ininit_page_count().I'd start with making sure that
page_alloc::memmap_alloc()actually returns accepted memory. If you build kernel withCONFIG_DEBUG_VM=ythe memory map will poisoned in this function, so my guess is it'd crash there.
That's a wonderful hint, thank you! I did not run this test CONFIG_DEBUG_VM set, but you think it's possible it could still be here?
Quoted 3 lines
--
Sincerely yours,
Mike.
Signature · 1 line
-Dionna Glaze, PhD (she/her)
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 4 lines · Dionna Amalie Glaze
Quoted 4 lines
Looks like the first access to the memory map fails, although I think it's not in
INIT_LIST_HEAD()but rather ininit_page_count().
Quoted 8 lines
Quoted 4 lines
I'd start with making sure that
page_alloc::memmap_alloc()actually returns accepted memory. If you build kernel withCONFIG_DEBUG_VM=ythe memory map will poisoned in this function, so my guess is it'd crash there.That's a wonderful hint, thank you! I did not run this test
CONFIG_DEBUG_VMset, but you think it's possible it could still be here?
It depends on how you configured your kernel. Say, defconfig does not set it.
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 10 lines · Mike Rapoport
Quoted 9 lines · Dionna Amalie Glaze
Quoted 8 lines
Looks like the first access to the memory map fails, although I think it's not in
INIT_LIST_HEAD()but rather ininit_page_count().I'd start with making sure that
page_alloc::memmap_alloc()actually returns accepted memory. If you build kernel withCONFIG_DEBUG_VM=ythe memory map will poisoned in this function, so my guess is it'd crash there.
Quoted 7 lines
Quoted 3 lines
That's a wonderful hint, thank you! I did not run this test
CONFIG_DEBUG_VMset, but you think it's possible it could still be here?It depends on how you configured your kernel. Say, defconfig does not set it.
I also hit the issue at 256GB. My config is using CONFIG_SPARSEMEM_VMEMMAP and fails in memmap_init_range() when attempting to add the first PFN. It looks like the underlying page that is backing the vmemmap has not been accepted (I receive a #VC 0x404 => page not validated).
Kirill, is this a path that you've looked at? It would appear that somewhere in the vmemmap_populate_hugepages() path, some memory acceptance needs to be done for the pages that are used to back vmemmap. I'm not very familiar with this code, so I'm not sure why everything works for a guest with 255GB of memory, but then fails for a guest with 256GB of memory.
Thanks,
Tom
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 24 lines · Tom Lendacky
Quoted 10 lines · Mike Rapoport
Quoted 9 lines · Dionna Amalie Glaze
Quoted 8 lines
Looks like the first access to the memory map fails, although I think it's not in
INIT_LIST_HEAD()but rather ininit_page_count().I'd start with making sure that
page_alloc::memmap_alloc()actually returns accepted memory. If you build kernel withCONFIG_DEBUG_VM=ythe memory map will poisoned in this function, so my guess is it'd crash there.Quoted 7 lines
Quoted 3 lines
That's a wonderful hint, thank you! I did not run this test
CONFIG_DEBUG_VMset, but you think it's possible it could still be here?It depends on how you configured your kernel. Say, defconfig does not set it.
I also hit the issue at 256GB. My config is using
CONFIG_SPARSEMEM_VMEMMAPand fails inmemmap_init_range()when attempting to add the first PFN. It looks like the underlying page that is backing the vmemmap has not been accepted (I receive a #VC0x404=> page not validated).
Quoted 5 lines
Kirill, is this a path that you've looked at? It would appear that somewhere in the
vmemmap_populate_hugepages()path, some memory acceptance needs to be done for the pages that are used to back vmemmap. I'm not very familiar with this code, so I'm not sure why everything works for a guest with 255GB of memory, but then fails for a guest with 256GB of memory.
Hm. I don't have machine that large at hands at the moment. And I have not looked at the codepath before.
I will try to look into the issue.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Mike Rapoport
maintainerRe: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 31 lines · Kirill A. Shutemov
Quoted 24 lines · Tom Lendacky
Quoted 10 lines · Mike Rapoport
Quoted 9 lines · Dionna Amalie Glaze
Quoted 8 lines
Looks like the first access to the memory map fails, although I think it's not in
INIT_LIST_HEAD()but rather ininit_page_count().I'd start with making sure that
page_alloc::memmap_alloc()actually returns accepted memory. If you build kernel withCONFIG_DEBUG_VM=ythe memory map will poisoned in this function, so my guess is it'd crash there.Quoted 7 lines
Quoted 3 lines
That's a wonderful hint, thank you! I did not run this test
CONFIG_DEBUG_VMset, but you think it's possible it could still be here?It depends on how you configured your kernel. Say, defconfig does not set it.
I also hit the issue at 256GB. My config is using
CONFIG_SPARSEMEM_VMEMMAPand fails inmemmap_init_range()when attempting to add the first PFN. It looks like the underlying page that is backing the vmemmap has not been accepted (I receive a #VC0x404=> page not validated).Quoted 5 lines
Kirill, is this a path that you've looked at? It would appear that somewhere in the
vmemmap_populate_hugepages()path, some memory acceptance needs to be done for the pages that are used to back vmemmap. I'm not very familiar with this code, so I'm not sure why everything works for a guest with 255GB of memory, but then fails for a guest with 256GB of memory.
Quoted 4 lines
Hm. I don't have machine that large at hands at the moment. And I have not looked at the codepath before.
I will try to look into the issue.
I'd add some printks to verify we actually try to accept the allocated memory. E.g. something like the patch below:
arch/x86/mm/unaccepted_memory.c +3 -0
@@ -22,6 +22,9 @@ void accept_memory(phys_addr_t start, phys_addr_t end)if (!boot_params.unaccepted_memory)return ; +if (system_state ==SYSTEM_BOOTING ) +pr_info ("%s: start: %pa end: %pa %pS\n" , __func__, &start, &end, (void *)_RET_IP_); + bitmap =__va (boot_params.unaccepted_memory); range_start = start /PMD_SIZE ;
mm/memblock.c +2 -2
@@ -1535,7 +1535,7 @@ void * __init memblock_alloc_exact_nid_raw( phys_addr_t min_addr, phys_addr_t max_addr, int nid) { - memblock_dbg("%s: %llu bytes align=0x%llx nid=%d from=%pa max_addr=%pa %pS\n", + pr_info("%s: %llu bytes align=0x%llx nid=%d from=%pa max_addr=%pa %pS\n", __func__, (u64 )size, (u64 )align, nid, &min_addr, &max_addr, (void *)_RET_IP_);
@@ -1567,7 +1567,7 @@ void * __init memblock_alloc_try_nid_raw( phys_addr_t min_addr, phys_addr_t max_addr, int nid) { - memblock_dbg("%s: %llu bytes align=0x%llx nid=%d from=%pa max_addr=%pa %pS\n", + pr_info("%s: %llu bytes align=0x%llx nid=%d from=%pa max_addr=%pa %pS\n", __func__, (u64 )size, (u64 )align, nid, &min_addr, &max_addr, (void *)_RET_IP_);
Quoted 3 lines
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
In reply to Kirill A. ShutemovQuoted 31 lines · Kirill A. Shutemov
Quoted 24 lines · Tom Lendacky
Quoted 10 lines · Mike Rapoport
Quoted 9 lines · Dionna Amalie Glaze
Quoted 8 lines
Looks like the first access to the memory map fails, although I think it's not in
INIT_LIST_HEAD()but rather ininit_page_count().I'd start with making sure that
page_alloc::memmap_alloc()actually returns accepted memory. If you build kernel withCONFIG_DEBUG_VM=ythe memory map will poisoned in this function, so my guess is it'd crash there.Quoted 7 lines
Quoted 3 lines
That's a wonderful hint, thank you! I did not run this test
CONFIG_DEBUG_VMset, but you think it's possible it could still be here?It depends on how you configured your kernel. Say, defconfig does not set it.
I also hit the issue at 256GB. My config is using
CONFIG_SPARSEMEM_VMEMMAPand fails inmemmap_init_range()when attempting to add the first PFN. It looks like the underlying page that is backing the vmemmap has not been accepted (I receive a #VC0x404=> page not validated).Quoted 5 lines
Kirill, is this a path that you've looked at? It would appear that somewhere in the
vmemmap_populate_hugepages()path, some memory acceptance needs to be done for the pages that are used to back vmemmap. I'm not very familiar with this code, so I'm not sure why everything works for a guest with 255GB of memory, but then fails for a guest with 256GB of memory.
Quoted 4 lines
Hm. I don't have machine that large at hands at the moment. And I have not looked at the codepath before.
I will try to look into the issue.
I'm not able to trigger the bug.
With help of vm.overcommit_memory=1, I was managed boot TDX guest to shell with 256G and 1T of guest memory just fine.
Any chance it is SEV-SNP specific?
Or maybe there some difference in kernel config? Could you share yours?
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 35 lines · Kirill A. Shutemov
Quoted 34 lines · Kirill A. Shutemov
Quoted 24 lines · Tom Lendacky
Quoted 10 lines · Mike Rapoport
Quoted 9 lines · Dionna Amalie Glaze
Quoted 8 lines
Looks like the first access to the memory map fails, although I think it's not in
INIT_LIST_HEAD()but rather ininit_page_count().I'd start with making sure that
page_alloc::memmap_alloc()actually returns accepted memory. If you build kernel withCONFIG_DEBUG_VM=ythe memory map will poisoned in this function, so my guess is it'd crash there.Quoted 7 lines
Quoted 3 lines
That's a wonderful hint, thank you! I did not run this test
CONFIG_DEBUG_VMset, but you think it's possible it could still be here?It depends on how you configured your kernel. Say, defconfig does not set it.
I also hit the issue at 256GB. My config is using
CONFIG_SPARSEMEM_VMEMMAPand fails inmemmap_init_range()when attempting to add the first PFN. It looks like the underlying page that is backing the vmemmap has not been accepted (I receive a #VC0x404=> page not validated).Quoted 5 lines
Kirill, is this a path that you've looked at? It would appear that somewhere in the
vmemmap_populate_hugepages()path, some memory acceptance needs to be done for the pages that are used to back vmemmap. I'm not very familiar with this code, so I'm not sure why everything works for a guest with 255GB of memory, but then fails for a guest with 256GB of memory.Hm. I don't have machine that large at hands at the moment. And I have not looked at the codepath before.
Quoted 8 lines
Quoted 1 line
I will try to look into the issue.
I'm not able to trigger the bug.
With help of vm.overcommit_memory=1, I was managed boot TDX guest to shell with 256G and 1T of guest memory just fine.
Any chance it is SEV-SNP specific?
There's always a chance. I'll do some more tracing and see what I can find to try and be certain.
Quoted 2 lines
Or maybe there some difference in kernel config? Could you share yours?
Yes, I'll send that to you off-list.
Thanks,
Tom
Quoted 1 line
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
Quoted 45 lines · Tom Lendacky
Quoted 35 lines · Kirill A. Shutemov
Quoted 34 lines · Kirill A. Shutemov
Quoted 24 lines · Tom Lendacky
Quoted 10 lines · Mike Rapoport
Quoted 9 lines · Dionna Amalie Glaze
Quoted 8 lines
Looks like the first access to the memory map fails, although I think it's not in
INIT_LIST_HEAD()but rather ininit_page_count().I'd start with making sure that
page_alloc::memmap_alloc()actually returns accepted memory. If you build kernel withCONFIG_DEBUG_VM=ythe memory map will poisoned in this function, so my guess is it'd crash there.Quoted 7 lines
Quoted 3 lines
That's a wonderful hint, thank you! I did not run this test
CONFIG_DEBUG_VMset, but you think it's possible it could still be here?It depends on how you configured your kernel. Say, defconfig does not set it.
I also hit the issue at 256GB. My config is using
CONFIG_SPARSEMEM_VMEMMAPand fails inmemmap_init_range()when attempting to add the first PFN. It looks like the underlying page that is backing the vmemmap has not been accepted (I receive a #VC0x404=> page not validated).Quoted 5 lines
Kirill, is this a path that you've looked at? It would appear that somewhere in the
vmemmap_populate_hugepages()path, some memory acceptance needs to be done for the pages that are used to back vmemmap. I'm not very familiar with this code, so I'm not sure why everything works for a guest with 255GB of memory, but then fails for a guest with 256GB of memory.Hm. I don't have machine that large at hands at the moment. And I have not looked at the codepath before.
Quoted 8 lines
Quoted 1 line
I will try to look into the issue.
I'm not able to trigger the bug.
With help of vm.overcommit_memory=1, I was managed boot TDX guest to shell with 256G and 1T of guest memory just fine.
Any chance it is SEV-SNP specific?
Quoted 7 lines
There's always a chance. I'll do some more tracing and see what I can find to try and be certain.
Quoted 2 lines
Or maybe there some difference in kernel config? Could you share yours?
Yes, I'll send that to you off-list.
Still nothing with your config :/
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCHv7 02/14] mm: Add support for unaccepted memory
In reply to Tom LendackyQuoted 38 lines · Tom Lendacky
Quoted 37 lines · Kirill A. Shutemov
Quoted 35 lines · Kirill A. Shutemov
Quoted 24 lines · Tom Lendacky
Quoted 10 lines · Mike Rapoport
Quoted 9 lines · Dionna Amalie Glaze
Quoted 8 lines
Looks like the first access to the memory map fails, although I think it's not in
INIT_LIST_HEAD()but rather ininit_page_count().I'd start with making sure that
page_alloc::memmap_alloc()actually returns accepted memory. If you build kernel withCONFIG_DEBUG_VM=ythe memory map will poisoned in this function, so my guess is it'd crash there.Quoted 7 lines
Quoted 3 lines
That's a wonderful hint, thank you! I did not run this test
CONFIG_DEBUG_VMset, but you think it's possible it could still be here?It depends on how you configured your kernel. Say, defconfig does not set it.
I also hit the issue at 256GB. My config is using
CONFIG_SPARSEMEM_VMEMMAPand fails inmemmap_init_range()when attempting to add the first PFN. It looks like the underlying page that is backing the vmemmap has not been accepted (I receive a #VC0x404=> page not validated).Quoted 5 lines
Kirill, is this a path that you've looked at? It would appear that somewhere in the
vmemmap_populate_hugepages()path, some memory acceptance needs to be done for the pages that are used to back vmemmap. I'm not very familiar with this code, so I'm not sure why everything works for a guest with 255GB of memory, but then fails for a guest with 256GB of memory.Hm. I don't have machine that large at hands at the moment. And I have not looked at the codepath before.
I will try to look into the issue.
Quoted 8 lines
Quoted 6 lines
I'm not able to trigger the bug.
With help of vm.overcommit_memory=1, I was managed boot TDX guest to shell with 256G and 1T of guest memory just fine.
Any chance it is SEV-SNP specific?
There's always a chance. I'll do some more tracing and see what I can find to try and be certain.
Indeed, it was an issue in the SNP path, shifting the number of pages as an unsigned int by PAGE_SIZE and adding to an unsigned long. The value to be shifted was 0x100000 and resulted in 0. Changing the number of pages to an unsigned long fixed it.
There are a few places where that is being done, so I'll address those with some pre-patches as fixes.
Thanks for verifying it was working for you.
Tom
Quoted 10 lines
Quoted 3 lines
Or maybe there some difference in kernel config? Could you share yours?
Yes, I'll send that to you off-list.
Thanks,
TomQuoted 1 line
PATCH v7 3/14 mm: Report unaccepted memory in meminfo
In reply to Kirill A. ShutemovTrack amount of unaccepted memory and report it in /proc/meminfo and in node meminfo.
--- drivers/base/node.c | 7 +++++++ fs/proc/meminfo.c | 5 +++++ include/linux/mmzone.h | 1 + mm/page_alloc.c | 9 ++++++++- mm/vmstat.c | 1 + 5 files changed, 22 insertions(+), 1 deletion(-)
drivers/base/node.c +7 -0
@@ -446,6 +446,9 @@ static ssize_t node_read_meminfo(struct device *dev,"Node %d ShmemPmdMapped: %8lu kB\n" "Node %d FileHugePages: %8lu kB\n" "Node %d FilePmdMapped: %8lu kB\n" +#endif +#ifdef CONFIG_UNACCEPTED_MEMORY +"Node %d UnacceptedPages: %8lu kB\n" #endif , nid, K(node_page_state(pgdat, NR_FILE_DIRTY)),
@@ -474,6 +477,10 @@ static ssize_t node_read_meminfo(struct device *dev, nid,K (node_page_state (pgdat,NR_SHMEM_PMDMAPPED )), nid,K (node_page_state (pgdat,NR_FILE_THPS )), nid, K(node_page_state(pgdat, NR_FILE_PMDMAPPED)) +#endif +#ifdef CONFIG_UNACCEPTED_MEMORY + , + nid,K (node_page_state (pgdat,NR_UNACCEPTED )) #endif ); len +=hugetlb_report_node_meminfo (buf, len, nid);
fs/proc/meminfo.c +5 -0
@@ -153,6 +153,11 @@ static int meminfo_proc_show(struct seq_file *m, void *v) global_zone_page_state(NR_FREE_CMA_PAGES));#endif +#ifdef CONFIG_UNACCEPTED_MEMORY +show_val_kb (m,"UnacceptedPages:" , +global_node_page_state (NR_UNACCEPTED )); +#endif +hugetlb_report_meminfo (m);arch_report_meminfo (m);
include/linux/mmzone.h +1 -0
@@ -212,6 +212,7 @@ enum node_stat_item {NR_FOLL_PIN_ACQUIRED ,/* via: pin_user_page(), gup flag: FOLL_PIN */ NR_FOLL_PIN_RELEASED ,/* pages returned via unpin_user_page() */ NR_KERNEL_STACK_KB ,/* measured in KiB */ +NR_UNACCEPTED , #if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) NR_KERNEL_SCS_KB, /* measured in KiB */ #endif
mm/page_alloc.c +8 -1
@@ -1012,6 +1012,7 @@ static void accept_page(struct page *page, unsigned int order)accept_memory (start, start + (PAGE_SIZE << order));__ClearPageUnaccepted (page); +mod_node_page_state (page_pgdat (page),NR_UNACCEPTED , -(1 << order));/* Assert that there is no PageUnaccepted() on tail pages */ if (IS_ENABLED (CONFIG_DEBUG_VM )) {
@@ -1063,6 +1064,7 @@ static inline void __free_one_page(struct page *page,struct page *buddy;bool to_tail;bool page_needs_acceptance = false; +int nr_unaccepted =0 ;VM_BUG_ON (!zone_is_initialized (zone));VM_BUG_ON_PAGE (page->flags &PAGE_FLAGS_CHECK_AT_PREP , page);
@@ -1076,6 +1078,7 @@ static inline void __free_one_page(struct page *page,if (PageUnaccepted (page)) { page_needs_acceptance = true; + nr_unaccepted +=1 << order;__ClearPageUnaccepted (page); }
@@ -1117,6 +1120,7 @@ static inline void __free_one_page(struct page *page,/* Mark page unaccepted if any of merged pages were unaccepted */ if (PageUnaccepted (buddy)) { page_needs_acceptance = true; + nr_unaccepted +=1 << order;__ClearPageUnaccepted (buddy); }
@@ -1143,8 +1147,11 @@ static inline void __free_one_page(struct page *page,*/ if (!page_needs_acceptance && (fpi_flags &FPI_UNACCEPTED_SLOWPATH )) page_needs_acceptance =page_contains_unaccepted (page, order); -if (page_needs_acceptance) +if (page_needs_acceptance) {__SetPageUnaccepted (page); +__mod_node_page_state (page_pgdat (page),NR_UNACCEPTED , + (1 << order) - nr_unaccepted); + }if (fpi_flags &FPI_TO_TAIL ) to_tail = true;
mm/vmstat.c +1 -0
@@ -1236,6 +1236,7 @@ const char * const vmstat_text[] = {"nr_foll_pin_acquired" ,"nr_foll_pin_released" ,"nr_kernel_stack" , +"nr_unaccepted" , #if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) "nr_shadow_call_stack", #endif
Signature · 1 line
2.35.1
Re: [PATCHv7 03/14] mm: Report unaccepted memory in meminfo
Quoted 17 lines · Kirill A. Shutemov
Track amount of unaccepted memory and report it in
/proc/meminfoand in node meminfo.Signed-off-by Kirill A. Shutemov <[email protected]>--- drivers/base/node.c | 7 +++++++ fs/proc/meminfo.c | 5 +++++ include/linux/mmzone.h | 1 + mm/page_alloc.c | 9 ++++++++- mm/vmstat.c | 1 + 5 files changed, 22 insertions(+), 1 deletion(-)drivers/base/node.c +0
-0
drivers/base/node.c static ssize_t node_read_meminfo(struct device *dev, · line 451
Quoted 6 lines
drivers/base/node.c +3
-0@@ -446,6 +446,9 @@ static ssize_t node_read_meminfo(struct device *dev,"Node %d ShmemPmdMapped: %8lu kB\n" "Node %d FileHugePages: %8lu kB\n" "Node %d FilePmdMapped: %8lu kB\n" +#endif +#ifdef CONFIG_UNACCEPTED_MEMORY +"Node %d UnacceptedPages: %8lu kB\n"
Nit: I'd just call that "Unaccepted", as the unit is kB.
Quoted 19 lines
drivers/base/node.c +4
-0#endif , nid, K(node_page_state(pgdat, NR_FILE_DIRTY)),@@ -474,6 +477,10 @@ static ssize_t node_read_meminfo(struct device *dev, nid,K (node_page_state (pgdat,NR_SHMEM_PMDMAPPED )), nid,K (node_page_state (pgdat,NR_FILE_THPS )), nid, K(node_page_state(pgdat, NR_FILE_PMDMAPPED)) +#endif +#ifdef CONFIG_UNACCEPTED_MEMORY + , + nid,K (node_page_state (pgdat,NR_UNACCEPTED )) #endif ); len +=hugetlb_report_node_meminfo (buf, len, nid);fs/proc/meminfo.c +0
-0
fs/proc/meminfo.c static int meminfo_proc_show(struct seq_file *m, void *v) · line 158
Quoted 6 lines
fs/proc/meminfo.c +3
-0@@ -153,6 +153,11 @@ static int meminfo_proc_show(struct seq_file *m, void *v) global_zone_page_state(NR_FREE_CMA_PAGES));#endif +#ifdef CONFIG_UNACCEPTED_MEMORY +show_val_kb (m,"UnacceptedPages:" , +global_node_page_state (NR_UNACCEPTED ));
Dito.
Quoted 10 lines
fs/proc/meminfo.c +2
-0+#endif +hugetlb_report_meminfo (m);arch_report_meminfo (m);include/linux/mmzone.h +0
-0
include/linux/mmzone.h enum node_stat_item { · line 215
Quoted 4 lines
include/linux/mmzone.h +1
-0@@ -212,6 +212,7 @@ enum node_stat_item {NR_FOLL_PIN_ACQUIRED ,/* via: pin_user_page(), gup flag: FOLL_PIN */ NR_FOLL_PIN_RELEASED ,/* pages returned via unpin_user_page() */ NR_KERNEL_STACK_KB ,/* measured in KiB */ + NR_UNACCEPTED,
Do we want to ifdef that as well? (and in all other cases below)
Quoted 57 lines
include/linux/mmzone.h +0
-0#if IS_ENABLED (CONFIG_SHADOW_CALL_STACK ) NR_KERNEL_SCS_KB, /* measured in KiB */#endif mm/page_alloc.c +8
-1@@ -1012,6 +1012,7 @@ static void accept_page(struct page *page, unsigned int order)accept_memory (start, start + (PAGE_SIZE << order));__ClearPageUnaccepted (page); +mod_node_page_state (page_pgdat (page),NR_UNACCEPTED , -(1 << order));/* Assert that there is no PageUnaccepted() on tail pages */ if (IS_ENABLED (CONFIG_DEBUG_VM )) {@@ -1063,6 +1064,7 @@ static inline void __free_one_page(struct page *page,struct page *buddy;bool to_tail;bool page_needs_acceptance = false; +int nr_unaccepted =0 ;VM_BUG_ON (!zone_is_initialized (zone));VM_BUG_ON_PAGE (page->flags &PAGE_FLAGS_CHECK_AT_PREP , page);@@ -1076,6 +1078,7 @@ static inline void __free_one_page(struct page *page,if (PageUnaccepted (page)) { page_needs_acceptance = true; + nr_unaccepted +=1 << order;__ClearPageUnaccepted (page); }@@ -1117,6 +1120,7 @@ static inline void __free_one_page(struct page *page,/* Mark page unaccepted if any of merged pages were unaccepted */ if (PageUnaccepted (buddy)) { page_needs_acceptance = true; + nr_unaccepted +=1 << order;__ClearPageUnaccepted (buddy); }@@ -1143,8 +1147,11 @@ static inline void __free_one_page(struct page *page,*/ if (!page_needs_acceptance && (fpi_flags &FPI_UNACCEPTED_SLOWPATH )) page_needs_acceptance =page_contains_unaccepted (page, order); -if (page_needs_acceptance) +if (page_needs_acceptance) {__SetPageUnaccepted (page); +__mod_node_page_state (page_pgdat (page),NR_UNACCEPTED , + (1 << order) - nr_unaccepted); + }if (fpi_flags &FPI_TO_TAIL ) to_tail = true;mm/vmstat.c +0
-0
mm/vmstat.c const char * const vmstat_text[] = { · line 1242
Quoted 7 lines
mm/vmstat.c +1
-0@@ -1236,6 +1236,7 @@ const char * const vmstat_text[] = {"nr_foll_pin_acquired" ,"nr_foll_pin_released" ,"nr_kernel_stack" , +"nr_unaccepted" , #if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) "nr_shadow_call_stack", #endif
LGTM
Signature · 3 lines
Thanks, David / dhildenb
PATCH v7 4/14 efi/x86: Get full memory map in allocate_e820()
In reply to Kirill A. ShutemovCurrently allocate_e820() only interested in the size of map and size of memory descriptor to determine how many e820 entries the kernel needs.
UEFI Specification version 2.9 introduces a new memory type -- unaccepted memory. To track unaccepted memory kernel needs to allocate a bitmap. The size of the bitmap is dependent on the maximum physical address present in the system. A full memory map is required to find the maximum address.
Modify allocate_e820() to get a full memory map.
This is preparation for the next patch that implements handling of unaccepted memory in EFI stub.
--- drivers/firmware/efi/libstub/x86-stub.c | 28 +++++++++++-------------- 1 file changed, 12 insertions(+), 16 deletions(-)
drivers/firmware/efi/libstub/x86-stub.c +12 -16
@@ -672,31 +672,27 @@ static efi_status_t alloc_e820ext(u32 nr_desc, struct setup_data **e820ext, }static efi_status_t allocate_e820 (struct boot_params *params, +struct efi_boot_memmap *map,struct setup_data **e820ext,u32 *e820ext_size) { -unsigned long map_size, desc_size, map_key;efi_status_t status; -__u32 nr_desc, desc_version; +__u32 nr_desc; -/* Only need the size of the mem map and size of each mem descriptor */ - map_size =0 ; - status =efi_bs_call (get_memory_map, &map_size,NULL , &map_key, - &desc_size, &desc_version); -if (status !=EFI_BUFFER_TOO_SMALL ) -return (status !=EFI_SUCCESS ) ? status :EFI_UNSUPPORTED ; - - nr_desc = map_size / desc_size +EFI_MMAP_NR_SLACK_SLOTS ; + status =efi_get_memory_map (map); +if (status !=EFI_SUCCESS ) +return status; -if (nr_desc >ARRAY_SIZE (params->e820_table)) { -u32 nr_e820ext = nr_desc -ARRAY_SIZE (params->e820_table); + nr_desc = *map->map_size / *map->desc_size; +if (nr_desc >ARRAY_SIZE (params->e820_table) -EFI_MMAP_NR_SLACK_SLOTS ) { +u32 nr_e820ext = nr_desc -ARRAY_SIZE (params->e820_table) + +EFI_MMAP_NR_SLACK_SLOTS ; status =alloc_e820ext (nr_e820ext, e820ext, e820ext_size); -if (status !=EFI_SUCCESS ) -return status; } -return EFI_SUCCESS ; +efi_bs_call (free_pool, *map->map); +return status; }struct exit_boot_struct {
@@ -745,7 +741,7 @@ static efi_status_t exit_boot(struct boot_params *boot_params, void *handle) priv.boot_params = boot_params; priv.efi = &boot_params->efi_info; - status =allocate_e820 (boot_params, &e820ext, &e820ext_size); + status =allocate_e820 (boot_params, &map, &e820ext, &e820ext_size);if (status !=EFI_SUCCESS )return status;
Signature · 1 line
2.35.1
Re: [PATCHv7 04/14] efi/x86: Get full memory map in allocate_e820()
Quoted 1 line · Kirill A. Shutemov
Currently
allocate_e820()only interested in the size of map and size of
^ is
Quoted 8 lines
memory descriptor to determine how many e820 entries the kernel needs.
UEFI Specification version 2.9 introduces a new memory type -- unaccepted memory. To track unaccepted memory kernel needs to allocate a bitmap. The size of the bitmap is dependent on the maximum physical address present in the system. A full memory map is required to find the maximum address.
Quoted 4 lines
Modify
allocate_e820()to get a full memory map.This is preparation for the next patch that implements handling of unaccepted memory in EFI stub.
As already pointed out, the concept of "next patch" is ambiguous in git. Just drop the whole sentence.
Quoted 4 lines
Signed-off-by Kirill A. Shutemov <[email protected]>--- drivers/firmware/efi/libstub/x86-stub.c | 28 +++++++++++-------------- 1 file changed, 12 insertions(+), 16 deletions(-)
With the above addressed:
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v7 5/14 x86/boot: Add infrastructure required for unaccepted memory support
In reply to Kirill A. ShutemovPull functionality from the main kernel headers and lib/ that is required for unaccepted memory support.
This is preparatory patch. The users for the functionality will come in following patches.
--- arch/x86/boot/bitops.h | 40 ++++++++++++ arch/x86/boot/compressed/align.h | 14 +++++ arch/x86/boot/compressed/bitmap.c | 43 +++++++++++++ arch/x86/boot/compressed/bitmap.h | 49 +++++++++++++++ arch/x86/boot/compressed/bits.h | 36 +++++++++++ arch/x86/boot/compressed/compiler.h | 9 +++ arch/x86/boot/compressed/find.c | 54 ++++++++++++++++ arch/x86/boot/compressed/find.h | 80 ++++++++++++++++++++++++ arch/x86/boot/compressed/math.h | 37 +++++++++++ arch/x86/boot/compressed/minmax.h | 61 ++++++++++++++++++ arch/x86/boot/compressed/pgtable_types.h | 25 ++++++++ 11 files changed, 448 insertions(+) create mode 100644 arch/x86/boot/compressed/align.h create mode 100644 arch/x86/boot/compressed/bitmap.c create mode 100644 arch/x86/boot/compressed/bitmap.h create mode 100644 arch/x86/boot/compressed/bits.h create mode 100644 arch/x86/boot/compressed/compiler.h create mode 100644 arch/x86/boot/compressed/find.c create mode 100644 arch/x86/boot/compressed/find.h create mode 100644 arch/x86/boot/compressed/math.h create mode 100644 arch/x86/boot/compressed/minmax.h create mode 100644 arch/x86/boot/compressed/pgtable_types.h
arch/x86/boot/bitops.h +40 -0
@@ -41,4 +41,44 @@ static inline void set_bit(int nr, void *addr) asm("btsl %1,%0" :"+m" (*(u32 *)addr) :"Ir" (nr)); } +static __always_inline void __set_bit(long nr, volatile unsigned long *addr) +{ + asm volatile(__ASM_SIZE(bts) " %1,%0" : : "m" (*(volatile long *) addr), +"Ir" (nr) :"memory" ); +} + +static __always_inline void __clear_bit(long nr, volatile unsigned long *addr) +{ + asm volatile(__ASM_SIZE(btr) " %1,%0" : : "m" (*(volatile long *) addr), +"Ir" (nr) :"memory" ); +} + +/** +* __ffs - find first set bit in word +* @word: The word to search +* +* Undefined if no bit exists, so code should check against 0 first. +*/ +static __always_inline unsigned long __ffs (unsigned long word) +{ + asm("rep; bsf %1,%0" + :"=r" (word) + :"rm" (word)); +return word; +} + +/** +* ffz - find first zero bit in word +* @word: The word to search +* +* Undefined if no zero exists, so code should check against ~0UL first. +*/ +static __always_inline unsigned long ffz (unsigned long word) +{ + asm("rep; bsf %1,%0" + :"=r" (word) + :"r" (~word)); +return word; +} +#endif /* BOOT_BITOPS_H */
arch/x86/boot/compressed/align.h new file +14 -0
@@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef BOOT_ALIGN_H +#define BOOT_ALIGN_H +#define _LINUX_ALIGN_H/* Inhibit inclusion of <linux/align.h> */ + +/* @a is a power of 2 value */ +#define ALIGN (x, a) __ALIGN_KERNEL((x), (a)) +#define ALIGN_DOWN (x, a) __ALIGN_KERNEL((x) - ((a) - 1), (a)) +#define __ALIGN_MASK (x, mask) __ALIGN_KERNEL_MASK((x), (mask)) +#define PTR_ALIGN (p, a) ((typeof(p))ALIGN((unsigned long)(p), (a))) +#define PTR_ALIGN_DOWN (p, a) ((typeof(p))ALIGN_DOWN((unsigned long)(p), (a))) +#define IS_ALIGNED (x, a) (((x) & ((typeof(x))(a) - 1)) == 0) + +#endif
arch/x86/boot/compressed/bitmap.c new file +43 -0
43 lines not on this page · raw message
arch/x86/boot/compressed/bitmap.h new file +49 -0
49 lines not on this page · raw message
arch/x86/boot/compressed/bits.h new file +36 -0
@@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef BOOT_BITS_H +#define BOOT_BITS_H +#define __LINUX_BITS_H/* Inhibit inclusion of <linux/bits.h> */ + +#ifdef __ASSEMBLY__ +#define _AC (X ,Y ) X +#define _AT (T ,X ) X +#else +#define __AC (X ,Y ) (X##Y) +#define _AC (X ,Y ) __AC(X,Y) +#define _AT (T ,X ) ((T)(X)) +#endif + +#define _UL (x) (_AC(x, UL)) +#define _ULL (x) (_AC(x, ULL)) +#define UL (x) (_UL(x)) +#define ULL (x) (_ULL(x)) + +#define BIT (nr) (UL(1) << (nr)) +#define BIT_ULL (nr) (ULL(1) << (nr)) +#define BIT_MASK (nr) (UL(1) << ((nr) % BITS_PER_LONG)) +#define BIT_WORD (nr) ((nr) / BITS_PER_LONG) +#define BIT_ULL_MASK (nr) (ULL(1) << ((nr) % BITS_PER_LONG_LONG)) +#define BIT_ULL_WORD (nr) ((nr) / BITS_PER_LONG_LONG) +#define BITS_PER_BYTE 8 + +#define GENMASK (h, l) \ + (((~UL(0)) - (UL(1) << (l)) + 1) & \ + (~UL(0) >> (BITS_PER_LONG - 1 - (h)))) + +#define GENMASK_ULL (h, l) \ + (((~ULL(0)) - (ULL(1) << (l)) + 1) & \ + (~ULL(0) >> (BITS_PER_LONG_LONG - 1 - (h)))) + +#endif
arch/x86/boot/compressed/compiler.h new file +9 -0
@@ -0,0 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef BOOT_COMPILER_H +#define BOOT_COMPILER_H +#define __LINUX_COMPILER_H/* Inhibit inclusion of <linux/compiler.h> */ + +# define likely (x) __builtin_expect(!!(x), 1) +# define unlikely (x) __builtin_expect(!!(x), 0) + +#endif
arch/x86/boot/compressed/find.c new file +54 -0
54 lines not on this page · raw message
arch/x86/boot/compressed/find.h new file +80 -0
80 lines not on this page · raw message
arch/x86/boot/compressed/math.h new file +37 -0
@@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef BOOT_MATH_H +#define BOOT_MATH_H +#define __LINUX_MATH_H/* Inhibit inclusion of <linux/math.h> */ + +/* +* +* This looks more complex than it should be. But we need to +* get the type for the ~ right in round_down (it needs to be +* as wide as the result!), and we want to evaluate the macro +* arguments just once each. +*/ +#define __round_mask (x, y) ((__typeof__(x))((y)-1)) + +/** +* round_up - round up to next specified power of 2 +* @x: the value to round +* @y: multiple to round up to (must be a power of 2) +* +* Rounds @x up to next multiple of @y (which must be a power of 2). +* To perform arbitrary rounding up, use roundup() below. +*/ +#define round_up (x, y) ((((x)-1) | __round_mask(x, y))+1) + +/** +* round_down - round down to next specified power of 2 +* @x: the value to round +* @y: multiple to round down to (must be a power of 2) +* +* Rounds @x down to next multiple of @y (which must be a power of 2). +* To perform arbitrary rounding down, use rounddown() below. +*/ +#define round_down (x, y) ((x) & ~__round_mask(x, y)) + +#define DIV_ROUND_UP (n, d) (((n) + (d) - 1) / (d)) + +#endif
arch/x86/boot/compressed/minmax.h new file +61 -0
61 lines not on this page · raw message
arch/x86/boot/compressed/pgtable_types.h new file +25 -0
@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef BOOT_COMPRESSED_PGTABLE_TYPES_H +#define BOOT_COMPRESSED_PGTABLE_TYPES_H +#define _ASM_X86_PGTABLE_DEFS_H/* Inhibit inclusion of <asm/pgtable_types.h> */ + +#define PAGE_SHIFT 12 + +#ifdef CONFIG_X86_64 +#define PTE_SHIFT 9 +#elif definedCONFIG_X86_PAE +#define PTE_SHIFT 9 +#else /* 2-level */ +#define PTE_SHIFT 10 +#endif + +enum pg_level { +PG_LEVEL_NONE , +PG_LEVEL_4K , +PG_LEVEL_2M , +PG_LEVEL_1G , +PG_LEVEL_512G , +PG_LEVEL_NUM +}; + +#endif
Signature · 1 line
2.35.1
Peter Zijlstra
maintainerRe: [PATCHv7 05/14] x86/boot: Add infrastructure required for unaccepted memory support
Quoted 7 lines · Kirill A. Shutemov
Pull functionality from the main kernel headers and
lib/that is required for unaccepted memory support.This is preparatory patch. The users for the functionality will come in following patches.
Signed-off-by Kirill A. Shutemov <[email protected]>
Quoted 12 lines
--- arch/x86/boot/bitops.h | 40 ++++++++++++ arch/x86/boot/compressed/align.h | 14 +++++ arch/x86/boot/compressed/bitmap.c | 43 +++++++++++++ arch/x86/boot/compressed/bitmap.h | 49 +++++++++++++++ arch/x86/boot/compressed/bits.h | 36 +++++++++++ arch/x86/boot/compressed/compiler.h | 9 +++ arch/x86/boot/compressed/find.c | 54 ++++++++++++++++ arch/x86/boot/compressed/find.h | 80 ++++++++++++++++++++++++ arch/x86/boot/compressed/math.h | 37 +++++++++++ arch/x86/boot/compressed/minmax.h | 61 ++++++++++++++++++ arch/x86/boot/compressed/pgtable_types.h | 25 ++++++++
That's quite a lot of duplicated code; is there really no way so share this?
Re: [PATCHv7 05/14] x86/boot: Add infrastructure required for unaccepted memory support
Quoted 21 lines · Peter Zijlstra
Quoted 7 lines · Kirill A. Shutemov
Pull functionality from the main kernel headers and
lib/that is required for unaccepted memory support.This is preparatory patch. The users for the functionality will come in following patches.
Signed-off-by Kirill A. Shutemov <[email protected]>Quoted 12 lines
--- arch/x86/boot/bitops.h | 40 ++++++++++++ arch/x86/boot/compressed/align.h | 14 +++++ arch/x86/boot/compressed/bitmap.c | 43 +++++++++++++ arch/x86/boot/compressed/bitmap.h | 49 +++++++++++++++ arch/x86/boot/compressed/bits.h | 36 +++++++++++ arch/x86/boot/compressed/compiler.h | 9 +++ arch/x86/boot/compressed/find.c | 54 ++++++++++++++++ arch/x86/boot/compressed/find.h | 80 ++++++++++++++++++++++++ arch/x86/boot/compressed/math.h | 37 +++++++++++ arch/x86/boot/compressed/minmax.h | 61 ++++++++++++++++++ arch/x86/boot/compressed/pgtable_types.h | 25 ++++++++
Quoted 2 lines
That's quite a lot of duplicated code; is there really no way so share this?
Code duplication also make me uncomfortable. But that what Borislav wanted to see. efi.h in the boot stub which copies bulk of <linux/efi.h> also sets the trend in the direction.
Alternative is creating a subset of headers that can be used in both in main kernel and boot stub. It is more complex and doesn't allow for short cuts that can be made on copy if you know the context it is used in.
It also sounds painfully similar to uapi/ project. I'm not sure we want to go this path.
Signature · 1 line
Kirill A. Shutemov
Borislav Petkov
maintainerRe: [PATCHv7 05/14] x86/boot: Add infrastructure required for unaccepted memory support
Quoted 2 lines · Kirill A. Shutemov
It also sounds painfully similar to uapi/ project. I'm not sure we want to go this path.
It is the same path perf tool is taking - see first paragraph:
https://lore.kernel.org/r/[email protected]
I don't want to deal with the regular breakages or hacks to boot/compressed/ just because little duplication. We copy the header once and that's it - it doesn't even have to get updated like perf tool does from time to time.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteBorislav Petkov
maintainerRe: [PATCHv7 05/14] x86/boot: Add infrastructure required for unaccepted memory support
In reply to Kirill A. ShutemovQuoted 6 lines · Kirill A. Shutemov
arch/x86/boot/compressed/compiler.h new file +0
-0
arch/x86/boot/compressed/compiler.h line 9
Quoted 9 lines
arch/x86/boot/compressed/compiler.h new file +9
-0@@ -0,0 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef BOOT_COMPILER_H +#define BOOT_COMPILER_H +#define __LINUX_COMPILER_H/* Inhibit inclusion of <linux/compiler.h> */ + +# define likely (x) __builtin_expect(!!(x), 1) +# define unlikely (x) __builtin_expect(!!(x), 0) + +#endif
I guess that header is not really needed - simply drop the annotations from the code too.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v7 6/14 efi/x86: Implement support for unaccepted memory
In reply to Kirill A. ShutemovUEFI Specification version 2.9 introduces the concept of memory
acceptance: Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, requiring memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific for the Virtual Machine platform.
Accepting memory is costly and it makes VMM allocate memory for the accepted guest physical address range. It's better to postpone memory acceptance until memory is needed. It lowers boot time and reduces memory overhead.
The kernel needs to know what memory has been accepted. Firmware communicates this information via memory map: a new memory type -- EFI_UNACCEPTED_MEMORY -- indicates such memory.
Range-based tracking works fine for firmware, but it gets bulky for the kernel: e820 has to be modified on every page acceptance. It leads to table fragmentation, but there's a limited number of entries in the e820 table
Another option is to mark such memory as usable in e820 and track if the range has been accepted in a bitmap. One bit in the bitmap represents 2MiB in the address space: one 4k page is enough to track 64GiB or physical address space.
In the worst-case scenario -- a huge hole in the middle of the address space -- It needs 256MiB to handle 4PiB of the address space.
Any unaccepted memory that is not aligned to 2M gets accepted upfront.
The bitmap is allocated and constructed in the EFI stub and passed down to the kernel via boot_params. allocate_e820() allocates the bitmap if unaccepted memory is present, according to the maximum address in the memory map.
The same boot_params.unaccepted_memory can be used to pass the bitmap between two kernels on kexec, but the use-case is not yet implemented.
The implementation requires some basic helpers in boot stub. They provided by linux/ includes in the main kernel image, but is not present in boot stub. Create copy of required functionality in the boot stub.
--- Documentation/x86/zero-page.rst | 1 + arch/x86/boot/compressed/Makefile | 1 + arch/x86/boot/compressed/mem.c | 68 ++++++++++++++++++++++++ arch/x86/include/asm/unaccepted_memory.h | 10 ++++ arch/x86/include/uapi/asm/bootparam.h | 2 +- drivers/firmware/efi/Kconfig | 14 +++++ drivers/firmware/efi/efi.c | 1 + drivers/firmware/efi/libstub/x86-stub.c | 68 ++++++++++++++++++++++++ include/linux/efi.h | 3 +- 9 files changed, 166 insertions(+), 2 deletions(-) create mode 100644 arch/x86/boot/compressed/mem.c create mode 100644 arch/x86/include/asm/unaccepted_memory.h
Documentation/x86/zero-page.rst +1 -0
@@ -20,6 +20,7 @@ Offset/Size Proto Name Meaning 060/010 ALL ist_info Intel SpeedStep (IST) BIOS support information (struct ist_info) 070/008 ALL acpi_rsdp_addr Physical address of ACPI RSDP table +078/008 ALL unaccepted_memory Bitmap of unaccepted memory (1bit == 2M) 080/010 ALL hd0_info hd0 disk parameter, OBSOLETE!! 090/010 ALL hd1_info hd1 disk parameter, OBSOLETE!! 0A0/010 ALL sys_desc_table System description table (struct sys_desc_table),
arch/x86/boot/compressed/Makefile +1 -0
@@ -102,6 +102,7 @@ endif vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o vmlinux-objs-$(CONFIG_INTEL_TDX_GUEST) += $(obj)/tdx.o $(obj)/tdcall.o +vmlinux-objs-$(CONFIG_UNACCEPTED_MEMORY) += $(obj)/bitmap.o $(obj)/mem.o vmlinux-objs-$(CONFIG_EFI_MIXED) += $(obj)/efi_thunk_$(BITS).o vmlinux-objs-$(CONFIG_EFI) += $(obj)/efi.o
arch/x86/boot/compressed/mem.c new file +68 -0
68 lines not on this page · raw message
arch/x86/include/asm/unaccepted_memory.h new file +10 -0
@@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (C) 2020 Intel Corporation */ +#ifndef _ASM_X86_UNACCEPTED_MEMORY_H +#define _ASM_X86_UNACCEPTED_MEMORY_H + +struct boot_params ; + +void process_unaccepted_memory (struct boot_params *params,u64 start,u64 num); + +#endif
arch/x86/include/uapi/asm/bootparam.h +1 -1
@@ -180,7 +180,7 @@ struct boot_params {__u64 tboot_addr;/* 0x058 */ struct ist_info ist_info;/* 0x060 */ __u64 acpi_rsdp_addr;/* 0x070 */ -__u8 _pad3[8 ];/* 0x078 */ +__u64 unaccepted_memory;/* 0x078 */ __u8 hd0_info[16 ];/* obsolete! */ /* 0x080 */ __u8 hd1_info[16 ];/* obsolete! */ /* 0x090 */ struct sys_desc_table sys_desc_table;/* obsolete! */ /* 0x0a0 */
drivers/firmware/efi/Kconfig +14 -0
@@ -305,6 +305,20 @@ config EFI_COCO_SECRET virt/coco/efi_secret module to access the secrets, which in turn allows userspace programs to access the injected secrets. +config UNACCEPTED_MEMORY + bool + depends on EFI_STUB + help + Some Virtual Machine platforms, such as Intel TDX, require + some memory to be "accepted" by the guest before it can be used. + This mechanism helps prevent malicious hosts from making changes + to guest memory. + + UEFI specification v2.9 introduced EFI_UNACCEPTED_MEMORY memory type. + + This option adds support for unaccepted memory and makes such memory + usable by the kernel. + config EFI_EMBEDDED_FIRMWARE bool select CRYPTO_LIB_SHA256
drivers/firmware/efi/efi.c +1 -0
@@ -758,6 +758,7 @@ static __initdata char memory_type_name[][13] = {"MMIO Port" ,"PAL Code" ,"Persistent" , + "Unaccepted", }; char * __init efi_md_typeattr_format(char *buf, size_t size,
drivers/firmware/efi/libstub/x86-stub.c +68 -0
92 lines not on this page · raw message
include/linux/efi.h +2 -1
@@ -108,7 +108,8 @@ typedef struct {#define EFI_MEMORY_MAPPED_IO_PORT_SPACE 12#define EFI_PAL_CODE 13#define EFI_PERSISTENT_MEMORY 14 -#define EFI_MAX_MEMORY_TYPE 15 +#define EFI_UNACCEPTED_MEMORY 15 +#define EFI_MAX_MEMORY_TYPE 16/* Attribute values: */ #define EFI_MEMORY_UC ((u64)0x0000000000000001ULL)/* uncached */
Signature · 1 line
2.35.1
Re: [PATCHv7 06/14] efi/x86: Implement support for unaccepted memory
On 6/14/22 05:02, Kirill A. Shutemov wrote:
...
Quoted 5 lines
+/*
+ * The accepted memory bitmap only works atPMD_SIZEgranularity. If a request
+ * comes in to mark memory as unaccepted which is not PMD_SIZE-aligned, simply
+ * accept the memory now since it can not be *marked* as unaccepted.
+ */
/*
* The accepted memory bitmap only works at PMD_SIZE granularity. This
* function takes unaligned start/end addresses and either:
* 1. Accepts the memory immediately and in its entirety
* 2. Accepts unaligned parts, and marks *some* aligned part unaccepted *
* The function will never reach the bitmap_set() with zero bits to set. */
arch/x86/boot/compressed/mem.c line 28
Quoted 6 lines
arch/x86/boot/compressed/mem.c new file +6
-0+void process_unaccepted_memory (struct boot_params *params,u64 start,u64 end) +{ +/* +* Accept small regions that might not be able to be represented +* in the bitmap. This is a bit imprecise and may accept some +* areas that could have been represented in the bitmap instead.
/*
* Ensure that at least one bit will be set in the bitmap by
* immediately accepting all regions under 2*PMD_SIZE. This is
* imprecise and may immediately accept some areas that could
* have been represented in the bitmap. But, results in simpler
* code below.
arch/x86/boot/compressed/mem.c line 39
Quoted 10 lines
arch/x86/boot/compressed/mem.c new file +10
-0+* Consider case like this: +* +* | 4k | 2044k | 2048k | +* ^ 0x0 ^ 2MB ^ 4MB +* +* all memory in the range is unaccepted, except for the first 4k. +* The second 2M can be represented in the bitmap, but kernel accept it +* right away. The imprecision makes the code simpler by ensuring that +* at least one bit will be set int the bitmap below. +*/
...
* Only the first 4k has been accepted. The 0MB->2MB region can
* not be represented in the bitmap. The 2MB->4MB region can be
* represented in the bitmap. But, the 0MB->4MB region is
* <2*PMD_SIZE and will be immediately accepted in its entirety. */
arch/x86/boot/compressed/mem.c line 48
Quoted 9 lines
arch/x86/boot/compressed/mem.c new file +9
-0+if (end - start <2 *PMD_SIZE ) { +__accept_memory (start, end); +return ; + } + +/* +* No matter how the start and end are aligned, at least one unaccepted +* PMD_SIZE area will remain. +*/
I'd probably add:
... to be marked in the bitmap
<snip>
Quoted 19 lines
drivers/firmware/efi/libstub/x86-stub.c +11
-0@@ -607,6 +608,17 @@ setup_e820(struct boot_params *params, struct setup_data *e820ext, u32 e820ext_s e820_type =E820_TYPE_PMEM ;break ; +case EFI_UNACCEPTED_MEMORY : +if (!IS_ENABLED (CONFIG_UNACCEPTED_MEMORY )) { +efi_warn_once ("The system has unaccepted memory," +" but kernel does not support it\n" ); +efi_warn_once ("Consider enabling CONFIG_UNACCEPTED_MEMORY\n" ); +continue ; + } + e820_type =E820_TYPE_RAM ; +process_unaccepted_memory (params, d->phys_addr, + d->phys_addr +PAGE_SIZE * d->num_pages); +break ;default :continue ; }
drivers/firmware/efi/libstub/x86-stub.c static efi_status_t alloc_e820ext(u32 nr_desc, struct setup_data **e820ext, · line 688
Quoted 6 lines
drivers/firmware/efi/libstub/x86-stub.c +3
-0@@ -671,6 +683,59 @@ static efi_status_t alloc_e820ext(u32 nr_desc, struct setup_data **e820ext,return status; } +static efi_status_t allocate_unaccepted_memory (struct boot_params *params, +__u32 nr_desc, +struct efi_boot_memmap *map)
I think this is misnamed. This function is allocating a bitmap, not "unaccepted_memory" itself. Right?
Quoted 15 lines
drivers/firmware/efi/libstub/x86-stub.c +15
-0+{ +unsigned long *mem =NULL ; +u64 size, max_addr =0 ; +efi_status_t status; +bool found = false; +int i; + +/* Check if there's any unaccepted memory and find the max address */ +for (i =0 ; i < nr_desc; i++) { +efi_memory_desc_t *d; + + d =efi_early_memdesc_ptr (*map->map, *map->desc_size, i); +if (d->type ==EFI_UNACCEPTED_MEMORY ) + found = true; +if (d->phys_addr + d->num_pages *PAGE_SIZE > max_addr)
drivers/firmware/efi/libstub/x86-stub.c static efi_status_t alloc_e820ext(u32 nr_desc, struct setup_data **e820ext, · line 713
Quoted 10 lines
drivers/firmware/efi/libstub/x86-stub.c +10
-0+ max_addr = d->phys_addr + d->num_pages *PAGE_SIZE ; + } + +if (!found) { + params->unaccepted_memory =0 ; +return EFI_SUCCESS ; + } + +/* +* If unaccepted memory is present allocate a bitmap to track what
^ comma
Quoted 52 lines
drivers/firmware/efi/libstub/x86-stub.c +28
-0+* memory has to be accepted before access. +* +* One bit in the bitmap represents 2MiB in the address space: +* A 4k bitmap can track 64GiB of physical address space. +* +* In the worst case scenario -- a huge hole in the middle of the +* address space -- It needs 256MiB to handle 4PiB of the address +* space. +* +* TODO: handle situation if params->unaccepted_memory is already set. +* It's required to deal with kexec. +* +* The bitmap will be populated in setup_e820() according to the memory +* map after efi_exit_boot_services(). +*/ + size =DIV_ROUND_UP (max_addr,PMD_SIZE *BITS_PER_BYTE ); + status =efi_allocate_pages (size, (unsigned long *)&mem,ULONG_MAX ); +if (status ==EFI_SUCCESS ) { +memset (mem,0 , size); + params->unaccepted_memory = (unsigned long )mem; + } + +return status; +} + static efi_status_t allocate_e820(struct boot_params *params, struct efi_boot_memmap *map, struct setup_data **e820ext,@@ -691,6 +756,9 @@ static efi_status_t allocate_e820(struct boot_params *params, status =alloc_e820ext (nr_e820ext, e820ext, e820ext_size); } +if (IS_ENABLED (CONFIG_UNACCEPTED_MEMORY ) && status ==EFI_SUCCESS ) + status =allocate_unaccepted_memory (params, nr_desc, map); +efi_bs_call (free_pool, *map->map);return status; }include/linux/efi.h +2
-1@@ -108,7 +108,8 @@ typedef struct {#define EFI_MEMORY_MAPPED_IO_PORT_SPACE 12#define EFI_PAL_CODE 13#define EFI_PERSISTENT_MEMORY 14 -#define EFI_MAX_MEMORY_TYPE 15 +#define EFI_UNACCEPTED_MEMORY 15 +#define EFI_MAX_MEMORY_TYPE 16/* Attribute values: */ #define EFI_MEMORY_UC ((u64)0x0000000000000001ULL)/* uncached */
Borislav Petkov
maintainerRe: [PATCHv7 06/14] efi/x86: Implement support for unaccepted memory
In reply to Kirill A. ShutemovQuoted 12 lines · Kirill A. Shutemov
drivers/firmware/efi/Kconfig +4
-0@@ -305,6 +305,20 @@ config EFI_COCO_SECRET virt/coco/efi_secret module to access the secrets, which in turn allows userspace programs to access the injected secrets. +config UNACCEPTED_MEMORY + bool + depends on EFI_STUB + help
drivers/firmware/efi/Kconfig config EFI_COCO_SECRET · line 321
Quoted 10 lines
drivers/firmware/efi/Kconfig +10
-0+ Some Virtual Machine platforms, such as Intel TDX, require + some memory to be "accepted" by the guest before it can be used. + This mechanism helps prevent malicious hosts from making changes + to guest memory. + + UEFI specification v2.9 introduced EFI_UNACCEPTED_MEMORY memory type. + + This option adds support for unaccepted memory and makes such memory + usable by the kernel. +
This looks weird:
$ grep EFI_STUB .config CONFIG_EFI_STUB=y $ grep UNACCEPTED_MEMORY .config $
So the bool needs to have a text string after it so that it is selectable or how is UNACCEPTED_MEMORY supposed to be enabled otherwise?
If I add the string and select UNACCEPTED_MEMORY, it won't build:
mm/page_alloc.c: In function ‘accept_page’:mm/page_alloc.c:1013:9: error: implicit declaration of function ‘accept_memory’ [-Werror=implicit-function-declaration]
1013 | accept_memory(start, start + (PAGE_SIZE << order));
| ^~~~~~~~~~~~~
mm/page_alloc.c: In function ‘page_contains_unaccepted’:
mm/page_alloc.c:1029:16: error: implicit declaration of function ‘range_contains_unaccepted_memory’; did you mean ‘page_contains_unaccepted’? [-Werror=implicit-function-declaration]
1029 | return range_contains_unaccepted_memory(start, end);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| page_contains_unaccepted
mm/memblock.c: In function ‘memblock_alloc_range_nid’:
mm/memblock.c:1415:9: error: implicit declaration of function ‘accept_memory’ [-Werror=implicit-function-declaration]
1415 | accept_memory(found, found + size);
| ^~~~~~~~~~~~~cc1: some warnings being treated as errors
make[1]: *** [scripts/Makefile.build:249: mm/memblock.o] Error 1 make[1]: *** Waiting for unfinished jobs....
cc1: some warnings being treated as errors
make[1]: *** [scripts/Makefile.build:249: mm/page_alloc.o] Error 1
make: *** [Makefile:1843: mm] Error 2
make: *** Waiting for unfinished jobs....
so this is weird.
Quoted 13 lines
drivers/firmware/efi/libstub/x86-stub.c +1
-0@@ -15,6 +15,7 @@#include <asm/setup.h> #include <asm/desc.h> #include <asm/boot.h> +#include <asm/unaccepted_memory.h> #include "efistub.h"
drivers/firmware/efi/libstub/x86-stub.c setup_e820(struct boot_params *params, struct setup_data *e820ext, u32 e820ext_s · line 617
Quoted 10 lines
drivers/firmware/efi/libstub/x86-stub.c +7
-0@@ -607,6 +608,17 @@ setup_e820(struct boot_params *params, struct setup_data *e820ext, u32 e820ext_s e820_type =E820_TYPE_PMEM ;break ; +case EFI_UNACCEPTED_MEMORY : +if (!IS_ENABLED (CONFIG_UNACCEPTED_MEMORY )) { +efi_warn_once ("The system has unaccepted memory," +" but kernel does not support it\n" ); +efi_warn_once ("Consider enabling CONFIG_UNACCEPTED_MEMORY\n" ); +continue ; + }
So that it can be grepped for:
drivers/firmware/efi/libstub/x86-stub.c +2 -3
@@ -610,9 +610,8 @@ setup_e820(struct boot_params *params, struct setup_data *e820ext, u32 e820ext_scase EFI_UNACCEPTED_MEMORY :if (!IS_ENABLED (CONFIG_UNACCEPTED_MEMORY )) { -efi_warn_once ("The system has unaccepted memory," -" but kernel does not support it\n" ); -efi_warn_once ("Consider enabling CONFIG_UNACCEPTED_MEMORY\n" ); +efi_warn_once ( +"The system has unaccepted memory, but kernel does not support it.\nConsider enabling CONFIG_UNACCEPTED_MEMORY\n" );continue ; } e820_type =E820_TYPE_RAM ;
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v7 7/14 x86/boot/compressed: Handle unaccepted memory
In reply to Kirill A. ShutemovThe firmware will pre-accept the memory used to run the stub. But, the stub is responsible for accepting the memory into which it decompresses the main kernel. Accept memory just before decompression starts.
The stub is also responsible for choosing a physical address in which to place the decompressed kernel image. The KASLR mechanism will randomize this physical address. Since the unaccepted memory region is relatively small, KASLR would be quite ineffective if it only used the pre-accepted area (EFI_CONVENTIONAL_MEMORY). Ensure that KASLR randomizes among the entire physical address space by also including EFI_UNACCEPTED_MEMORY.
--- arch/x86/boot/compressed/Makefile | 2 +- arch/x86/boot/compressed/efi.h | 1 + arch/x86/boot/compressed/kaslr.c | 35 ++++++++++++++++-------- arch/x86/boot/compressed/mem.c | 18 ++++++++++++ arch/x86/boot/compressed/misc.c | 6 ++++ arch/x86/boot/compressed/misc.h | 6 ++++ arch/x86/include/asm/unaccepted_memory.h | 2 ++ 7 files changed, 57 insertions(+), 13 deletions(-)
arch/x86/boot/compressed/Makefile +1 -1
@@ -102,7 +102,7 @@ endif vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o vmlinux-objs-$(CONFIG_INTEL_TDX_GUEST) += $(obj)/tdx.o $(obj)/tdcall.o -vmlinux-objs-$(CONFIG_UNACCEPTED_MEMORY) += $(obj)/bitmap.o $(obj)/mem.o +vmlinux-objs-$(CONFIG_UNACCEPTED_MEMORY) += $(obj)/bitmap.o $(obj)/find.o $(obj)/mem.o vmlinux-objs-$(CONFIG_EFI_MIXED) += $(obj)/efi_thunk_$(BITS).o vmlinux-objs-$(CONFIG_EFI) += $(obj)/efi.o
arch/x86/boot/compressed/efi.h +1 -0
@@ -32,6 +32,7 @@ typedef struct { } efi_table_hdr_t;#define EFI_CONVENTIONAL_MEMORY 7 +#define EFI_UNACCEPTED_MEMORY 15#define EFI_MEMORY_MORE_RELIABLE \ ((u64)0x0000000000010000ULL)/* higher reliability */
arch/x86/boot/compressed/kaslr.c +23 -12
@@ -671,6 +671,28 @@ static bool process_mem_region(struct mem_vector *region, }#ifdef CONFIG_EFI + +/* +* Only EFI_CONVENTIONAL_MEMORY and EFI_UNACCEPTED_MEMORY (if supported) are +* guaranteed to be free. +* +* It is more conservative in picking free memory than the EFI spec allows: +* +* According to the spec, EFI_BOOT_SERVICES_{CODE|DATA} are also free memory +* and thus available to place the kernel image into, but in practice there's +* firmware where using that memory leads to crashes. +*/ +static inline bool memory_type_is_free (efi_memory_desc_t *md) +{ +if (md->type ==EFI_CONVENTIONAL_MEMORY ) +return true; + +if (md->type ==EFI_UNACCEPTED_MEMORY ) +return IS_ENABLED (CONFIG_UNACCEPTED_MEMORY ); + +return false; +} +/* * Returns true if we processed the EFI memmap, which we prefer over the E820 * table if it is available.
@@ -715,18 +737,7 @@ process_efi_entries(unsigned long minimum, unsigned long image_size)for (i =0 ; i < nr_desc; i++) { md =efi_early_memdesc_ptr (pmap, e->efi_memdesc_size, i); -/* -* Here we are more conservative in picking free memory than -* the EFI spec allows: -* -* According to the spec, EFI_BOOT_SERVICES_{CODE|DATA} are also -* free memory and thus available to place the kernel image into, -* but in practice there's firmware where using that memory leads -* to crashes. -* -* Only EFI_CONVENTIONAL_MEMORY is guaranteed to be free. -*/ -if (md->type !=EFI_CONVENTIONAL_MEMORY ) +if (!memory_type_is_free (md))continue ; if (efi_soft_reserve_enabled() &&
arch/x86/boot/compressed/mem.c +18 -0
@@ -3,12 +3,15 @@#include "../cpuflags.h" #include "bitmap.h" #include "error.h" +#include "find.h" #include "math.h" #define PMD_SHIFT 21#define PMD_SIZE (_AC(1, UL) << PMD_SHIFT)#define PMD_MASK (~(PMD_SIZE - 1)) +extern struct boot_params *boot_params; +static inline void __accept_memory (phys_addr_t start,phys_addr_t end) {/* Platform-specific memory-acceptance call goes here */
@@ -66,3 +69,18 @@ void process_unaccepted_memory(struct boot_params *params, u64 start, u64 end)bitmap_set ((unsigned long *)params->unaccepted_memory, start /PMD_SIZE , (end - start) /PMD_SIZE ); } + +void accept_memory (phys_addr_t start,phys_addr_t end) +{ +unsigned long range_start, range_end; +unsigned long *bitmap, bitmap_size; + + bitmap = (unsigned long *)boot_params->unaccepted_memory; + range_start = start /PMD_SIZE ; + bitmap_size =DIV_ROUND_UP (end,PMD_SIZE ); + +for_each_set_bitrange_from (range_start, range_end, bitmap, bitmap_size) { +__accept_memory (range_start *PMD_SIZE , range_end *PMD_SIZE ); +bitmap_clear (bitmap, range_start, range_end - range_start); + } +}
arch/x86/boot/compressed/misc.c +6 -0
@@ -454,6 +454,12 @@ asmlinkage __visible void *extract_kernel(void *rmode, memptr heap,#endif debug_putstr ("\nDecompressing Linux... " ); + +if (boot_params->unaccepted_memory) { +debug_putstr ("Accepting memory... " ); +accept_memory (__pa (output),__pa (output) + needed_size); + } +__decompress (input_data, input_len,NULL ,NULL , output, output_len,NULL , error);parse_elf (output);
arch/x86/boot/compressed/misc.h +6 -0
@@ -235,4 +235,10 @@ static inline unsigned long efi_find_vendor_table(struct boot_params *bp, }#endif /* CONFIG_EFI */ +#ifdef CONFIG_UNACCEPTED_MEMORY +void accept_memory (phys_addr_t start,phys_addr_t end); +#else +static inline void accept_memory (phys_addr_t start,phys_addr_t end) {} +#endif +#endif /* BOOT_COMPRESSED_MISC_H */
arch/x86/include/asm/unaccepted_memory.h +2 -0
@@ -7,4 +7,6 @@ struct boot_params;void process_unaccepted_memory (struct boot_params *params,u64 start,u64 num); +void accept_memory (phys_addr_t start,phys_addr_t end); +#endif
Signature · 1 line
2.35.1
PATCH v7 8/14 x86/mm: Reserve unaccepted memory bitmap
In reply to Kirill A. ShutemovA given page of memory can only be accepted once. The kernel has to accept memory both in the early decompression stage and during normal runtime.
A bitmap is used to communicate the acceptance state of each page between the decompression stage and normal runtime.
boot_params is used to communicate location of the bitmap throughout the boot. The bitmap is allocated and initially populated in EFI stub. Decompression stage accepts pages required for kernel/initrd and marks these pages accordingly in the bitmap. The main kernel picks up the bitmap from the same boot_params and uses it to determine what has to be accepted on allocation.
In the runtime kernel, reserve the bitmap's memory to ensure nothing overwrites it.
The size of bitmap is determined with e820__end_of_ram_pfn() which relies on setup_e820() marking unaccepted memory as E820_TYPE_RAM.
--- arch/x86/kernel/e820.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/x86/kernel/e820.c +10 -0
@@ -1316,6 +1316,16 @@ void __init e820__memblock_setup(void)int i;u64 end; +/* Mark unaccepted memory bitmap reserved */ +if (boot_params.unaccepted_memory) { +unsigned long size; + +/* One bit per 2MB */ + size =DIV_ROUND_UP (e820__end_of_ram_pfn () *PAGE_SIZE , +PMD_SIZE *BITS_PER_BYTE ); +memblock_reserve (boot_params.unaccepted_memory, size); + } +/* * The bootstrap memblock region count maximum is 128 entries * (INIT_MEMBLOCK_REGIONS), but EFI might pass us more E820 entries
Signature · 1 line
2.35.1
Borislav Petkov
maintainerRe: [PATCHv7 08/14] x86/mm: Reserve unaccepted memory bitmap
Quoted 8 lines · Kirill A. Shutemov
arch/x86/kernel/e820.c +0
-0@@ -1316,6 +1316,16 @@ void __init e820__memblock_setup(void)int i;u64 end;
arch/x86/kernel/e820.c void __init e820__memblock_setup(void) · line 1328
Quoted 10 lines
arch/x86/kernel/e820.c +10
-0+/* Mark unaccepted memory bitmap reserved */ +if (boot_params.unaccepted_memory) { +unsigned long size; + +/* One bit per 2MB */ + size =DIV_ROUND_UP (e820__end_of_ram_pfn () *PAGE_SIZE , +PMD_SIZE *BITS_PER_BYTE ); +memblock_reserve (boot_params.unaccepted_memory, size); + } +
Hmm, I don't like how this is dropped right in the middle of a unrelated function.
You're adding arch/x86/mm/unaccepted_memory.c later. Why don't you put that chunk in a function there which is called by early_reserve_memory() which does exactly what you want - reserve memory early, before memblock allocations?
Hmm.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCHv7 08/14] x86/mm: Reserve unaccepted memory bitmap
Quoted 23 lines · Borislav Petkov
Quoted 8 lines · Kirill A. Shutemov
arch/x86/kernel/e820.c +0
-0@@ -1316,6 +1316,16 @@ void __init e820__memblock_setup(void)int i;u64 end;arch/x86/kernel/e820.c void __init e820__memblock_setup(void) · line 1328
Quoted 10 lines
arch/x86/kernel/e820.c +10
-0+/* Mark unaccepted memory bitmap reserved */ +if (boot_params.unaccepted_memory) { +unsigned long size; + +/* One bit per 2MB */ + size =DIV_ROUND_UP (e820__end_of_ram_pfn () *PAGE_SIZE , +PMD_SIZE *BITS_PER_BYTE ); +memblock_reserve (boot_params.unaccepted_memory, size); + } +Hmm, I don't like how this is dropped right in the middle of a unrelated function.
Quoted 4 lines
You're adding
arch/x86/mm/unaccepted_memory.clater. Why don't you put that chunk in a function there which is called byearly_reserve_memory()which does exactly what you want - reserve memory early, before memblock allocations?
early_reserve_memory() specifically called before e820__memory_setup() (see comment in setup_arch()), so we don't have e820_table finalized and we need it to get correct RAM size from e820__end_of_ram_pfn().
I guess we can hide the chunk in a function in unaccepted_memory.c and call it from here, but it would require #ifdeffery in a header file as the .c is only compiled for CONFIG_UNACCEPTED_MEMORY=y.
Looks like an overkill to me, no?
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Mike Rapoport
maintainerRe: [PATCHv7 08/14] x86/mm: Reserve unaccepted memory bitmap
Quoted 33 lines · Kirill A. Shutemov
Quoted 23 lines · Borislav Petkov
Quoted 8 lines · Kirill A. Shutemov
arch/x86/kernel/e820.c +0
-0@@ -1316,6 +1316,16 @@ void __init e820__memblock_setup(void)int i;u64 end;arch/x86/kernel/e820.c void __init e820__memblock_setup(void) · line 1328
Quoted 10 lines
arch/x86/kernel/e820.c +10
-0+/* Mark unaccepted memory bitmap reserved */ +if (boot_params.unaccepted_memory) { +unsigned long size; + +/* One bit per 2MB */ + size =DIV_ROUND_UP (e820__end_of_ram_pfn () *PAGE_SIZE , +PMD_SIZE *BITS_PER_BYTE ); +memblock_reserve (boot_params.unaccepted_memory, size); + } +Hmm, I don't like how this is dropped right in the middle of a unrelated function.
Quoted 4 lines
You're adding
arch/x86/mm/unaccepted_memory.clater. Why don't you put that chunk in a function there which is called byearly_reserve_memory()which does exactly what you want - reserve memory early, before memblock allocations?
early_reserve_memory()specifically called beforee820__memory_setup()(see comment insetup_arch()), so we don't havee820_tablefinalized and we need it to get correct RAM size frome820__end_of_ram_pfn().
Quoted 5 lines
I guess we can hide the chunk in a function in
unaccepted_memory.cand call it from here, but it would require #ifdeffery in a header file as the .c is only compiled forCONFIG_UNACCEPTED_MEMORY=y.Looks like an overkill to me, no?
Agree. Can we just extend the comment to explain why we reserve the bitmap at e820__memblock_setup() rather than in early_reserve_memory(), pretty much with the explanation above?
Quoted 2 lines
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCHv7 08/14] x86/mm: Reserve unaccepted memory bitmap
Quoted 38 lines · Mike Rapoport
Quoted 37 lines · Kirill A. Shutemov
Quoted 23 lines · Borislav Petkov
Quoted 8 lines · Kirill A. Shutemov
arch/x86/kernel/e820.c +0
-0@@ -1316,6 +1316,16 @@ void __init e820__memblock_setup(void)int i;u64 end;arch/x86/kernel/e820.c void __init e820__memblock_setup(void) · line 1328
Quoted 10 lines
arch/x86/kernel/e820.c +10
-0+/* Mark unaccepted memory bitmap reserved */ +if (boot_params.unaccepted_memory) { +unsigned long size; + +/* One bit per 2MB */ + size =DIV_ROUND_UP (e820__end_of_ram_pfn () *PAGE_SIZE , +PMD_SIZE *BITS_PER_BYTE ); +memblock_reserve (boot_params.unaccepted_memory, size); + } +Hmm, I don't like how this is dropped right in the middle of a unrelated function.
Quoted 4 lines
You're adding
arch/x86/mm/unaccepted_memory.clater. Why don't you put that chunk in a function there which is called byearly_reserve_memory()which does exactly what you want - reserve memory early, before memblock allocations?
early_reserve_memory()specifically called beforee820__memory_setup()(see comment insetup_arch()), so we don't havee820_tablefinalized and we need it to get correct RAM size frome820__end_of_ram_pfn().I guess we can hide the chunk in a function in
unaccepted_memory.cand call it from here, but it would require #ifdeffery in a header file as the .c is only compiled forCONFIG_UNACCEPTED_MEMORY=y.
Quoted 5 lines
Quoted 1 line
Looks like an overkill to me, no?
Agree. Can we just extend the comment to explain why we reserve the bitmap at
e820__memblock_setup()rather than inearly_reserve_memory(), pretty much with the explanation above?
Okay, I will do this:
arch/x86/kernel/e820.c +8 -1
@@ -1316,7 +1316,14 @@ void __init e820__memblock_setup(void)int i;u64 end; -/* Mark unaccepted memory bitmap reserved */ +/* +* Mark unaccepted memory bitmap reserved. +* +* This kind of reservation usually done from early_reserve_memory(), +* but early_reserve_memory() called before e820__memory_setup(), so +* e820_table is not finalized and e820__end_of_ram_pfn() cannot be +* used to get correct RAM size. +*/ if (boot_params.unaccepted_memory) {unsigned long size;
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
PATCH v7 9/14 x86/mm: Provide helpers for unaccepted memory
In reply to Kirill A. ShutemovCore-mm requires few helpers to support unaccepted memory:
- accept_memory() checks the range of addresses against the bitmap and accept memory if needed.
- range_contains_unaccepted_memory() checks if anything within the range requires acceptance.
--- arch/x86/include/asm/page.h | 3 ++ arch/x86/include/asm/unaccepted_memory.h | 4 ++ arch/x86/mm/Makefile | 2 + arch/x86/mm/unaccepted_memory.c | 61 ++++++++++++++++++++++++ 4 files changed, 70 insertions(+) create mode 100644 arch/x86/mm/unaccepted_memory.c
arch/x86/include/asm/page.h +3 -0
@@ -19,6 +19,9 @@struct page ;#include <linux/range.h> + +#include <asm/unaccepted_memory.h> +extern struct range pfn_mapped[];extern int nr_pfn_mapped;
arch/x86/include/asm/unaccepted_memory.h +4 -0
@@ -7,6 +7,10 @@ struct boot_params;void process_unaccepted_memory (struct boot_params *params,u64 start,u64 num); +#ifdef CONFIG_UNACCEPTED_MEMORY +void accept_memory (phys_addr_t start,phys_addr_t end); +bool range_contains_unaccepted_memory (phys_addr_t start,phys_addr_t end);#endif +#endif
arch/x86/mm/Makefile +2 -0
@@ -59,3 +59,5 @@ obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_amd.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_identity.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_boot.o + +obj-$(CONFIG_UNACCEPTED_MEMORY) += unaccepted_memory.o
arch/x86/mm/unaccepted_memory.c new file +61 -0
61 lines not on this page · raw message
Signature · 1 line
2.35.1
PATCH v7 10/14 x86/mm: Avoid load_unaligned_zeropad() stepping into unaccepted memory
In reply to Kirill A. Shutemovload_unaligned_zeropad() can lead to unwanted loads across page boundaries. The unwanted loads are typically harmless. But, they might be made to totally unrelated or even unmapped memory. load_unaligned_zeropad() relies on exception fixup (#PF, #GP and now #VE) to recover from these unwanted loads.
But, this approach does not work for unaccepted memory. For TDX, a load from unaccepted memory will not lead to a recoverable exception within the guest. The guest will exit to the VMM where the only recourse is to terminate the guest.
There are three parts to fix this issue and comprehensively avoid access to unaccepted memory. Together these ensure that an extra “guard” page is accepted in addition to the memory that needs to be used.
1. Implicitly extend the range_contains_unaccepted_memory(start, end) checks up to end+2M if ‘end’ is aligned on a 2M boundary.
2. Implicitly extend accept_memory(start, end) to end+2M if ‘end’ is aligned on a 2M boundary.
3. Set PageUnaccepted() on both memory that itself needs to be accepted *and* memory where the next page needs to be accepted. Essentially, make PageUnaccepted(page) a marker for whether work needs to be done to make ‘page’ usable. That work might include accepting pages in addition to ‘page’ itself.
Side note: This leads to something strange. Pages which were accepted
at boot, marked by the firmware as accepted and will never _need_ to be accepted might have PageUnaccepted() set on them. PageUnaccepted(page) is a cue to ensure that the next page is accepted before ‘page’ can be used.
This is an actual, real-world problem which was discovered during TDX testing.
--- arch/x86/mm/unaccepted_memory.c | 36 +++++++++++++++++++++++++ drivers/firmware/efi/libstub/x86-stub.c | 7 +++++ 2 files changed, 43 insertions(+)
arch/x86/mm/unaccepted_memory.c +36 -0
@@ -23,6 +23,38 @@ void accept_memory(phys_addr_t start, phys_addr_t end) bitmap =__va (boot_params.unaccepted_memory); range_start = start /PMD_SIZE ; +/* +* load_unaligned_zeropad() can lead to unwanted loads across page +* boundaries. The unwanted loads are typically harmless. But, they +* might be made to totally unrelated or even unmapped memory. +* load_unaligned_zeropad() relies on exception fixup (#PF, #GP and now +* #VE) to recover from these unwanted loads. +* +* But, this approach does not work for unaccepted memory. For TDX, a +* load from unaccepted memory will not lead to a recoverable exception +* within the guest. The guest will exit to the VMM where the only +* recourse is to terminate the guest. +* +* There are three parts to fix this issue and comprehensively avoid +* access to unaccepted memory. Together these ensure that an extra +* “guard” page is accepted in addition to the memory that needs to be +* used: +* +* 1. Implicitly extend the range_contains_unaccepted_memory(start, end) +* checks up to end+2M if ‘end’ is aligned on a 2M boundary. +* +* 2. Implicitly extend accept_memory(start, end) to end+2M if ‘end’ is +* aligned on a 2M boundary. +* +* 3. Set PageUnaccepted() on both memory that itself needs to be +* accepted *and* memory where the next page needs to be accepted. +* Essentially, make PageUnaccepted(page) a marker for whether work +* needs to be done to make ‘page’ usable. That work might include +* accepting pages in addition to ‘page’ itself. +*/ +if (!(end %PMD_SIZE )) + end +=PMD_SIZE ; +spin_lock_irqsave (&unaccepted_memory_lock, flags);for_each_set_bitrange_from (range_start, range_end, bitmap,DIV_ROUND_UP (end,PMD_SIZE )) {
@@ -46,6 +78,10 @@ bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) bitmap =__va (boot_params.unaccepted_memory); +/* See comment on load_unaligned_zeropad() in accept_memory() */ +if (!(end %PMD_SIZE )) + end +=PMD_SIZE ; +spin_lock_irqsave (&unaccepted_memory_lock, flags);while (start < end) {if (test_bit (start /PMD_SIZE , bitmap)) {
drivers/firmware/efi/libstub/x86-stub.c +7 -0
@@ -709,6 +709,13 @@ static efi_status_t allocate_unaccepted_memory(struct boot_params *params,return EFI_SUCCESS ; } +/* +* range_contains_unaccepted_memory() may need to check one 2M chunk +* beyond the end of RAM to deal with load_unaligned_zeropad(). Make +* sure that the bitmap is large enough handle it. +*/ + max_addr +=PMD_SIZE ; +/* * If unaccepted memory is present allocate a bitmap to track what * memory has to be accepted before access.
Signature · 1 line
2.35.1
Re: [PATCHv7 10/14] x86/mm: Avoid load_unaligned_zeropad() stepping into unaccepted memory
Quoted 15 lines · Kirill A. Shutemov
load_unaligned_zeropad()can lead to unwanted loads across page boundaries. The unwanted loads are typically harmless. But, they might be made to totally unrelated or even unmapped memory.load_unaligned_zeropad()relies on exception fixup (#PF, #GP and now #VE) to recover from these unwanted loads.But, this approach does not work for unaccepted memory. For TDX, a load from unaccepted memory will not lead to a recoverable exception within the guest. The guest will exit to the VMM where the only recourse is to terminate the guest.
There are three parts to fix this issue and comprehensively avoid access to unaccepted memory. Together these ensure that an extra “guard” page is accepted in addition to the memory that needs to be used.
Quoted 9 lines
1. Implicitly extend the
range_contains_unaccepted_memory(start, end)checks up to end+2M if ‘end’ is aligned on a 2M boundary.
2. Implicitly extendaccept_memory(start, end)to end+2M if ‘end’ is aligned on a 2M boundary.
3. SetPageUnaccepted()on both memory that itself needs to be accepted *and* memory where the next page needs to be accepted. Essentially, make PageUnaccepted(page) a marker for whether work needs to be done to make ‘page’ usable. That work might include accepting pages in addition to ‘page’ itself.
...
That all looks pretty good.
Quoted 27 lines
arch/x86/mm/unaccepted_memory.c +19
-0@@ -23,6 +23,38 @@ void accept_memory(phys_addr_t start, phys_addr_t end) bitmap =__va (boot_params.unaccepted_memory); range_start = start /PMD_SIZE ; +/* +* load_unaligned_zeropad() can lead to unwanted loads across page +* boundaries. The unwanted loads are typically harmless. But, they +* might be made to totally unrelated or even unmapped memory. +* load_unaligned_zeropad() relies on exception fixup (#PF, #GP and now +* #VE) to recover from these unwanted loads. +* +* But, this approach does not work for unaccepted memory. For TDX, a +* load from unaccepted memory will not lead to a recoverable exception +* within the guest. The guest will exit to the VMM where the only +* recourse is to terminate the guest. +* +* There are three parts to fix this issue and comprehensively avoid +* access to unaccepted memory. Together these ensure that an extra +* “guard” page is accepted in addition to the memory that needs to be +* used: +* +* 1. Implicitly extend the range_contains_unaccepted_memory(start, end) +* checks up to end+2M if ‘end’ is aligned on a 2M boundary.
arch/x86/mm/unaccepted_memory.c void accept_memory(phys_addr_t start, phys_addr_t end) · line 54
Quoted 10 lines
arch/x86/mm/unaccepted_memory.c +10
-0+* +* 2. Implicitly extend accept_memory(start, end) to end+2M if ‘end’ is +* aligned on a 2M boundary. +* +* 3. Set PageUnaccepted() on both memory that itself needs to be +* accepted *and* memory where the next page needs to be accepted. +* Essentially, make PageUnaccepted(page) a marker for whether work +* needs to be done to make ‘page’ usable. That work might include +* accepting pages in addition to ‘page’ itself. +*/
One nit with this: I'd much rather add one sentence to these to help tie the code implementing it with this comment. Maybe something like:
* 2. Implicitly extend accept_memory(start, end) to end+2M if ‘end’ is
* aligned on a 2M boundary. (immediately following this comment)
arch/x86/mm/unaccepted_memory.c bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) · line 83
Quoted 13 lines
arch/x86/mm/unaccepted_memory.c +6
-0+if (!(end %PMD_SIZE )) + end +=PMD_SIZE ; +spin_lock_irqsave (&unaccepted_memory_lock, flags);for_each_set_bitrange_from (range_start, range_end, bitmap,DIV_ROUND_UP (end,PMD_SIZE )) {@@ -46,6 +78,10 @@ bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) bitmap =__va (boot_params.unaccepted_memory); +/* See comment on load_unaligned_zeropad() in accept_memory() */ +if (!(end %PMD_SIZE )) + end +=PMD_SIZE ;
It's a wee bit hard to follow this back to the comment that it references, even with them sitting next to each other in this diff. How about adding:
/*
* Also consider the unaccepted state of the *next* page. See
* fix #1 in the comment on load_unaligned_zeropad() in
* accept_memory().
*/
Quoted 8 lines
arch/x86/mm/unaccepted_memory.c +0
-0spin_lock_irqsave (&unaccepted_memory_lock, flags);while (start < end) {if (test_bit (start /PMD_SIZE , bitmap)) {drivers/firmware/efi/libstub/x86-stub.c +0
-0
drivers/firmware/efi/libstub/x86-stub.c static efi_status_t allocate_unaccepted_memory(struct boot_params *params, · line 717
Quoted 9 lines
drivers/firmware/efi/libstub/x86-stub.c +6
-0@@ -709,6 +709,13 @@ static efi_status_t allocate_unaccepted_memory(struct boot_params *params,return EFI_SUCCESS ; } +/* +* range_contains_unaccepted_memory() may need to check one 2M chunk +* beyond the end of RAM to deal with load_unaligned_zeropad(). Make +* sure that the bitmap is large enough handle it. +*/ + max_addr +=PMD_SIZE ;
I guess the alternative to this would have been to record 'max_addr', then special case 'max_addr'+2M in the bitmap checks. I agree this is probably nicer.
Also, the changelog needs to at least *mention* this little tidbit. It was a bit of a surprise when I got here.
With those fixed:
Borislav Petkov
maintainerRe: [PATCHv7 10/14] x86/mm: Avoid load_unaligned_zeropad() stepping into unaccepted memory
In reply to Kirill A. ShutemovQuoted 4 lines · Kirill A. Shutemov
But, this approach does not work for unaccepted memory. For TDX, a load from unaccepted memory will not lead to a recoverable exception within the guest. The guest will exit to the VMM where the only recourse is to terminate the guest.
FTR, this random-memory-access-to-unaccepted-memory-is-deadly thing is really silly. We should be able to handle such cases - because they do happen often - in a more resilient way. Just look at the complex dance this patch needs to do just to avoid this.
IOW, this part of the coco technology needs improvement.
Just sayin...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCHv7 10/14] x86/mm: Avoid load_unaligned_zeropad() stepping into unaccepted memory
Quoted 11 lines · Borislav Petkov
Quoted 4 lines · Kirill A. Shutemov
But, this approach does not work for unaccepted memory. For TDX, a load from unaccepted memory will not lead to a recoverable exception within the guest. The guest will exit to the VMM where the only recourse is to terminate the guest.
FTR, this random-memory-access-to-unaccepted-memory-is-deadly thing is really silly. We should be able to handle such cases - because they do happen often - in a more resilient way. Just look at the complex dance this patch needs to do just to avoid this.
IOW, this part of the coco technology needs improvement.
This particular wound is self-inflicted. The hardware can *today* generate a #VE for these accesses. But, to make writing the #VE code more straightforward, we asked that the hardware not even bother delivering the exception. At the time, nobody could come up with a case why there would ever be a legitimate, non-buggy access to unaccepted memory.
We learned about load_unaligned_zeropad() the hard way. I never ran into it and never knew it was there. Dangit.
We _could_ go back to the way it was originally. We could add load_unaligned_zeropad() support to the #VE handler, and there's little risk of load_unaligned_zeropad() itself being used in the interrupts-disabled window early in the #VE handler. That would get rid of all the nasty adjacent page handling in the unaccepted memory code.
But, that would mean that we can land in the #VE handler from more contexts. Any normal, non-buggy use of load_unaligned_zeropad() can end up there, obviously. We would, for instance, need to be more careful about #VE recursion. We'd also have to make sure that _bugs_ that land in the #VE handler can still be handled in a sane way.
To sum it all up, I'm not happy with the complexity of the page acceptance code either but I'm not sure that it's bad tradeoff compared to greater #VE complexity or fragility.
Does anyone think we should go back and really reconsider this?
Re: [PATCHv7 10/14] x86/mm: Avoid load_unaligned_zeropad() stepping into unaccepted memory
Quoted 5 lines · Dave Hansen
To sum it all up, I'm not happy with the complexity of the page acceptance code either but I'm not sure that it's bad tradeoff compared to greater #VE complexity or fragility.
Does anyone think we should go back and really reconsider this?
One other thing I remembered as I re-read my write up on this.
In the "new" mode, guests never get #VE's for unaccepted memory. They just exit to the host and can never be reentered. They must be killed.
In the "old" mode, I _believe_ that the guest always gets a #VE for non-EPT-present memory. The #VE is basically the same no matter if the page is unaccepted or if the host goes out and makes a previously-accepted page non-present.
One really nasty implication of this "old" mode is that the host can remove *accepted* pages that are used in the syscall gap. That means that the #VE handler would need to be of the paranoid variety which opens up all kinds of other fun.
* "Old" - #VE's can happen in the syscall gap
* "New" - #VE's happen at better-defined times. Unexpected ones are fatal.
There's a third option which I proposed but doesn't yet exist. The TDX module _could_ separate the behavior of unaccepted memory #VE's and host-induced #VEs. This way, we could use load_unaligned_zeropad() with impunity and handle it in the #VE handler. At the same time, the host would not be allowed to remove accepted memory and cause problems in the syscall gap. Kinda the best of both worlds.
But, I'm not sure how valuable that would be now that we have the (admittedly squirrelly) code to avoid load_unaligned_zeropad() #VE's.
Borislav Petkov
maintainerRe: [PATCHv7 10/14] x86/mm: Avoid load_unaligned_zeropad() stepping into unaccepted memory
Quoted 4 lines · Dave Hansen
One other thing I remembered as I re-read my write up on this.
In the "new" mode, guests never get #VE's for unaccepted memory. They just exit to the host and can never be reentered. They must be killed.
Yeah, this is the part which I think is really silly.
OSes, in their execution lifetime, can - erroneously or not - but it happens pretty often in real life, touch some unrelated memory. And this has never been a big deal - #PF, that's it.
But now they don't even get a chance to correct their mistake - VMEXIT, die.
load_unaligned_zeropad() is just one case.
Imagine the user loads some buggy driver in the guest and that driver starts doing stray memory accesses through a wild pointer into the fields. Guest dies immediately.
Dunno bit it all feels a bit too harsh and unfriendly to me.
Sure, if that user is really unlucky, those stray accesses can kill his OS on baremetal too. So maybe you could argue here that such stray accesses are actually a good thing. :)
All I know is, there should be a more resilient way to handle those.
Quoted 9 lines
In the "old" mode, I _believe_ that the guest always gets a #VE for non-EPT-present memory. The #VE is basically the same no matter if the page is unaccepted or if the host goes out and makes a previously-accepted page non-present.
One really nasty implication of this "old" mode is that the host can remove *accepted* pages that are used in the syscall gap. That means that the #VE handler would need to be of the paranoid variety which opens up all kinds of other fun.
Yeah, I believe this needs to be dealt with anyway, for SNP at least. But on AMD it would simply cause an exception and it'll be handled in the #VC thing. And there's some ugly code to deal with the gap too.
Quoted 10 lines
* "Old" - #VE's can happen in the syscall gap
* "New" - #VE's happen at better-defined times. Unexpected ones are fatal.There's a third option which I proposed but doesn't yet exist. The TDX module _could_ separate the behavior of unaccepted memory #VE's and host-induced #VEs. This way, we could use
load_unaligned_zeropad()with impunity and handle it in the #VE handler. At the same time, the host would not be allowed to remove accepted memory and cause problems in the syscall gap. Kinda the best of both worlds.
I like that. This should've been the default from the get-go. Oh well, what's it called in English, hindsight is 20 20...?
Quoted 2 lines
But, I'm not sure how valuable that would be now that we have the (admittedly squirrelly) code to avoid
load_unaligned_zeropad()#VE's.
I think you should push for the bestest solution and one day we can kill those ugly workarounds.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteAndy Lutomirski
maintainerRe: [PATCHv7 10/14] x86/mm: Avoid load_unaligned_zeropad() stepping into unaccepted memory
Quoted 34 lines · Borislav Petkov
Quoted 4 lines · Dave Hansen
One other thing I remembered as I re-read my write up on this.
In the "new" mode, guests never get #VE's for unaccepted memory. They just exit to the host and can never be reentered. They must be killed.
Yeah, this is the part which I think is really silly.
OSes, in their execution lifetime, can - erroneously or not - but it happens pretty often in real life, touch some unrelated memory. And this has never been a big deal - #PF, that's it.
But now they don't even get a chance to correct their mistake - VMEXIT, die.
load_unaligned_zeropad()is just one case.Imagine the user loads some buggy driver in the guest and that driver starts doing stray memory accesses through a wild pointer into the fields. Guest dies immediately.
Dunno bit it all feels a bit too harsh and unfriendly to me.
Sure, if that user is really unlucky, those stray accesses can kill his OS on baremetal too. So maybe you could argue here that such stray accesses are actually a good thing. :)
All I know is, there should be a more resilient way to handle those.
Quoted 5 lines
In the "old" mode, I _believe_ that the guest always gets a #VE for non-EPT-present memory. The #VE is basically the same no matter if the page is unaccepted or if the host goes out and makes a previously-accepted page non-present.
Quoted 8 lines
Quoted 4 lines
One really nasty implication of this "old" mode is that the host can remove *accepted* pages that are used in the syscall gap. That means that the #VE handler would need to be of the paranoid variety which opens up all kinds of other fun.
Yeah, I believe this needs to be dealt with anyway, for SNP at least. But on AMD it would simply cause an exception and it'll be handled in the #VC thing. And there's some ugly code to deal with the gap too.
I do not believe for a second that the “ugly” code in question is correct. Let’s please not go there for TDX. The whole point of this thing is security — I would rather see a somewhat fragile situation than an exploit.
Now if the TD module could deliver an unrecoverable #MC instead of an impossible-to-handle #VE, maybe we could at least get a nice debug trace out? Of course it’s not so easy to do anything with a debug trace that doesn’t break confidentiality.
So, Dave (and other Intel folks), here’s a different feature I think we will want: a secure way to get debug logs out of a TDX guest. Maybe the guest could have a way to send syslog lines to the TD module (via hypercall or a ring buffer) and the TD module could encrypt them against a secret that is only accessible to future boots of the same guest or to an attested outside key. TDX already has an exceedingly flexible attestation mechanism — there is surely room for the guest config to include a public key that is used for encryption of exported logs.
(Yes, this is sort of doable in software, but the part where any kind of buffer actually gets saved anywhere if the guest goes kaboom requires either help from the TD module or a defined protocol with key negotiation involving the untrusted host.)
The model where the guest leaves a panic on the screen or serial console won’t work in TDX…
Quoted 28 lines
Quoted 11 lines
* "Old" - #VE's can happen in the syscall gap
* "New" - #VE's happen at better-defined times. Unexpected ones are fatal.There's a third option which I proposed but doesn't yet exist. The TDX module _could_ separate the behavior of unaccepted memory #VE's and host-induced #VEs. This way, we could use
load_unaligned_zeropad()with impunity and handle it in the #VE handler. At the same time, the host would not be allowed to remove accepted memory and cause problems in the syscall gap. Kinda the best of both worlds.I like that. This should've been the default from the get-go. Oh well, what's it called in English, hindsight is 20 20...?
Quoted 2 lines
But, I'm not sure how valuable that would be now that we have the (admittedly squirrelly) code to avoid
load_unaligned_zeropad()#VE's.I think you should push for the bestest solution and one day we can kill those ugly workarounds.
Thx.
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Re: [PATCHv7 10/14] x86/mm: Avoid load_unaligned_zeropad() stepping into unaccepted memory
Quoted 4 lines · Andy Lutomirski
Now if the TD module could deliver an unrecoverable #MC instead of an impossible-to-handle #VE, maybe we could at least get a nice debug trace out? Of course it’s not so easy to do anything with a debug trace that doesn’t break confidentiality.
It is not impossible-to-handle #VE, it is no #VE for the guest and exit to the host that cannot be recovered. Yes, it is not friednly for debugging.
Our plan was to allow SEPT_VE_DISABLE=0 for debug TD. It helps with debugging stepping on unaccepted memory as allows #VE in the guest which leads to panic() and nice traceback.
Would it be enough?
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Andy Lutomirski
maintainerRe: [PATCHv7 10/14] x86/mm: Avoid load_unaligned_zeropad() stepping into unaccepted memory
In reply to Dave HansenQuoted 33 lines · Dave Hansen
Quoted 5 lines · Dave Hansen
To sum it all up, I'm not happy with the complexity of the page acceptance code either but I'm not sure that it's bad tradeoff compared to greater #VE complexity or fragility.
Does anyone think we should go back and really reconsider this?
One other thing I remembered as I re-read my write up on this.
In the "new" mode, guests never get #VE's for unaccepted memory. They just exit to the host and can never be reentered. They must be killed.
In the "old" mode, I _believe_ that the guest always gets a #VE for non-EPT-present memory. The #VE is basically the same no matter if the page is unaccepted or if the host goes out and makes a previously-accepted page non-present.
One really nasty implication of this "old" mode is that the host can remove *accepted* pages that are used in the syscall gap. That means that the #VE handler would need to be of the paranoid variety which opens up all kinds of other fun.
* "Old" - #VE's can happen in the syscall gap
* "New" - #VE's happen at better-defined times. Unexpected ones are fatal.There's a third option which I proposed but doesn't yet exist. The TDX module _could_ separate the behavior of unaccepted memory #VE's and host-induced #VEs. This way, we could use
load_unaligned_zeropad()with impunity and handle it in the #VE handler. At the same time, the host would not be allowed to remove accepted memory and cause problems in the syscall gap. Kinda the best of both worlds.
Quoted 2 lines
But, I'm not sure how valuable that would be now that we have the (admittedly squirrelly) code to avoid
load_unaligned_zeropad()#VE's.
How would that be implemented? It would need to track which GPAs *were* accepted across a host-induced unmap/remap cycle. This would involve preventing the host from ever completely removing a secure EPT table without the guest’s help, right?
Admittedly this would IMO be better behavior. Is it practical to implement?
Re: [PATCHv7 10/14] x86/mm: Avoid load_unaligned_zeropad() stepping into unaccepted memory
Quoted 39 lines · Andy Lutomirski
Quoted 33 lines · Dave Hansen
Quoted 5 lines · Dave Hansen
To sum it all up, I'm not happy with the complexity of the page acceptance code either but I'm not sure that it's bad tradeoff compared to greater #VE complexity or fragility.
Does anyone think we should go back and really reconsider this?
One other thing I remembered as I re-read my write up on this.
In the "new" mode, guests never get #VE's for unaccepted memory. They just exit to the host and can never be reentered. They must be killed.
In the "old" mode, I _believe_ that the guest always gets a #VE for non-EPT-present memory. The #VE is basically the same no matter if the page is unaccepted or if the host goes out and makes a previously-accepted page non-present.
One really nasty implication of this "old" mode is that the host can remove *accepted* pages that are used in the syscall gap. That means that the #VE handler would need to be of the paranoid variety which opens up all kinds of other fun.
* "Old" - #VE's can happen in the syscall gap
* "New" - #VE's happen at better-defined times. Unexpected ones are fatal.There's a third option which I proposed but doesn't yet exist. The TDX module _could_ separate the behavior of unaccepted memory #VE's and host-induced #VEs. This way, we could use
load_unaligned_zeropad()with impunity and handle it in the #VE handler. At the same time, the host would not be allowed to remove accepted memory and cause problems in the syscall gap. Kinda the best of both worlds.Quoted 2 lines
But, I'm not sure how valuable that would be now that we have the (admittedly squirrelly) code to avoid
load_unaligned_zeropad()#VE's.
Quoted 6 lines
How would that be implemented? It would need to track which GPAs *were* accepted across a host-induced unmap/remap cycle. This would involve preventing the host from ever completely removing a secure EPT table without the guest’s help, right?
Admittedly this would IMO be better behavior. Is it practical to implement?
I don't think it is better if you look from host POV. It owns resources of the machine and it has to have a way to pull memory from uncooperative TD at any point.
It also would require more complicated private->shared conversion as guest has to notify TDX module about the change, not only host as we do now.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Borislav Petkov
maintainerRe: [PATCHv7 10/14] x86/mm: Avoid load_unaligned_zeropad() stepping into unaccepted memory
In reply to Kirill A. ShutemovQuoted 20 lines · Kirill A. Shutemov
arch/x86/mm/unaccepted_memory.c +12
-0@@ -23,6 +23,38 @@ void accept_memory(phys_addr_t start, phys_addr_t end) bitmap =__va (boot_params.unaccepted_memory); range_start = start /PMD_SIZE ; +/* +* load_unaligned_zeropad() can lead to unwanted loads across page +* boundaries. The unwanted loads are typically harmless. But, they +* might be made to totally unrelated or even unmapped memory. +* load_unaligned_zeropad() relies on exception fixup (#PF, #GP and now +* #VE) to recover from these unwanted loads. +* +* But, this approach does not work for unaccepted memory. For TDX, a +* load from unaccepted memory will not lead to a recoverable exception +* within the guest. The guest will exit to the VMM where the only +* recourse is to terminate the guest. +*
arch/x86/mm/unaccepted_memory.c void accept_memory(phys_addr_t start, phys_addr_t end) · line 47
Quoted 10 lines
arch/x86/mm/unaccepted_memory.c +10
-0+* There are three parts to fix this issue and comprehensively avoid +* access to unaccepted memory. Together these ensure that an extra +* “guard” page is accepted in addition to the memory that needs to be +* used: +* +* 1. Implicitly extend the range_contains_unaccepted_memory(start, end) +* checks up to end+2M if ‘end’ is aligned on a 2M boundary. +* +* 2. Implicitly extend accept_memory(start, end) to end+2M if ‘end’ is +* aligned on a 2M boundary.
Why do we need those unicode quotes and backticks in there?
verify_diff: Warning: Unicode char [“] (0x8220 in line: + * “guard” page is accepted in addition to the memory that needs to beverify_diff: Warning: Unicode char [‘] (0x8216 in line: + * checks up to end+2M if ‘end’ is aligned on a 2M boundary.verify_diff: Warning: Unicode char [‘] (0x8216 in line: + * 2. Implicitly extend accept_memory(start, end) to end+2M if ‘end’ isverify_diff: Warning: Unicode char [‘] (0x8216 in line: + * needs to be done to make ‘page’ usable. That work might includeverify_diff: Warning: Unicode char [‘] (0x8216 in line: + * accepting pages in addition to ‘page’ itself.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCHv7 10/14] x86/mm: Avoid load_unaligned_zeropad() stepping into unaccepted memory
Quoted 7 lines · Borislav Petkov
Why do we need those unicode quotes and backticks in there?
verify_diff: Warning: Unicode char [“] (0x8220in line: + * “guard” page is accepted in addition to the memory that needs to beverify_diff: Warning: Unicode char [‘] (0x8216in line: + * checks up to end+2M if ‘end’ is aligned on a 2M boundary.verify_diff: Warning: Unicode char [‘] (0x8216in line: + * 2. Implicitly extendaccept_memory(start, end)to end+2M if ‘end’ isverify_diff: Warning: Unicode char [‘] (0x8216in line: + * needs to be done to make ‘page’ usable. That work might includeverify_diff: Warning: Unicode char [‘] (0x8216in line: + * accepting pages in addition to ‘page’ itself.
I've been encouraging folks to stick their changelogs in a Google Doc (or even Word) when they're writing them. This gives them better spelling and grammar checking than is available in most editors and also makes it easier for folks to improve it collaboratively. I find it a lot more efficient than sending 10 copies back and forth in email.
The downside is that those fancy programs insert unicode willy nilly for stuff like this. You usually need to catch it with scripts because it's hard to spot visually.
It might make a good checkpatch addition, if it's not already there.
Andy Lutomirski
maintainerRe: [PATCHv7 10/14] x86/mm: Avoid load_unaligned_zeropad() stepping into unaccepted memory
In reply to Kirill A. ShutemovQuoted 10 lines · Kirill A. Shutemov
load_unaligned_zeropad()can lead to unwanted loads across page boundaries. The unwanted loads are typically harmless. But, they might be made to totally unrelated or even unmapped memory.load_unaligned_zeropad()relies on exception fixup (#PF, #GP and now #VE) to recover from these unwanted loads.But, this approach does not work for unaccepted memory. For TDX, a load from unaccepted memory will not lead to a recoverable exception within the guest. The guest will exit to the VMM where the only recourse is to terminate the guest.
Why is unaccepted memory marked present in the direct map in the first place?
Having kernel code assume that every valid address is followed by several bytes of memory that may be read without side effects other than #PF also seems like a mistake, but I probably won’t win that fight. But sticking guard pages in front of definitely-not-logically present pages seems silly to me. Let’s just not map it.
(What if MMIO memory is mapped next to regular memory? Doing random unaligned reads that cross into MMIO seems unwise.)
Re: [PATCHv7 10/14] x86/mm: Avoid load_unaligned_zeropad() stepping into unaccepted memory
Quoted 14 lines · Andy Lutomirski
Quoted 10 lines · Kirill A. Shutemov
load_unaligned_zeropad()can lead to unwanted loads across page boundaries. The unwanted loads are typically harmless. But, they might be made to totally unrelated or even unmapped memory.load_unaligned_zeropad()relies on exception fixup (#PF, #GP and now #VE) to recover from these unwanted loads.But, this approach does not work for unaccepted memory. For TDX, a load from unaccepted memory will not lead to a recoverable exception within the guest. The guest will exit to the VMM where the only recourse is to terminate the guest.
Quoted 7 lines
Why is unaccepted memory marked present in the direct map in the first place?
Having kernel code assume that every valid address is followed by several bytes of memory that may be read without side effects other than #PF also seems like a mistake, but I probably won’t win that fight. But sticking guard pages in front of definitely-not-logically present pages seems silly to me. Let’s just not map it.
It would mean no 1G pages in direct mapping for TDX as we accept 2M a time.
Quoted 2 lines
(What if MMIO memory is mapped next to regular memory? Doing random unaligned reads that cross into MMIO seems unwise.)
MMIO is shared, not unaccpted private. We already handle the situation. See 1e7769653b06 ("x86/tdx: Handle load_unaligned_zeropad() page-cross to a shared page").
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Andy Lutomirski
maintainerRe: [PATCHv7 10/14] x86/mm: Avoid load_unaligned_zeropad() stepping into unaccepted memory
Quoted 26 lines · Kirill A. Shutemov
Quoted 14 lines · Andy Lutomirski
Quoted 10 lines · Kirill A. Shutemov
load_unaligned_zeropad()can lead to unwanted loads across page boundaries. The unwanted loads are typically harmless. But, they might be made to totally unrelated or even unmapped memory.load_unaligned_zeropad()relies on exception fixup (#PF, #GP and now #VE) to recover from these unwanted loads.But, this approach does not work for unaccepted memory. For TDX, a load from unaccepted memory will not lead to a recoverable exception within the guest. The guest will exit to the VMM where the only recourse is to terminate the guest.
Quoted 7 lines
Why is unaccepted memory marked present in the direct map in the first place?
Having kernel code assume that every valid address is followed by several bytes of memory that may be read without side effects other than #PF also seems like a mistake, but I probably won’t win that fight. But sticking guard pages in front of definitely-not-logically present pages seems silly to me. Let’s just not map it.
It would mean no 1G pages in direct mapping for TDX as we accept 2M a time.
Quoted 7 lines
Quoted 2 lines
(What if MMIO memory is mapped next to regular memory? Doing random unaligned reads that cross into MMIO seems unwise.)
MMIO is shared, not unaccpted private. We already handle the situation. See
1e7769653b06("x86/tdx: Handleload_unaligned_zeropad()page-cross to a shared page").
I don’t mean in a confidential guest — I mean generally. This whole model of “overrun the buffer — no big deal” is just fragile.
Quoted 2 lines
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCHv7 10/14] x86/mm: Avoid load_unaligned_zeropad() stepping into unaccepted memory
Quoted 20 lines · Andy Lutomirski
Quoted 17 lines · Kirill A. Shutemov
Quoted 16 lines · Andy Lutomirski
Quoted 10 lines · Kirill A. Shutemov
load_unaligned_zeropad()can lead to unwanted loads across page boundaries. The unwanted loads are typically harmless. But, they might be made to totally unrelated or even unmapped memory.load_unaligned_zeropad()relies on exception fixup (#PF, #GP and now #VE) to recover from these unwanted loads.But, this approach does not work for unaccepted memory. For TDX, a load from unaccepted memory will not lead to a recoverable exception within the guest. The guest will exit to the VMM where the only recourse is to terminate the guest.
Why is unaccepted memory marked present in the direct map in the first place?
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Having kernel code assume that every valid address is followed by several bytes of memory that may be read without side effects other than #PF also seems like a mistake, but I probably won’t win that fight. But sticking guard pages in front of definitely-not-logically present pages seems silly to me. Let’s just not map it.
It would mean no 1G pages in direct mapping for TDX as we accept 2M a time.
As of now, we don't have a way to recover direct mapping from fragmentation. So once we split 1G to 2M it stays this way.
Quoted 10 lines
Quoted 7 lines
Quoted 2 lines
(What if MMIO memory is mapped next to regular memory? Doing random unaligned reads that cross into MMIO seems unwise.)
MMIO is shared, not unaccpted private. We already handle the situation. See
1e7769653b06("x86/tdx: Handleload_unaligned_zeropad()page-cross to a shared page").I don’t mean in a confidential guest — I mean generally. This whole model of “overrun the buffer — no big deal” is just fragile.
If you want to remove load_unaligned_zeropad(), I would not object. It can make life easier.
I presumed that optimization it brings has measuarable benefit (otherwise, why bother).
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
PATCH v7 11/14 x86: Disable kexec if system has unaccepted memory
In reply to Kirill A. ShutemovOn kexec, the target kernel has to know what memory has been accepted. Information in EFI map is out of date and cannot be used.
boot_params.unaccepted_memory can be used to pass the bitmap between two kernels on kexec, but the use-case is not yet implemented.
Disable kexec on machines with unaccepted memory for now.
--- arch/x86/mm/unaccepted_memory.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
arch/x86/mm/unaccepted_memory.c +19 -0
@@ -1,4 +1,5 @@// SPDX-License-Identifier: GPL-2.0-only +#include <linux/kexec.h> #include <linux/memblock.h> #include <linux/mm.h> #include <linux/pfn.h>
@@ -95,3 +96,21 @@ bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end)return ret; } + +static int __init unaccepted_init (void ) +{ +if (!boot_params.unaccepted_memory) +return 0 ; + +#ifdef CONFIG_KEXEC_CORE +/* +* TODO: Information on memory acceptance status has to be communicated +* between kernel. +*/ +pr_warn ("Disable kexec: not yet supported on systems with unaccepted memory\n" ); + kexec_load_disabled =1 ; +#endif + +return 0 ; +} +fs_initcall (unaccepted_init);
Signature · 1 line
2.35.1
Re: [PATCHv7 11/14] x86: Disable kexec if system has unaccepted memory
... adding kexec folks
Quoted 7 lines · Kirill A. Shutemov
On kexec, the target kernel has to know what memory has been accepted. Information in EFI map is out of date and cannot be used.
boot_params.unaccepted_memory can be used to pass the bitmap between two kernels on kexec, but the use-case is not yet implemented.
Disable kexec on machines with unaccepted memory for now.
...
arch/x86/mm/unaccepted_memory.c bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) · line 112
Quoted 13 lines
arch/x86/mm/unaccepted_memory.c +13
-0+static int __init unaccepted_init (void ) +{ +if (!boot_params.unaccepted_memory) +return 0 ; + +#ifdef CONFIG_KEXEC_CORE +/* +* TODO: Information on memory acceptance status has to be communicated +* between kernel. +*/ +pr_warn ("Disable kexec: not yet supported on systems with unaccepted memory\n" ); + kexec_load_disabled =1 ; +#endif
This looks to be the *only* in-kernel user tweaking kexec_load_disabled.
It doesn't feel great to just be disabling kexec like this. Why not
just fix it properly?
What do the kexec folks think?
Re: [PATCHv7 11/14] x86: Disable kexec if system has unaccepted memory
Quoted 29 lines · Dave Hansen
... adding kexec folks
Quoted 7 lines · Kirill A. Shutemov
On kexec, the target kernel has to know what memory has been accepted. Information in EFI map is out of date and cannot be used.
boot_params.unaccepted_memory can be used to pass the bitmap between two kernels on kexec, but the use-case is not yet implemented.
Disable kexec on machines with unaccepted memory for now.
...
arch/x86/mm/unaccepted_memory.c bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) · line 112
Quoted 13 lines
arch/x86/mm/unaccepted_memory.c +13
-0+static int __init unaccepted_init (void ) +{ +if (!boot_params.unaccepted_memory) +return 0 ; + +#ifdef CONFIG_KEXEC_CORE +/* +* TODO: Information on memory acceptance status has to be communicated +* between kernel. +*/ +pr_warn ("Disable kexec: not yet supported on systems with unaccepted memory\n" ); + kexec_load_disabled =1 ; +#endif This looks to be the *only* in-kernel user tweaking
kexec_load_disabled.
It doesn't feel great to just be disabling kexec like this. Why not
just fix it properly?
Quoted 1 line
What do the kexec folks think?
I didn't realized someone had implemented kexec_load_disabled. I am not particularly happy about that. It looks like an over-broad stick that we will have to support forever.
This change looks like it just builds on that bad decision.
If people don't want to deal with this situation right now, then I recommend they make this new code and KEXEC conflict at the Kconfig level. That would give serious incentive to adding the missing implementation.
If there is some deep and fundamental why this can not be supported then it probably makes sense to put some code in the arch_kexec_load hook that verifies that deep and fundamental reason is present.
With the kexec code all we have to verify it works is a little testing and careful code review. Something like this makes code review much harder because the entire kernel has to be checked to see if some random driver without locking changed a variable. Rather than having it apparent that this special case exists when reading through the kexec code.
Eric
Re: [PATCHv7 11/14] x86: Disable kexec if system has unaccepted memory
Quoted 22 lines · Eric W. Biederman
Quoted 20 lines · Dave Hansen
... adding kexec folks
Quoted 7 lines · Kirill A. Shutemov
On kexec, the target kernel has to know what memory has been accepted. Information in EFI map is out of date and cannot be used.
boot_params.unaccepted_memory can be used to pass the bitmap between two kernels on kexec, but the use-case is not yet implemented.
Disable kexec on machines with unaccepted memory for now.
...
Quoted 9 lines
arch/x86/mm/unaccepted_memory.c +9
-0+static int __init unaccepted_init (void ) +{ +if (!boot_params.unaccepted_memory) +return 0 ; + +#ifdef CONFIG_KEXEC_CORE +/* +* TODO: Information on memory acceptance status has to be communicated +* between kernel.
Quoted 8 lines
Quoted 8 lines
arch/x86/mm/unaccepted_memory.c bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) · line 112
Quoted 4 lines
arch/x86/mm/unaccepted_memory.c +4
-0+*/ +pr_warn ("Disable kexec: not yet supported on systems with unaccepted memory\n" ); + kexec_load_disabled =1 ; +#endif This looks to be the *only* in-kernel user tweaking
kexec_load_disabled.
It doesn't feel great to just be disabling kexec like this. Why not
just fix it properly?
Unfortunately, problems with kexec are not limited to the unaccepted memory. Isaku pointed out that MADT CPU wake is also problematic for kexec. It doesn't allow CPU offline so secondary kernel will not be able to wake it up. So additional limitation (as of now) for kexec is !SMP on TDX guest.
I guess we can implement CPU offlining by going to a loop that checks mailbox and responds to the command. That loops has to be somehow protected from being overwritten on kexec.
Other issues may come up as we actually try to implement it.
That's all doable, but feels like a scope creep for unaccepted memory enabling patchset :/
Is it a must for merge consideration?
Quoted 6 lines
Quoted 1 line
What do the kexec folks think?
I didn't realized someone had implemented
kexec_load_disabled. I am not particularly happy about that. It looks like an over-broad stick that we will have to support forever.
Quoted 6 lines
This change looks like it just builds on that bad decision.
If people don't want to deal with this situation right now, then I recommend they make this new code and KEXEC conflict at the
Kconfiglevel. That would give serious incentive to adding the missing implementation.
I tried to limit KEXEC on Kconfig level before[1]. Naive approach does not work[2]:
WARNING: unmet direct dependencies detected for UNACCEPTED_MEMORY
Depends on [n]: EFI [=y] && EFI_STUB [=y] && !KEXEC_CORE [=y]
Selected by [y]:
- INTEL_TDX_GUEST [=y] && HYPERVISOR_GUEST [=y] && X86_64 [=y] && CPU_SUP_INTEL [=y] && X86_X2APIC [=y]
Maybe my Kconfig-fu is not strong enough, I donno.
[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]
Quoted 3 lines
If there is some deep and fundamental why this can not be supported then it probably makes sense to put some code in the
arch_kexec_loadhook that verifies that deep and fundamental reason is present.
Sounds straight-forward. I can do this.
Quoted 9 lines
With the kexec code all we have to verify it works is a little testing and careful code review. Something like this makes code review much harder because the entire kernel has to be checked to see if some random driver without locking changed a variable. Rather than having it apparent that this special case exists when reading through the kexec code.
Eric
Signature · 1 line
Kirill A. Shutemov
Re: [PATCHv7 11/14] x86: Disable kexec if system has unaccepted memory
Quoted 5 lines · Kirill A. Shutemov
Quoted 3 lines
If there is some deep and fundamental why this can not be supported then it probably makes sense to put some code in the
arch_kexec_loadhook that verifies that deep and fundamental reason is present.Sounds straight-forward. I can do this.
What about the patch below?
From 0b758600e1eef5525f2a46630ab3559f118a272a Mon Sep 17 00:00:00 2001
From: "Kirill A. Shutemov" <[email protected]>
Date: Tue, 10 May 2022 19:02:18 +0300
Subject: [PATCH] x86: Disable kexec if system has unaccepted memory
On kexec, the target kernel has to know what memory has been accepted. Information in EFI map is out of date and cannot be used.
boot_params.unaccepted_memory can be used to pass the bitmap between two kernels on kexec, but the use-case is not yet implemented.
Disable kexec on machines with unaccepted memory for now.
--- arch/x86/mm/unaccepted_memory.c | 16 ++++++++++++++++ include/linux/kexec.h | 2 ++ kernel/kexec.c | 4 ++++ kernel/kexec_core.c | 5 +++++ kernel/kexec_file.c | 4 ++++ 5 files changed, 31 insertions(+)
arch/x86/mm/unaccepted_memory.c +16 -0
@@ -1,4 +1,5 @@// SPDX-License-Identifier: GPL-2.0-only +#include <linux/kexec.h> #include <linux/memblock.h> #include <linux/mm.h> #include <linux/pfn.h>
@@ -98,3 +99,18 @@ bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end)return ret; } + +#ifdef CONFIG_KEXEC_CORE +int arch_kexec_load (void ) +{ +if (!boot_params.unaccepted_memory) +return 0 ; + +/* +* TODO: Information on memory acceptance status has to be communicated +* between kernel. +*/ +pr_warn_once ("Disable kexec: not yet supported on systems with unaccepted memory\n" ); +return -EOPNOTSUPP ; +} +#endif
include/linux/kexec.h +2 -0
@@ -396,6 +396,8 @@ void crash_free_reserved_phys_range(unsigned long begin, unsigned long end);void arch_kexec_protect_crashkres (void );void arch_kexec_unprotect_crashkres (void ); +int arch_kexec_load (void ); +#ifndef page_to_boot_pfnstatic inline unsigned long page_to_boot_pfn (struct page *page) {
kernel/kexec.c +4 -0
@@ -195,6 +195,10 @@ static inline int kexec_load_check(unsigned long nr_segments, {int result; + result =arch_kexec_load (); +if (result) +return result; +/* We only trust the superuser with rebooting the system. */ if (!capable (CAP_SYS_BOOT ) || kexec_load_disabled)return -EPERM ;
kernel/kexec_core.c +5 -0
@@ -1238,3 +1238,8 @@ void __weak arch_kexec_protect_crashkres(void)void __weak arch_kexec_unprotect_crashkres (void ) {} + +int __weak arch_kexec_load (void ) +{ +return 0 ; +}
kernel/kexec_file.c +4 -0
@@ -324,6 +324,10 @@ SYSCALL_DEFINE5(kexec_file_load, int, kernel_fd, int, initrd_fd,int ret =0 , i;struct kimage **dest_image, *image; + ret =arch_kexec_load (); +if (ret) +return ret; +/* We only trust the superuser with rebooting the system. */ if (!capable (CAP_SYS_BOOT ) || kexec_load_disabled)return -EPERM ;
Signature · 1 line
Kirill A. Shutemov
Re: [PATCHv7 11/14] x86: Disable kexec if system has unaccepted memory
Quoted 4 lines · Kirill A. Shutemov
Quoted 3 lines · Kirill A. Shutemov
Quoted 3 lines
If there is some deep and fundamental why this can not be supported then it probably makes sense to put some code in the
arch_kexec_loadhook that verifies that deep and fundamental reason is present.
...
arch/x86/mm/unaccepted_memory.c bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) · line 112
Quoted 4 lines
arch/x86/mm/unaccepted_memory.c +4
-0+/* +* TODO: Information on memory acceptance status has to be communicated +* between kernel. +*/
So, the deep and fundamental reason is... drum roll... you haven't
gotten around to implementing bitmap passing yet?!?!? I have the
feeling that wasn't what Eric was looking for.
Re: [PATCHv7 11/14] x86: Disable kexec if system has unaccepted memory
Quoted 6 lines · Dave Hansen
Quoted 4 lines · Kirill A. Shutemov
Quoted 3 lines · Kirill A. Shutemov
Quoted 3 lines
If there is some deep and fundamental why this can not be supported then it probably makes sense to put some code in the
arch_kexec_loadhook that verifies that deep and fundamental reason is present....
Quoted 8 lines
arch/x86/mm/unaccepted_memory.c bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) · line 112
Quoted 4 lines
arch/x86/mm/unaccepted_memory.c +4
-0+/* +* TODO: Information on memory acceptance status has to be communicated +* between kernel. +*/ So, the deep and fundamental reason is... drum roll... you haven't
gotten around to implementing bitmap passing yet?!?!? I have the
feeling that wasn't what Eric was looking for.
The deep fundamental reason is that everything cannot be implemented and upstreamed at once.
Signature · 1 line
Kirill A. Shutemov
Re: [PATCHv7 11/14] x86: Disable kexec if system has unaccepted memory
Quoted 13 lines · Kirill A. Shutemov
Quoted 11 lines · Dave Hansen
Quoted 4 lines · Kirill A. Shutemov
Quoted 3 lines · Kirill A. Shutemov
Quoted 3 lines
If there is some deep and fundamental why this can not be supported then it probably makes sense to put some code in the
arch_kexec_loadhook that verifies that deep and fundamental reason is present....
Quoted 4 lines
arch/x86/mm/unaccepted_memory.c +4
-0+/* +* TODO: Information on memory acceptance status has to be communicated +* between kernel. +*/
Quoted 6 lines
Quoted 3 lines
So, the deep and fundamental reason is... drum roll... you haven't
gotten around to implementing bitmap passing yet?!?!? I have the
feeling that wasn't what Eric was looking for.The deep fundamental reason is that everything cannot be implemented and upstreamed at once.
If the only thing is to pass bitmap to kexec kernel, since you have reserved the bitmap memory I guess it is straightforward to set the kexec bootparams.unaccepted_memory as the old value. Not sure if there are problems when the decompress code accepts memory again though.
for kernel kexec_file_load, refer to function setup_boot_parameters() in arch/x86/kernel/kexec-bzimage64.c for kexec_file_load, for kexec-tools kexec_load code refer to setup_linux_system_parameters() kexec/arch/i386/x86-linux-setup.c
Thanks
Dave
PATCH v7 12/14 x86/tdx: Make _tdx_hypercall() and __tdx_module_call() available in boot stub
In reply to Kirill A. ShutemovMemory acceptance requires a hypercall and one or multiple module calls.
Make helpers for the calls available in boot stub. It has to accept memory where kernel image and initrd are placed.
--- arch/x86/coco/tdx/tdx.c | 26 ------------------ arch/x86/include/asm/shared/tdx.h | 45 +++++++++++++++++++++++++++++++ arch/x86/include/asm/tdx.h | 19 ------------- 3 files changed, 45 insertions(+), 45 deletions(-)
arch/x86/coco/tdx/tdx.c +0 -26
@@ -12,14 +12,6 @@#include <asm/insn-eval.h> #include <asm/pgtable.h> -/* TDX module Call Leaf IDs */ -#define TDX_GET_INFO 1 -#define TDX_GET_VEINFO 3 -#define TDX_ACCEPT_PAGE 6 - -/* TDX hypercall Leaf IDs */ -#define TDVMCALL_MAP_GPA 0x10001 -/* MMIO direction */ #define EPT_READ 0#define EPT_WRITE 1
@@ -34,24 +26,6 @@#define VE_GET_PORT_NUM (e) ((e) >> 16)#define VE_IS_IO_STRING (e) ((e) & BIT(4)) -/* -* Wrapper for standard use of __tdx_hypercall with no output aside from -* return code. -*/ -static inline u64 _tdx_hypercall (u64 fn,u64 r12,u64 r13,u64 r14,u64 r15) -{ -struct tdx_hypercall_args args = { - .r10 =TDX_HYPERCALL_STANDARD , - .r11 = fn, - .r12 = r12, - .r13 = r13, - .r14 = r14, - .r15 = r15, - }; - -return __tdx_hypercall (&args,0 ); -} -/* Called from __tdx_hypercall() for unrecoverable failure */ void __tdx_hypercall_failed (void ) {
arch/x86/include/asm/shared/tdx.h +45 -0
59 lines not on this page · raw message
arch/x86/include/asm/tdx.h +0 -19
@@ -20,21 +20,6 @@#ifndef __ASSEMBLY__ -/* -* Used to gather the output registers values of the TDCALL and SEAMCALL -* instructions when requesting services from the TDX module. -* -* This is a software only structure and not part of the TDX module/VMM ABI. -*/ -struct tdx_module_output { -u64 rcx; -u64 rdx; -u64 r8; -u64 r9; -u64 r10; -u64 r11; -}; -/* * Used by the #VE exception handler to gather the #VE exception * info from the TDX module. This is a software only structure
@@ -55,10 +40,6 @@ struct ve_info {void __init tdx_early_init (void ); -/* Used to communicate with the TDX module */ -u64 __tdx_module_call (u64 fn,u64 rcx,u64 rdx,u64 r8,u64 r9, -struct tdx_module_output *out); -void tdx_get_ve_info (struct ve_info *ve);bool tdx_handle_virt_exception (struct pt_regs *regs,struct ve_info *ve);
Signature · 1 line
2.35.1
Re: [PATCHv7 12/14] x86/tdx: Make _tdx_hypercall() and __tdx_module_call() available in boot stub
Quoted 11 lines · Kirill A. Shutemov
Memory acceptance requires a hypercall and one or multiple module calls.
Make helpers for the calls available in boot stub. It has to accept memory where kernel image and initrd are placed.
Signed-off-by Kirill A. Shutemov <[email protected]>--- arch/x86/coco/tdx/tdx.c | 26 ------------------ arch/x86/include/asm/shared/tdx.h | 45 +++++++++++++++++++++++++++++++ arch/x86/include/asm/tdx.h | 19 ------------- 3 files changed, 45 insertions(+), 45 deletions(-)
PATCH v7 13/14 x86/tdx: Refactor try_accept_one()
In reply to Kirill A. ShutemovRework try_accept_one() to return accepted size instead of modifying 'start' inside the helper. It makes 'start' in-only argumaent and streamlines code on the caller side.
--- arch/x86/coco/tdx/tdx.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-)
arch/x86/coco/tdx/tdx.c +19 -19
@@ -624,18 +624,18 @@ static bool tdx_cache_flush_required(void)return true; } -static bool try_accept_one (phys_addr_t *start,unsigned long len, -enum pg_level pg_level) +static unsigned long try_accept_one (phys_addr_t start,unsigned long len, +enum pg_level pg_level) {unsigned long accept_size =page_level_size (pg_level);u64 tdcall_rcx;u8 page_size; -if (!IS_ALIGNED (*start, accept_size)) -return false; +if (!IS_ALIGNED (start, accept_size)) +return 0 ;if (len < accept_size) -return false; +return 0 ;/* * Pass the page physical address to the TDX module to accept the
@@ -654,15 +654,14 @@ static bool try_accept_one(phys_addr_t *start, unsigned long len, page_size =2 ;break ;default : -return false; +return 0 ; } - tdcall_rcx = *start | page_size; + tdcall_rcx = start | page_size;if (__tdx_module_call (TDX_ACCEPT_PAGE , tdcall_rcx,0 ,0 ,0 ,NULL )) -return false; +return 0 ; - *start += accept_size; -return true; +return accept_size; }/*
@@ -699,21 +698,22 @@ static bool tdx_enc_status_changed(unsigned long vaddr, int numpages, bool enc)*/ while (start < end) {unsigned long len = end - start; +unsigned long accept_size;/* * Try larger accepts first. It gives chance to VMM to keep -* 1G/2M SEPT entries where possible and speeds up process by -* cutting number of hypercalls (if successful). +* 1G/2M Secure EPT entries where possible and speeds up +* process by cutting number of hypercalls (if successful). */ -if (try_accept_one (&start, len,PG_LEVEL_1G )) -continue ; - -if (try_accept_one (&start, len,PG_LEVEL_2M )) -continue ; - -if (!try_accept_one (&start, len,PG_LEVEL_4K )) + accept_size =try_accept_one (start, len,PG_LEVEL_1G ); +if (!accept_size) + accept_size =try_accept_one (start, len,PG_LEVEL_2M ); +if (!accept_size) + accept_size =try_accept_one (start, len,PG_LEVEL_4K ); +if (!accept_size)return false; + start += accept_size; }return true;
Signature · 1 line
2.35.1
Re: [PATCHv7 13/14] x86/tdx: Refactor try_accept_one()
Quoted 2 lines · Kirill A. Shutemov
Rework
try_accept_one()to return accepted size instead of modifying 'start' inside the helper. It makes 'start' in-only argumaent and
^ argument
Quoted 4 lines
streamlines code on the caller side.
Signed-off-by Kirill A. Shutemov <[email protected]>Suggested-by Borislav Petkov <[email protected]>
I'm not sure how much it actually streamlines things. The line count looks pretty similar. But, I do generally dislike implicit return values, so with that typo fixed:
Borislav Petkov
maintainerRe: [PATCHv7 13/14] x86/tdx: Refactor try_accept_one()
In reply to Kirill A. ShutemovQuoted 2 lines · Kirill A. Shutemov
Rework
try_accept_one()to return accepted size instead of modifying 'start' inside the helper. It makes 'start' in-only argumaent and
Unknown word [argumaent] in commit message.
Suggestions: ['argument', 'augment', 'arguments', 'armament', 'argent', "argument's"]
You need a spellchecker enabled in your editor with which you write your commit messages.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v7 14/14 x86/tdx: Add unaccepted memory support
In reply to Kirill A. ShutemovHookup TDX-specific code to accept memory.
Accepting the memory is the same process as converting memory from shared to private: kernel notifies VMM with MAP_GPA hypercall and then accept pages with ACCEPT_PAGE module call.
The implementation in core kernel uses tdx_enc_status_changed(). It already used for converting memory to shared and back for I/O transactions.
Boot stub provides own implementation of tdx_accept_memory(). It is similar in structure to tdx_enc_status_changed(), but only cares about converting memory to private.
--- arch/x86/Kconfig | 1 + arch/x86/boot/compressed/mem.c | 27 ++++++++++- arch/x86/boot/compressed/tdx.c | 78 +++++++++++++++++++++++++++++++ arch/x86/coco/tdx/tdx.c | 30 ++++++++---- arch/x86/include/asm/shared/tdx.h | 2 + arch/x86/mm/unaccepted_memory.c | 9 +++- 6 files changed, 136 insertions(+), 11 deletions(-)
arch/x86/Kconfig +1 -0
@@ -892,6 +892,7 @@ config INTEL_TDX_GUEST select ARCH_HAS_CC_PLATFORM select X86_MEM_ENCRYPT select X86_MCE + select UNACCEPTED_MEMORY help Support running as a guest under Intel TDX. Without this support, the guest kernel can not boot or run under TDX.
arch/x86/boot/compressed/mem.c +26 -1
@@ -5,6 +5,8 @@#include "error.h" #include "find.h" #include "math.h" +#include "tdx.h" +#include <asm/shared/tdx.h> #define PMD_SHIFT 21#define PMD_SIZE (_AC(1, UL) << PMD_SHIFT)
@@ -12,10 +14,33 @@extern struct boot_params *boot_params; +static bool is_tdx_guest (void ) +{ +static bool once; +static bool is_tdx; + +if (!IS_ENABLED (CONFIG_INTEL_TDX_GUEST )) +return false; + +if (!once) { +u32 eax, sig[3 ]; + +cpuid_count (TDX_CPUID_LEAF_ID ,0 , &eax, + &sig[0 ], &sig[2 ], &sig[1 ]); + is_tdx = !memcmp (TDX_IDENT , sig,sizeof (sig)); + once = true; + } + +return is_tdx; +} +static inline void __accept_memory (phys_addr_t start,phys_addr_t end) {/* Platform-specific memory-acceptance call goes here */ -error ("Cannot accept memory" ); +if (is_tdx_guest ()) +tdx_accept_memory (start, end); +else +error ("Cannot accept memory: unknown platform\n" ); }/*
arch/x86/boot/compressed/tdx.c +78 -0
93 lines not on this page · raw message
arch/x86/coco/tdx/tdx.c +21 -9
@@ -664,16 +664,9 @@ static unsigned long try_accept_one(phys_addr_t start, unsigned long len,return accept_size; } -/* -* Inform the VMM of the guest's intent for this physical page: shared with -* the VMM or private to the guest. The VMM is expected to change its mapping -* of the page in response. -*/ -static bool tdx_enc_status_changed (unsigned long vaddr,int numpages,bool enc) +static bool tdx_enc_status_changed_phys (phys_addr_t start,phys_addr_t end, +bool enc) { -phys_addr_t start =__pa (vaddr); -phys_addr_t end =__pa (vaddr + numpages *PAGE_SIZE ); -if (!enc) {/* Set the shared (decrypted) bits: */ start |=cc_mkdec (0 );
@@ -719,6 +712,25 @@ static bool tdx_enc_status_changed(unsigned long vaddr, int numpages, bool enc)return true; } +void tdx_accept_memory (phys_addr_t start,phys_addr_t end) +{ +if (!tdx_enc_status_changed_phys (start, end, true)) +panic ("Accepting memory failed: %#llx-%#llx\n" , start, end); +} + +/* +* Inform the VMM of the guest's intent for this physical page: shared with +* the VMM or private to the guest. The VMM is expected to change its mapping +* of the page in response. +*/ +static bool tdx_enc_status_changed (unsigned long vaddr,int numpages,bool enc) +{ +phys_addr_t start =__pa (vaddr); +phys_addr_t end =__pa (vaddr + numpages *PAGE_SIZE ); + +return tdx_enc_status_changed_phys (start, end, enc); +} +void __init tdx_early_init (void ) {u64 cc_mask;
arch/x86/include/asm/shared/tdx.h +2 -0
@@ -81,5 +81,7 @@ struct tdx_module_output {u64 __tdx_module_call (u64 fn,u64 rcx,u64 rdx,u64 r8,u64 r9,struct tdx_module_output *out); +void tdx_accept_memory (phys_addr_t start,phys_addr_t end); +#endif /* !__ASSEMBLY__ */ #endif /* _ASM_X86_SHARED_TDX_H */
arch/x86/mm/unaccepted_memory.c +8 -1
@@ -7,6 +7,7 @@#include <asm/io.h> #include <asm/setup.h> +#include <asm/shared/tdx.h> #include <asm/unaccepted_memory.h> /* Protects unaccepted memory bitmap */
@@ -62,7 +63,13 @@ void accept_memory(phys_addr_t start, phys_addr_t end)unsigned long len = range_end - range_start;/* Platform-specific memory-acceptance call goes here */ -panic ("Cannot accept memory: unknown platform\n" ); +if (cpu_feature_enabled (X86_FEATURE_TDX_GUEST )) { +tdx_accept_memory (range_start *PMD_SIZE , + range_end *PMD_SIZE ); + }else { +panic ("Cannot accept memory: unknown platform\n" ); + } +bitmap_clear (bitmap, range_start, len); }spin_unlock_irqrestore (&unaccepted_memory_lock, flags);
Signature · 1 line
2.35.1
Re: [PATCHv7 14/14] x86/tdx: Add unaccepted memory support
arch/x86/boot/compressed/mem.c line 44
Quoted 9 lines · Kirill A. Shutemov
static inline void
__accept_memory(phys_addr_t start, phys_addr_t end){arch/x86/boot/compressed/mem.c +4
-1/* Platform-specific memory-acceptance call goes here */ -error ("Cannot accept memory" ); +if (is_tdx_guest ()) +tdx_accept_memory (start, end); +else +error ("Cannot accept memory: unknown platform\n" ); }
There are quite a few of these
if (tdx()) ...
conditions in common code here. Shouldn't this be something like a CC_ATTR_MEM_ACCEPT?
if (cc_platform_has(CC_ATTR_MEM_ACCEPT))
cc_accept_memory(...);
else
error("Cannot accept memory: unknown platform\n");I understand that TDX is the first one to the party. Is this the time to add the cc_ infrastructure?
Re: [PATCHv7 14/14] x86/tdx: Add unaccepted memory support
Quoted 19 lines · Dave Hansen
arch/x86/boot/compressed/mem.c line 44
Quoted 9 lines · Kirill A. Shutemov
static inline void
__accept_memory(phys_addr_t start, phys_addr_t end){arch/x86/boot/compressed/mem.c +4
-1/* Platform-specific memory-acceptance call goes here */ -error ("Cannot accept memory" ); +if (is_tdx_guest ()) +tdx_accept_memory (start, end); +else +error ("Cannot accept memory: unknown platform\n" ); }There are quite a few of these
if (tdx()) ...conditions in common code here. Shouldn't this be something like a
CC_ATTR_MEM_ACCEPT?
Quoted 7 lines
if (cc_platform_has(CC_ATTR_MEM_ACCEPT)) cc_accept_memory(...); else error("Cannot accept memory: unknown platform\n");I understand that TDX is the first one to the party. Is this the time to add the
cc_infrastructure?
We need if tdx() check *somewhere* as how exactly memory gets accepted is specific to a particular platform.
There are two callsites where memory acceptance happens. One of them is in boot stub where we don't have cc_ infrastructure. So it will boil down to a single cc_accept_memory() that will have 'if tdx()' inside.
I don't see much sense in the exercise. We can as well keep the 'if' in accept_memory().
Signature · 1 line
Kirill A. Shutemov
Borislav Petkov
maintainerRe: [PATCHv7 14/14] x86/tdx: Add unaccepted memory support
In reply to Kirill A. ShutemovQuoted 9 lines · Kirill A. Shutemov
arch/x86/boot/compressed/mem.c +9
-0+static bool is_tdx_guest (void ) +{ +static bool once; +static bool is_tdx; + +if (!IS_ENABLED (CONFIG_INTEL_TDX_GUEST )) +return false; + +if (!once) {
arch/x86/boot/compressed/mem.c line 35
Quoted 10 lines
arch/x86/boot/compressed/mem.c +10
-0+u32 eax, sig[3 ]; + +cpuid_count (TDX_CPUID_LEAF_ID ,0 , &eax, + &sig[0 ], &sig[2 ], &sig[1 ]); + is_tdx = !memcmp (TDX_IDENT , sig,sizeof (sig)); + once = true; + } + +return is_tdx; +}
early_tdx_detect() already calls this CPUID function. It assigns function pointers too.
So why can't you assign an accept_memory() function pointer there and get rid of this sprinkled if (tdx) everywhere?
Quoted 21 lines
arch/x86/boot/compressed/tdx.c +3
-0@@ -3,12 +3,15 @@#include "../cpuflags.h" #include "../string.h" #include "../io.h" +#include "align.h" #include "error.h" +#include "pgtable_types.h" #include <vdso/limits.h> #include <uapi/asm/vmx.h> #include <asm/shared/tdx.h> +#include <asm/page_types.h> /* Called from __tdx_hypercall() for unrecoverable failure */ void __tdx_hypercall_failed (void )
arch/x86/boot/compressed/tdx.c void early_tdx_detect(void) · line 83
Quoted 6 lines
arch/x86/boot/compressed/tdx.c +3
-0@@ -75,3 +78,78 @@ void early_tdx_detect(void) pio_ops.f_outb = tdx_outb; pio_ops.f_outw = tdx_outw; } + +static unsigned long try_accept_one (phys_addr_t start,unsigned long len, +enum pg_level level)
That's pretty much a copy of the same function in arch/x86/coco/tdx/tdx.c.
Yeah, you need a tdx-shared.c which you include in both places just like it is done with sev-shared.c
...
arch/x86/boot/compressed/tdx.c void early_tdx_detect(void) · line 84
Quoted 1 line
arch/x86/boot/compressed/tdx.c +1
-0+void tdx_accept_memory (phys_addr_t start,phys_addr_t end)
That one too.
Quoted 21 lines
arch/x86/boot/compressed/tdx.c +21
-0+{ +/* +* Notify the VMM about page mapping conversion. More info about ABI +* can be found in TDX Guest-Host-Communication Interface (GHCI), +* section "TDG.VP.VMCALL<MapGPA>" +*/ +if (_tdx_hypercall (TDVMCALL_MAP_GPA , start, end - start,0 ,0 )) +error ("Accepting memory failed\n" ); + +/* +* For shared->private conversion, accept the page using +* TDX_ACCEPT_PAGE TDX module call. +*/ +while (start < end) { +unsigned long len = end - start; +unsigned long accept_size; + +/* +* Try larger accepts first. It gives chance to VMM to keep +* 1G/2M Secure EPT entries where possible and speeds up +* process by cutting number of hypercalls (if successful).
arch/x86/boot/compressed/tdx.c void early_tdx_detect(void) · line 153
Quoted 10 lines
arch/x86/boot/compressed/tdx.c +10
-0+*/ + + accept_size =try_accept_one (start, len,PG_LEVEL_1G ); +if (!accept_size) + accept_size =try_accept_one (start, len,PG_LEVEL_2M ); +if (!accept_size) + accept_size =try_accept_one (start, len,PG_LEVEL_4K ); +if (!accept_size) +error ("Accepting memory failed\n" ); + start += accept_size;
This series of calls to try_accept_one() appear in at least three places. Please carve them out into a separate function can put it in tdx-shared.c.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCHv7 14/14] x86/tdx: Add unaccepted memory support
Quoted 24 lines · Borislav Petkov
Quoted 9 lines · Kirill A. Shutemov
arch/x86/boot/compressed/mem.c +9
-0+static bool is_tdx_guest (void ) +{ +static bool once; +static bool is_tdx; + +if (!IS_ENABLED (CONFIG_INTEL_TDX_GUEST )) +return false; + +if (!once) {arch/x86/boot/compressed/mem.c line 35
Quoted 10 lines
arch/x86/boot/compressed/mem.c +10
-0+u32 eax, sig[3 ]; + +cpuid_count (TDX_CPUID_LEAF_ID ,0 , &eax, + &sig[0 ], &sig[2 ], &sig[1 ]); + is_tdx = !memcmp (TDX_IDENT , sig,sizeof (sig)); + once = true; + } + +return is_tdx; +}
early_tdx_detect()already calls this CPUID function. It assigns function pointers too.
Quoted 2 lines
So why can't you assign an
accept_memory()function pointer there and get rid of this sprinkled if (tdx) everywhere?
This code called from EFI stub which runs before decompressor code and therefore before early_tdx_detect().
Quoted 30 lines
Quoted 21 lines
arch/x86/boot/compressed/tdx.c +3
-0@@ -3,12 +3,15 @@#include "../cpuflags.h" #include "../string.h" #include "../io.h" +#include "align.h" #include "error.h" +#include "pgtable_types.h" #include <vdso/limits.h> #include <uapi/asm/vmx.h> #include <asm/shared/tdx.h> +#include <asm/page_types.h> /* Called from __tdx_hypercall() for unrecoverable failure */ void __tdx_hypercall_failed (void )arch/x86/boot/compressed/tdx.c void early_tdx_detect(void) · line 83
Quoted 6 lines
arch/x86/boot/compressed/tdx.c +3
-0@@ -75,3 +78,78 @@ void early_tdx_detect(void) pio_ops.f_outb = tdx_outb; pio_ops.f_outw = tdx_outw; } + +static unsigned long try_accept_one (phys_addr_t start,unsigned long len, +enum pg_level level)That's pretty much a copy of the same function in
arch/x86/coco/tdx/tdx.c.
Quoted 2 lines
Yeah, you need a
tdx-shared.cwhich you include in both places just like it is done withsev-shared.c
Okay, will look into this.
Quoted 4 lines
Quoted 4 lines
arch/x86/boot/compressed/tdx.c +4
-0+ accept_size =try_accept_one (start, len,PG_LEVEL_1G ); +if (!accept_size) + accept_size =try_accept_one (start, len,PG_LEVEL_2M ); +if (!accept_size)
Quoted 8 lines
arch/x86/boot/compressed/tdx.c void early_tdx_detect(void) · line 153
Quoted 4 lines
arch/x86/boot/compressed/tdx.c +4
-0+ accept_size =try_accept_one (start, len,PG_LEVEL_4K ); +if (!accept_size) +error ("Accepting memory failed\n" ); + start += accept_size;This series of calls to
try_accept_one()appear in at least three places. Please carve them out into a separate function can put it intdx-shared.c.
Okay.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Borislav Petkov
maintainerRe: [PATCHv7 14/14] x86/tdx: Add unaccepted memory support
Quoted 2 lines · Kirill A. Shutemov
This code called from EFI stub which runs before decompressor code and therefore before
early_tdx_detect().
Then pls call that function early_is_tdx_guest() and slap a comment above it explaining why it needs to be a separate thing.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 35 lines · Kirill A. Shutemov
UEFI Specification version 2.9 introduces the concept of memory
acceptance: some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, requiring memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific for the Virtual Machine platform.Accepting memory is costly and it makes VMM allocate memory for the accepted guest physical address range. It's better to postpone memory acceptance until memory is needed. It lowers boot time and reduces memory overhead.
The kernel needs to know what memory has been accepted. Firmware communicates this information via memory map: a new memory type --
EFI_UNACCEPTED_MEMORY-- indicates such memory.Range-based tracking works fine for firmware, but it gets bulky for the kernel: e820 has to be modified on every page acceptance. It leads to table fragmentation, but there's a limited number of entries in the e820 table
Another option is to mark such memory as usable in e820 and track if the range has been accepted in a bitmap. One bit in the bitmap represents 2MiB in the address space: one 4k page is enough to track 64GiB or physical address space.
In the worst-case scenario -- a huge hole in the middle of the address space -- It needs 256MiB to handle 4PiB of the address space.
Any unaccepted memory that is not aligned to 2M gets accepted upfront.
The approach lowers boot time substantially. Boot to shell is ~2.5x faster for 4G TDX VM and ~4x faster for 64G.
Quoted 7 lines
TDX-specific code isolated from the core of unaccepted memory support. It supposed to help to plug-in different implementation of unaccepted memory such as SEV-SNP.
The tree can be found here:
https://github.com/intel/tdx.git guest-unaccepted-memory
Hi Kirill,
I have a couple questions about this feature mainly about how cloud customers can use this, I assume since this is a confidential compute feature a large number of the users of these patches will be cloud customers using TDX and SNP. One issue I see with these patches is how do we as a cloud provider know whether a customer's linux image supports this feature, if the image doesn't have these patches UEFI needs to fully validate the memory, if the image does we can use this new protocol. In GCE we supply our VMs with a version of the EDK2 FW and the customer doesn't input into which UEFI we run, as far as I can tell from the Azure SNP VM documentation it seems very similar. We need to somehow tell our UEFI in the VM what to do based on the image. The current way I can see to solve this issue would be to have our customers give us metadata about their VM's image but this seems kinda burdensome on our customers (I assume we'll have more features which both UEFI and kernel need to both support inorder to be turned on like this one) and error-prone, if a customer incorrectly labels their image it may fail to boot.. Has there been any discussion about how to solve this? My naive thoughts were what if UEFI and Kernel had some sort of feature negotiation. Maybe that could happen via an extension to exit boot services or a UEFI runtime driver, I'm not sure what's best here just some ideas.
Quoted 115 lines
v7:
- Rework meminfo counter to usePageUnaccepted()and move to generic code;
- Fixrange_contains_unaccepted_memory()on machines without unaccepted memory;
- Add Reviewed-by from David;
v6:
- Fixload_unaligned_zeropad()on machine with unaccepted memory;
- ClearPageUnaccepted()on merged pages, leaving it only on head;
- Clarify error handling inallocate_e820();
- Fix build withCONFIG_UNACCEPTED_MEMORY=y, but without TDX;
- Disable kexec at boottime instead of build conflict;
- Rebased to tip/master;
- Spelling fixes;
- Add Reviewed-by from Mike and David;
v5:
- Updates comments and commit messages;
+ Explain options for unaccepted memory handling;
- Expose amount of unaccepted memory in/proc/meminfo
- Adjust check inpage_expected_state();
- Fix error code handling inallocate_e820();
- Centralize __pa()/__va() definitions in the boot stub;
- Avoid includes from the main kernel in the boot stub;
- Use an existing hole inboot_paramforunaccepted_memory, instead of adding to the end of the structure;
- Extractallocate_unaccepted_memory()formallocate_e820();
- Complain if there's unaccepted memory, but kernel does not support it;
- Fix vmstat counter;
- Split up few preparatory patches;
- Random readability adjustments;
v4:
-PageBuddyUnaccepted()-> PageUnaccepted;
- Use separatepage_type, not shared with offline;
- Rework interface between core-mm and arch code;
- Adjust commit messages;
- Ack from Mike;Kirill A. Shutemov (14):
x86/boot: Centralize __pa()/__va() definitions
mm: Add support for unaccepted memory
mm: Report unaccepted memory in meminfo
efi/x86: Get full memory map inallocate_e820()
x86/boot: Add infrastructure required for unaccepted memory support efi/x86: Implement support for unaccepted memory x86/boot/compressed: Handle unaccepted memory
x86/mm: Reserve unaccepted memory bitmap
x86/mm: Provide helpers for unaccepted memory
x86/mm: Avoidload_unaligned_zeropad()stepping into unaccepted memory
x86: Disable kexec if system has unaccepted memory x86/tdx: Make_tdx_hypercall()and__tdx_module_call()available in
boot stub
x86/tdx: Refactortry_accept_one()
x86/tdx: Add unaccepted memory supportDocumentation/x86/zero-page.rst | 1 + arch/x86/Kconfig | 1 + arch/x86/boot/bitops.h | 40 ++++++++ arch/x86/boot/compressed/Makefile | 1 + arch/x86/boot/compressed/align.h | 14 +++ arch/x86/boot/compressed/bitmap.c | 43 ++++++++ arch/x86/boot/compressed/bitmap.h | 49 +++++++++ arch/x86/boot/compressed/bits.h | 36 +++++++ arch/x86/boot/compressed/compiler.h | 9 ++ arch/x86/boot/compressed/efi.h | 1 + arch/x86/boot/compressed/find.c | 54 ++++++++++ arch/x86/boot/compressed/find.h | 80 +++++++++++++++ arch/x86/boot/compressed/ident_map_64.c | 8 -- arch/x86/boot/compressed/kaslr.c | 35 ++++--- arch/x86/boot/compressed/math.h | 37 +++++++ arch/x86/boot/compressed/mem.c | 111 ++++++++++++++++++++ arch/x86/boot/compressed/minmax.h | 61 +++++++++++ arch/x86/boot/compressed/misc.c | 6 ++ arch/x86/boot/compressed/misc.h | 15 +++ arch/x86/boot/compressed/pgtable_types.h | 25 +++++ arch/x86/boot/compressed/sev.c | 2 - arch/x86/boot/compressed/tdx.c | 78 ++++++++++++++ arch/x86/coco/tdx/tdx.c | 94 ++++++++--------- arch/x86/include/asm/page.h | 3 + arch/x86/include/asm/shared/tdx.h | 47 +++++++++ arch/x86/include/asm/tdx.h | 19 ---- arch/x86/include/asm/unaccepted_memory.h | 16 +++ arch/x86/include/uapi/asm/bootparam.h | 2 +- arch/x86/kernel/e820.c | 10 ++ arch/x86/mm/Makefile | 2 + arch/x86/mm/unaccepted_memory.c | 123 +++++++++++++++++++++++ drivers/base/node.c | 7 ++ drivers/firmware/efi/Kconfig | 14 +++ drivers/firmware/efi/efi.c | 1 + drivers/firmware/efi/libstub/x86-stub.c | 103 ++++++++++++++++--- fs/proc/meminfo.c | 5 + include/linux/efi.h | 3 +- include/linux/mmzone.h | 1 + include/linux/page-flags.h | 31 ++++++ mm/internal.h | 12 +++ mm/memblock.c | 9 ++ mm/page_alloc.c | 96 +++++++++++++++++- mm/vmstat.c | 1 + 43 files changed, 1191 insertions(+), 115 deletions(-) create mode 100644 arch/x86/boot/compressed/align.h create mode 100644 arch/x86/boot/compressed/bitmap.c create mode 100644 arch/x86/boot/compressed/bitmap.h create mode 100644 arch/x86/boot/compressed/bits.h create mode 100644 arch/x86/boot/compressed/compiler.h create mode 100644 arch/x86/boot/compressed/find.c create mode 100644 arch/x86/boot/compressed/find.h create mode 100644 arch/x86/boot/compressed/math.h create mode 100644 arch/x86/boot/compressed/mem.c create mode 100644 arch/x86/boot/compressed/minmax.h create mode 100644 arch/x86/boot/compressed/pgtable_types.h create mode 100644 arch/x86/include/asm/unaccepted_memory.h create mode 100644 arch/x86/mm/unaccepted_memory.c--
2.35.1
Peter, is your enter key broken? You seem to be typing all your text in a single unreadable paragraph.
Quoted 1 line · Peter Gonda
if a customer incorrectly labels their image it may fail to boot..
You're saying that firmware basically has two choices:
1. Accept all the memory up front and boot slowly, but reliably
2. Use thus "unaccepted memory" mechanism, boot fast, but risk that the VM loses a bunch of memory.
If the guest can't even boot because of a lack of memory, then the pre-accepted chunk is probably too small in the first place.
If the customer screws up, they lose a bunch of the RAM they paid for. That seems like a rather self-correcting problem to me.
Quoted 12 lines · Dave Hansen
Peter, is your enter key broken? You seem to be typing all your text in a single unreadable paragraph.
Quoted 1 line · Peter Gonda
if a customer incorrectly labels their image it may fail to boot..
You're saying that firmware basically has two choices:
1. Accept all the memory up front and boot slowly, but reliably
2. Use thus "unaccepted memory" mechanism, boot fast, but risk that the VM loses a bunch of memory.
Quoted 5 lines
If the guest can't even boot because of a lack of memory, then the pre-accepted chunk is probably too small in the first place.
If the customer screws up, they lose a bunch of the RAM they paid for. That seems like a rather self-correcting problem to me.
I think Peter's point is a little more nuanced than that. Once lazy accept goes into the guest firmware -- without the feature negotiation that Peter is suggesting -- cloud providers now have a bookkeeping problem. Which images have kernels that can boot from a guest firmware that doesn't pre-validate all the guest memory?
The way we've been solving similar bookkeeping problems up to now (e.g., Which guest can run with CVM features like TDX/SEV enabled? Which SEV guests can live migrate?) is as follows. We tag images with feature tags. But this is sort of a hack. And not a great one. It's confusing to customers, hard for the cloud service provider to support, and easy to mess up.
It would be better if the guest FW knew whether or not the kernel it was going to launch supported lazy accept.
That being said, this does seem like a difficult problem to solve, since it's sort of backward from how things work, in that when the guest firmware wants to switch between pre-validating all memory vs. minimizing what it pre-validates, the guest kernel is not running yet! But if there is some way to do this, it would be a huge improvement over the current status quo of pushing the feature negotiation up to the cloud service provider and ultimately the cloud customer.
Quoted 5 lines · Marc Orr
I think Peter's point is a little more nuanced than that. Once lazy accept goes into the guest firmware -- without the feature negotiation that Peter is suggesting -- cloud providers now have a bookkeeping problem. Which images have kernels that can boot from a guest firmware that doesn't pre-validate all the guest memory?
Hold on a sec though...
Is this a matter of
can boot from a guest firmware that doesn't pre-validate all the guest memory?
or
can boot from a guest firmware that doesn't pre-validate all the guest memory ... with access to all of that guest's RAM?
In other words, are we talking about "fails to boot" or "can't see all the RAM"?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Peter, is your enter key broken? You seem to be typing all your text in a single unreadable paragraph.
Sorry I will try to format better in the future.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
You're saying that firmware basically has two choices:
1. Accept all the memory up front and boot slowly, but reliably
2. Use thus "unaccepted memory" mechanism, boot fast, but risk that the VM loses a bunch of memory.
That's right. Given that the first round of SNP guest patches are in but this work to support unaccepted memory for SNP is not we assume we will have distros that support SNP without this "unaccepted memory" feature.
Quoted 15 lines · Dave Hansen
Quoted 5 lines · Marc Orr
I think Peter's point is a little more nuanced than that. Once lazy accept goes into the guest firmware -- without the feature negotiation that Peter is suggesting -- cloud providers now have a bookkeeping problem. Which images have kernels that can boot from a guest firmware that doesn't pre-validate all the guest memory?
Hold on a sec though...
Is this a matter of
can boot from a guest firmware that doesn't pre-validate all the guest memory?
Quoted 8 lines
or
can boot from a guest firmware that doesn't pre-validate all the guest memory ... with access to all of that guest's RAM?
In other words, are we talking about "fails to boot" or "can't see all the RAM"?
Yes, I'm sorry I was mistaken. If FW uses unaccepted memory but the kernel doesn't support it the VM should still boot but will fail to utilize all of its given RAM.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
If the customer screws up, they lose a bunch of the RAM they paid for. That seems like a rather self-correcting problem to me.
Providing customers with an easy to use product is a problem for us the cloud provider, encoding foot-guns doesn't sound like what's best for the user here. I wanted to bring this up here since it seems like a problem most vendors/users of SNP and TDX would run into. We can of course figure this out internally if no one else sees this as an issue.
Quoted 15 lines · Dave Hansen
Quoted 5 lines · Marc Orr
I think Peter's point is a little more nuanced than that. Once lazy accept goes into the guest firmware -- without the feature negotiation that Peter is suggesting -- cloud providers now have a bookkeeping problem. Which images have kernels that can boot from a guest firmware that doesn't pre-validate all the guest memory?
Hold on a sec though...
Is this a matter of
can boot from a guest firmware that doesn't pre-validate all the guest memory?
Quoted 7 lines
or
can boot from a guest firmware that doesn't pre-validate all the guest memory ... with access to all of that guest's RAM?
In other words, are we talking about "fails to boot" or "can't see all the RAM"?
Ah... yeah, you're right, Dave -- I guess it's the latter. The guest won't have access to all of the memory that the customer is paying for. But that's still bad. If the customer buys a 96 GB VM and can only see 4GB because they're kernel doesn't have these patches they're going to be confused and frustrated.
Quoted 22 lines · Marc Orr
Quoted 20 lines · Dave Hansen
Quoted 5 lines · Marc Orr
I think Peter's point is a little more nuanced than that. Once lazy accept goes into the guest firmware -- without the feature negotiation that Peter is suggesting -- cloud providers now have a bookkeeping problem. Which images have kernels that can boot from a guest firmware that doesn't pre-validate all the guest memory?
Hold on a sec though...
Is this a matter of
can boot from a guest firmware that doesn't pre-validate all the guest memory?
or
can boot from a guest firmware that doesn't pre-validate all the guest memory ... with access to all of that guest's RAM?
Quoted 8 lines
Quoted 2 lines
In other words, are we talking about "fails to boot" or "can't see all the RAM"?
Ah... yeah, you're right, Dave -- I guess it's the latter. The guest won't have access to all of the memory that the customer is paying for. But that's still bad. If the customer buys a 96 GB VM and can only see 4GB because they're kernel doesn't have these patches they're going to be confused and frustrated.
The other error case which might be more confusing to the customer is their kernel does have these patches, there is some misconfiguration and their VM boots slowly because the FW uses the accept all memory approach.
Quoted 10 lines · Marc Orr
Quoted 10 lines
Is this a matter of
can boot from a guest firmware that doesn't pre-validate all the guest memory?
or
can boot from a guest firmware that doesn't pre-validate all the guest memory ... with access to all of that guest's RAM?
Quoted 7 lines
Quoted 2 lines
In other words, are we talking about "fails to boot" or "can't see all the RAM"?
Ah... yeah, you're right, Dave -- I guess it's the latter. The guest won't have access to all of the memory that the customer is paying for. But that's still bad. If the customer buys a 96 GB VM and can only see 4GB because they're kernel doesn't have these patches they're going to be confused and frustrated.
They'll at least be a _bit_ less angry and frustrated than if they were staring at a blank screen. ;) But, yeah, I totally get the point.
How big is the window going to be where we have guests that can have unaccepted memory, but don't have acceptance support? For TDX, it's looking like it'll probably _just_ be 5.19. Is TDX on 5.19 in shape that cloud providers can deploy it? Or, is stuff like lack of attestation a deal breaker?
Quoted 14 lines · Dave Hansen
Quoted 12 lines · Marc Orr
Quoted 12 lines
Is this a matter of
can boot from a guest firmware that doesn't pre-validate all the guest memory?
or
can boot from a guest firmware that doesn't pre-validate all the guest memory ... with access to all of that guest's RAM?
In other words, are we talking about "fails to boot" or "can't see all the RAM"?
Quoted 8 lines
Quoted 5 lines
Ah... yeah, you're right, Dave -- I guess it's the latter. The guest won't have access to all of the memory that the customer is paying for. But that's still bad. If the customer buys a 96 GB VM and can only see 4GB because they're kernel doesn't have these patches they're going to be confused and frustrated.
They'll at least be a _bit_ less angry and frustrated than if they were staring at a blank screen. ;) But, yeah, I totally get the point.
Ha! Well we do have that issue in some cases. If you try to run an SEV VM with an image that doesn't support SEV you will just get a blank serial screen. If we had something like this back then the FW could have surfaced a nice error to the user but that's history now.
Quoted 6 lines
How big is the window going to be where we have guests that can have unaccepted memory, but don't have acceptance support? For TDX, it's looking like it'll probably _just_ be 5.19. Is TDX on 5.19 in shape that cloud providers can deploy it? Or, is stuff like lack of attestation a deal breaker?
This is complicated because distros don't run upstream linux versions. If I understand correctly (I see some distro emails on here so please correct me) distros normally maintain forks which they backport things into. So I cannot answer this question. It is possible that a hypothetical distro backports only the SNP/TDX initial patches and doesn't take these for many releases.
I am more familiar with SNP and it does have some attestation support in the first patch sets.
Also I should have been more clear. I don't want to try and hold up this feature but instead discuss a future usability add-on feature.
Quoted 2 lines
Quoted 11 lines · Peter Gonda
Quoted 5 lines
How big is the window going to be where we have guests that can have unaccepted memory, but don't have acceptance support? For TDX, it's looking like it'll probably _just_ be 5.19. Is TDX on 5.19 in shape that cloud providers can deploy it? Or, is stuff like lack of attestation a deal breaker?
This is complicated because distros don't run upstream linux versions. If I understand correctly (I see some distro emails on here so please correct me) distros normally maintain forks which they backport things into. So I cannot answer this question. It is possible that a hypothetical distro backports only the SNP/TDX initial patches and doesn't take these for many releases.
Distros could also backport a bare-bones version of this set that doesn't do anything fancy and just synchronously accepts the memory at boot. No bitmap, no page allocator changes. It'll slow boot down, but is better than having no RAM.
Quoted 64 lines · Peter Gonda
Quoted 35 lines · Kirill A. Shutemov
UEFI Specification version 2.9 introduces the concept of memory
acceptance: some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, requiring memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific for the Virtual Machine platform.Accepting memory is costly and it makes VMM allocate memory for the accepted guest physical address range. It's better to postpone memory acceptance until memory is needed. It lowers boot time and reduces memory overhead.
The kernel needs to know what memory has been accepted. Firmware communicates this information via memory map: a new memory type --
EFI_UNACCEPTED_MEMORY-- indicates such memory.Range-based tracking works fine for firmware, but it gets bulky for the kernel: e820 has to be modified on every page acceptance. It leads to table fragmentation, but there's a limited number of entries in the e820 table
Another option is to mark such memory as usable in e820 and track if the range has been accepted in a bitmap. One bit in the bitmap represents 2MiB in the address space: one 4k page is enough to track 64GiB or physical address space.
In the worst-case scenario -- a huge hole in the middle of the address space -- It needs 256MiB to handle 4PiB of the address space.
Any unaccepted memory that is not aligned to 2M gets accepted upfront.
The approach lowers boot time substantially. Boot to shell is ~2.5x faster for 4G TDX VM and ~4x faster for 64G.
Quoted 7 lines
TDX-specific code isolated from the core of unaccepted memory support. It supposed to help to plug-in different implementation of unaccepted memory such as SEV-SNP.
The tree can be found here:
Hi Kirill,
I have a couple questions about this feature mainly about how cloud customers can use this, I assume since this is a confidential compute feature a large number of the users of these patches will be cloud customers using TDX and SNP. One issue I see with these patches is how do we as a cloud provider know whether a customer's linux image supports this feature, if the image doesn't have these patches UEFI needs to fully validate the memory, if the image does we can use this new protocol. In GCE we supply our VMs with a version of the EDK2 FW and the customer doesn't input into which UEFI we run, as far as I can tell from the Azure SNP VM documentation it seems very similar. We need to somehow tell our UEFI in the VM what to do based on the image. The current way I can see to solve this issue would be to have our customers give us metadata about their VM's image but this seems kinda burdensome on our customers (I assume we'll have more features which both UEFI and kernel need to both support inorder to be turned on like this one) and error-prone, if a customer incorrectly labels their
Quoted 5 lines
image it may fail to boot.. Has there been any discussion about how to solve this? My naive thoughts were what if UEFI and Kernel had some sort of feature negotiation. Maybe that could happen via an extension to exit boot services or a UEFI runtime driver, I'm not sure what's best here just some ideas.
Not sure if you've seen this thread or not, but there's also been some discussion around this in the context of the UEFI support:
2 things being discussed there really, which I think roughly boil down to:
1) how to configure OVMF to enable/disable lazy acceptance
- compile time option most likely: accept-all/accept-minimum/accept-1GB
2) how to introduce an automatic mode in the future where OVMF does the right thing based on what the guest supports. Gerd floated the idea of tying it to ExitBootServices as well, but not sure there's a solid plan on what to do here yet.
If that's accurate, it seems like the only 'safe' option is to disable it via #1 (accept-all), and then when #2 comes along, compile OVMF to just Do The Right Thing.
Users who know their VMs implement lazy acceptance can force it on via accept-all OVMF compile option.
-Mike
Quoted 9 lines · Michael Roth
1) how to configure OVMF to enable/disable lazy acceptance
- compile time option most likely: accept-all/accept-minimum/accept-1GB2) how to introduce an automatic mode in the future where OVMF does the right thing based on what the guest supports. Gerd floated the idea of tying it to ExitBootServices as well, but not sure there's a solid plan on what to do here yet.
Quoted 6 lines
If that's accurate, it seems like the only 'safe' option is to disable it via #1 (accept-all), and then when #2 comes along, compile OVMF to just Do The Right Thing.
Users who know their VMs implement lazy acceptance can force it on via accept-all OVMF compile option.
accept-min / accept-X I mean.
Quoted 88 lines · Michael Roth
Quoted 64 lines · Peter Gonda
Quoted 35 lines · Kirill A. Shutemov
UEFI Specification version 2.9 introduces the concept of memory
acceptance: some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, requiring memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific for the Virtual Machine platform.Accepting memory is costly and it makes VMM allocate memory for the accepted guest physical address range. It's better to postpone memory acceptance until memory is needed. It lowers boot time and reduces memory overhead.
The kernel needs to know what memory has been accepted. Firmware communicates this information via memory map: a new memory type --
EFI_UNACCEPTED_MEMORY-- indicates such memory.Range-based tracking works fine for firmware, but it gets bulky for the kernel: e820 has to be modified on every page acceptance. It leads to table fragmentation, but there's a limited number of entries in the e820 table
Another option is to mark such memory as usable in e820 and track if the range has been accepted in a bitmap. One bit in the bitmap represents 2MiB in the address space: one 4k page is enough to track 64GiB or physical address space.
In the worst-case scenario -- a huge hole in the middle of the address space -- It needs 256MiB to handle 4PiB of the address space.
Any unaccepted memory that is not aligned to 2M gets accepted upfront.
The approach lowers boot time substantially. Boot to shell is ~2.5x faster for 4G TDX VM and ~4x faster for 64G.
Quoted 7 lines
TDX-specific code isolated from the core of unaccepted memory support. It supposed to help to plug-in different implementation of unaccepted memory such as SEV-SNP.
The tree can be found here:
Hi Kirill,
I have a couple questions about this feature mainly about how cloud customers can use this, I assume since this is a confidential compute feature a large number of the users of these patches will be cloud customers using TDX and SNP. One issue I see with these patches is how do we as a cloud provider know whether a customer's linux image supports this feature, if the image doesn't have these patches UEFI needs to fully validate the memory, if the image does we can use this new protocol. In GCE we supply our VMs with a version of the EDK2 FW and the customer doesn't input into which UEFI we run, as far as I can tell from the Azure SNP VM documentation it seems very similar. We need to somehow tell our UEFI in the VM what to do based on the image. The current way I can see to solve this issue would be to have our customers give us metadata about their VM's image but this seems kinda burdensome on our customers (I assume we'll have more features which both UEFI and kernel need to both support inorder to be turned on like this one) and error-prone, if a customer incorrectly labels their
Quoted 5 lines
image it may fail to boot.. Has there been any discussion about how to solve this? My naive thoughts were what if UEFI and Kernel had some sort of feature negotiation. Maybe that could happen via an extension to exit boot services or a UEFI runtime driver, I'm not sure what's best here just some ideas.
Not sure if you've seen this thread or not, but there's also been some discussion around this in the context of the UEFI support:
2 things being discussed there really, which I think roughly boil down to:
1) how to configure OVMF to enable/disable lazy acceptance
- compile time option most likely: accept-all/accept-minimum/accept-1GB2) how to introduce an automatic mode in the future where OVMF does the right thing based on what the guest supports. Gerd floated the idea of tying it to ExitBootServices as well, but not sure there's a solid plan on what to do here yet.
Quoted 6 lines
If that's accurate, it seems like the only 'safe' option is to disable it via #1 (accept-all), and then when #2 comes along, compile OVMF to just Do The Right Thing.
Users who know their VMs implement lazy acceptance can force it on via accept-all OVMF compile option.
Thanks for this Mike! I will bring this to the EDK2 community.
The issue for us is our users use a GCE built EDK2 not their own compiled version so they don't have the choice. Reading the Azure docs it seems the same for them, and for AWS so I don't know how often customers actually get to bring their own firmware.
Quoted 2 lines
-Mike
Quoted 60 lines · Peter Gonda
Quoted 35 lines · Kirill A. Shutemov
UEFI Specification version 2.9 introduces the concept of memory
acceptance: some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, requiring memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific for the Virtual Machine platform.Accepting memory is costly and it makes VMM allocate memory for the accepted guest physical address range. It's better to postpone memory acceptance until memory is needed. It lowers boot time and reduces memory overhead.
The kernel needs to know what memory has been accepted. Firmware communicates this information via memory map: a new memory type --
EFI_UNACCEPTED_MEMORY-- indicates such memory.Range-based tracking works fine for firmware, but it gets bulky for the kernel: e820 has to be modified on every page acceptance. It leads to table fragmentation, but there's a limited number of entries in the e820 table
Another option is to mark such memory as usable in e820 and track if the range has been accepted in a bitmap. One bit in the bitmap represents 2MiB in the address space: one 4k page is enough to track 64GiB or physical address space.
In the worst-case scenario -- a huge hole in the middle of the address space -- It needs 256MiB to handle 4PiB of the address space.
Any unaccepted memory that is not aligned to 2M gets accepted upfront.
The approach lowers boot time substantially. Boot to shell is ~2.5x faster for 4G TDX VM and ~4x faster for 64G.
Quoted 7 lines
TDX-specific code isolated from the core of unaccepted memory support. It supposed to help to plug-in different implementation of unaccepted memory such as SEV-SNP.
The tree can be found here:
https://github.com/intel/tdx.git guest-unaccepted-memory
Hi Kirill,
I have a couple questions about this feature mainly about how cloud customers can use this, I assume since this is a confidential compute feature a large number of the users of these patches will be cloud customers using TDX and SNP. One issue I see with these patches is how do we as a cloud provider know whether a customer's linux image supports this feature, if the image doesn't have these patches UEFI needs to fully validate the memory, if the image does we can use this new protocol. In GCE we supply our VMs with a version of the EDK2 FW and the customer doesn't input into which UEFI we run, as far as I can tell from the Azure SNP VM documentation it seems very similar. We need to somehow tell our UEFI in the VM what to do based on the image. The current way I can see to solve this issue would be to have our customers give us metadata about their VM's image but this seems kinda
Quoted 8 lines
burdensome on our customers (I assume we'll have more features which both UEFI and kernel need to both support inorder to be turned on like this one) and error-prone, if a customer incorrectly labels their image it may fail to boot.. Has there been any discussion about how to solve this? My naive thoughts were what if UEFI and Kernel had some sort of feature negotiation. Maybe that could happen via an extension to exit boot services or a UEFI runtime driver, I'm not sure what's best here just some ideas.
Just as an idea, we can put info into UTS_VERSION which can be read from the built bzImage. We have info on SMP and preeption there already.
Patch below does this:
$ file arch/x86/boot/bzImage arch/x86/boot/bzImage: Linux kernel x86 boot executable bzImage, version 5.19.0-rc3-00016-g2f6aa48e28d9-dirty (kas@box) #2300 SMP PREEMPT_DYNAMIC UNACCEPTED_MEMORY Mon Jun 27 14:23:04 , RO-rootFS, swap_dev 0XC, Normal VGA
Note UNACCEPTED_MEMORY in the output.
Probably we want to have there info on which flavour of unaccepted memory is supported (TDX/SNP/whatever). It is a bit more tricky.
Any opinion?
init/Makefile +1 -1
@@ -32,7 +32,7 @@ quiet_cmd_compile.h = CHK $@ $(CONFIG_SHELL) $(srctree)/scripts/mkcompile_h $@ \ "$(UTS_MACHINE)" "$(CONFIG_SMP)" "$(CONFIG_PREEMPT_BUILD)" \ "$(CONFIG_PREEMPT_DYNAMIC)" "$(CONFIG_PREEMPT_RT)" \ - "$(CONFIG_CC_VERSION_TEXT)" "$(LD)" + "$(CONFIG_UNACCEPTED_MEMORY)" "$(CONFIG_CC_VERSION_TEXT)" "$(LD)" include/generated/compile.h: FORCE $(call cmd,compile.h)
scripts/mkcompile_h +7 -2
@@ -7,8 +7,9 @@ SMP=$3 PREEMPT=$4 PREEMPT_DYNAMIC=$5 PREEMPT_RT=$6 -CC_VERSION="$7" -LD=$8 +UNACCEPTED_MEMORY=$7 +CC_VERSION="$8" +LD=$9 # Do not expand names set -f
@@ -51,6 +52,10 @@ elif [ -n "$PREEMPT" ] ; then CONFIG_FLAGS="$CONFIG_FLAGS PREEMPT" fi +if [ -n "$UNACCEPTED_MEMORY" ] ; then + CONFIG_FLAGS="$CONFIG_FLAGS UNACCEPTED_MEMORY" +fi + # Truncate to maximum length UTS_LEN=64 UTS_VERSION="$(echo $UTS_VERSION $CONFIG_FLAGS $TIMESTAMP | cut -b -$UTS_LEN)"
Signature · 1 line
Kirill A. Shutemov
Ard Biesheuvel
maintainerQuoted 71 lines · Kirill A. Shutemov
Quoted 60 lines · Peter Gonda
Quoted 35 lines · Kirill A. Shutemov
UEFI Specification version 2.9 introduces the concept of memory
acceptance: some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, requiring memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific for the Virtual Machine platform.Accepting memory is costly and it makes VMM allocate memory for the accepted guest physical address range. It's better to postpone memory acceptance until memory is needed. It lowers boot time and reduces memory overhead.
The kernel needs to know what memory has been accepted. Firmware communicates this information via memory map: a new memory type --
EFI_UNACCEPTED_MEMORY-- indicates such memory.Range-based tracking works fine for firmware, but it gets bulky for the kernel: e820 has to be modified on every page acceptance. It leads to table fragmentation, but there's a limited number of entries in the e820 table
Another option is to mark such memory as usable in e820 and track if the range has been accepted in a bitmap. One bit in the bitmap represents 2MiB in the address space: one 4k page is enough to track 64GiB or physical address space.
In the worst-case scenario -- a huge hole in the middle of the address space -- It needs 256MiB to handle 4PiB of the address space.
Any unaccepted memory that is not aligned to 2M gets accepted upfront.
The approach lowers boot time substantially. Boot to shell is ~2.5x faster for 4G TDX VM and ~4x faster for 64G.
Quoted 7 lines
TDX-specific code isolated from the core of unaccepted memory support. It supposed to help to plug-in different implementation of unaccepted memory such as SEV-SNP.
The tree can be found here:
https://github.com/intel/tdx.git guest-unaccepted-memory
Hi Kirill,
I have a couple questions about this feature mainly about how cloud customers can use this, I assume since this is a confidential compute feature a large number of the users of these patches will be cloud customers using TDX and SNP. One issue I see with these patches is how do we as a cloud provider know whether a customer's linux image supports this feature, if the image doesn't have these patches UEFI needs to fully validate the memory, if the image does we can use this new protocol. In GCE we supply our VMs with a version of the EDK2 FW and the customer doesn't input into which UEFI we run, as far as I can tell from the Azure SNP VM documentation it seems very similar. We need to somehow tell our UEFI in the VM what to do based on the image. The current way I can see to solve this issue would be to have our customers give us metadata about their VM's image but this seems kinda
Quoted 8 lines
burdensome on our customers (I assume we'll have more features which both UEFI and kernel need to both support inorder to be turned on like this one) and error-prone, if a customer incorrectly labels their image it may fail to boot.. Has there been any discussion about how to solve this? My naive thoughts were what if UEFI and Kernel had some sort of feature negotiation. Maybe that could happen via an extension to exit boot services or a UEFI runtime driver, I'm not sure what's best here just some ideas.
Quoted 3 lines
Just as an idea, we can put info into
UTS_VERSIONwhich can be read from the built bzImage. We have info on SMP and preeption there already.
Instead of hacking this into the binary, couldn't we define a protocol that the kernel will call from the EFI stub (before EBS()) to identify itself as an image that understands unaccepted memory, and knows how to deal with it?
That way, the firmware can accept all the memory on behalf of the OS at ExitBootServices() time, unless the OS has indicated there is no need to do so.
Quoted 54 lines
Patch below does this:
$ file arch/x86/boot/bzImage arch/x86/boot/bzImage: Linux kernel x86 boot executable bzImage, version 5.19.0-rc3-00016-g2f6aa48e28d9-dirty (kas@box) #2300 SMP PREEMPT_DYNAMIC UNACCEPTED_MEMORY Mon Jun 27 14:23:04 , RO-rootFS, swap_dev 0XC, Normal VGANote
UNACCEPTED_MEMORYin the output.Probably we want to have there info on which flavour of unaccepted memory is supported (TDX/SNP/whatever). It is a bit more tricky.
Any opinion?
init/Makefile +1
-1@@ -32,7 +32,7 @@ quiet_cmd_compile.h = CHK $@ $(CONFIG_SHELL) $(srctree)/scripts/mkcompile_h $@ \ "$(UTS_MACHINE)" "$(CONFIG_SMP)" "$(CONFIG_PREEMPT_BUILD)" \ "$(CONFIG_PREEMPT_DYNAMIC)" "$(CONFIG_PREEMPT_RT)" \ - "$(CONFIG_CC_VERSION_TEXT)" "$(LD)" + "$(CONFIG_UNACCEPTED_MEMORY)" "$(CONFIG_CC_VERSION_TEXT)" "$(LD)" include/generated/compile.h: FORCE $(call cmd,compile.h)scripts/mkcompile_h +7
-3@@ -7,8 +7,9 @@ SMP=$3 PREEMPT=$4 PREEMPT_DYNAMIC=$5 PREEMPT_RT=$6 -CC_VERSION="$7" -LD=$8 +UNACCEPTED_MEMORY=$7 +CC_VERSION="$8" +LD=$9 # Do not expand names set -f@@ -51,6 +52,10 @@ elif [ -n "$PREEMPT" ] ; then CONFIG_FLAGS="$CONFIG_FLAGS PREEMPT" fi +if [ -n "$UNACCEPTED_MEMORY" ] ; then + CONFIG_FLAGS="$CONFIG_FLAGS UNACCEPTED_MEMORY" +fi + # Truncate to maximum length UTS_LEN=64 UTS_VERSION="$(echo $UTS_VERSION $CONFIG_FLAGS $TIMESTAMP | cut -b -$UTS_LEN)" -- Kirill A. Shutemov
Quoted 77 lines · Ard Biesheuvel
Quoted 71 lines · Kirill A. Shutemov
Quoted 60 lines · Peter Gonda
Quoted 35 lines · Kirill A. Shutemov
UEFI Specification version 2.9 introduces the concept of memory
acceptance: some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, requiring memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific for the Virtual Machine platform.Accepting memory is costly and it makes VMM allocate memory for the accepted guest physical address range. It's better to postpone memory acceptance until memory is needed. It lowers boot time and reduces memory overhead.
The kernel needs to know what memory has been accepted. Firmware communicates this information via memory map: a new memory type --
EFI_UNACCEPTED_MEMORY-- indicates such memory.Range-based tracking works fine for firmware, but it gets bulky for the kernel: e820 has to be modified on every page acceptance. It leads to table fragmentation, but there's a limited number of entries in the e820 table
Another option is to mark such memory as usable in e820 and track if the range has been accepted in a bitmap. One bit in the bitmap represents 2MiB in the address space: one 4k page is enough to track 64GiB or physical address space.
In the worst-case scenario -- a huge hole in the middle of the address space -- It needs 256MiB to handle 4PiB of the address space.
Any unaccepted memory that is not aligned to 2M gets accepted upfront.
The approach lowers boot time substantially. Boot to shell is ~2.5x faster for 4G TDX VM and ~4x faster for 64G.
Quoted 7 lines
TDX-specific code isolated from the core of unaccepted memory support. It supposed to help to plug-in different implementation of unaccepted memory such as SEV-SNP.
The tree can be found here:
https://github.com/intel/tdx.git guest-unaccepted-memory
Hi Kirill,
I have a couple questions about this feature mainly about how cloud customers can use this, I assume since this is a confidential compute feature a large number of the users of these patches will be cloud customers using TDX and SNP. One issue I see with these patches is how do we as a cloud provider know whether a customer's linux image supports this feature, if the image doesn't have these patches UEFI needs to fully validate the memory, if the image does we can use this new protocol. In GCE we supply our VMs with a version of the EDK2 FW and the customer doesn't input into which UEFI we run, as far as I can tell from the Azure SNP VM documentation it seems very similar. We need to somehow tell our UEFI in the VM what to do based on the image. The current way I can see to solve this issue would be to have our customers give us metadata about their VM's image but this seems kinda
Quoted 8 lines
burdensome on our customers (I assume we'll have more features which both UEFI and kernel need to both support inorder to be turned on like this one) and error-prone, if a customer incorrectly labels their image it may fail to boot.. Has there been any discussion about how to solve this? My naive thoughts were what if UEFI and Kernel had some sort of feature negotiation. Maybe that could happen via an extension to exit boot services or a UEFI runtime driver, I'm not sure what's best here just some ideas.
Quoted 3 lines
Just as an idea, we can put info into
UTS_VERSIONwhich can be read from the built bzImage. We have info on SMP and preeption there already.
Quoted 8 lines
Instead of hacking this into the binary, couldn't we define a protocol that the kernel will call from the EFI stub (before
EBS()) to identify itself as an image that understands unaccepted memory, and knows how to deal with it?That way, the firmware can accept all the memory on behalf of the OS at
ExitBootServices()time, unless the OS has indicated there is no need to do so.
I agree it would be better. But I think it would require change to EFI spec, no?
Signature · 1 line
Kirill A. Shutemov
Quoted 84 lines · Kirill A. Shutemov
Quoted 82 lines · Ard Biesheuvel
Quoted 71 lines · Kirill A. Shutemov
Quoted 60 lines · Peter Gonda
Quoted 35 lines · Kirill A. Shutemov
UEFI Specification version 2.9 introduces the concept of memory
acceptance: some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, requiring memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific for the Virtual Machine platform.Accepting memory is costly and it makes VMM allocate memory for the accepted guest physical address range. It's better to postpone memory acceptance until memory is needed. It lowers boot time and reduces memory overhead.
The kernel needs to know what memory has been accepted. Firmware communicates this information via memory map: a new memory type --
EFI_UNACCEPTED_MEMORY-- indicates such memory.Range-based tracking works fine for firmware, but it gets bulky for the kernel: e820 has to be modified on every page acceptance. It leads to table fragmentation, but there's a limited number of entries in the e820 table
Another option is to mark such memory as usable in e820 and track if the range has been accepted in a bitmap. One bit in the bitmap represents 2MiB in the address space: one 4k page is enough to track 64GiB or physical address space.
In the worst-case scenario -- a huge hole in the middle of the address space -- It needs 256MiB to handle 4PiB of the address space.
Any unaccepted memory that is not aligned to 2M gets accepted upfront.
The approach lowers boot time substantially. Boot to shell is ~2.5x faster for 4G TDX VM and ~4x faster for 64G.
Quoted 7 lines
TDX-specific code isolated from the core of unaccepted memory support. It supposed to help to plug-in different implementation of unaccepted memory such as SEV-SNP.
The tree can be found here:
https://github.com/intel/tdx.git guest-unaccepted-memory
Hi Kirill,
I have a couple questions about this feature mainly about how cloud customers can use this, I assume since this is a confidential compute feature a large number of the users of these patches will be cloud customers using TDX and SNP. One issue I see with these patches is how do we as a cloud provider know whether a customer's linux image supports this feature, if the image doesn't have these patches UEFI needs to fully validate the memory, if the image does we can use this new protocol. In GCE we supply our VMs with a version of the EDK2 FW and the customer doesn't input into which UEFI we run, as far as I can tell from the Azure SNP VM documentation it seems very similar. We need to somehow tell our UEFI in the VM what to do based on the image. The current way I can see to solve this issue would be to have our customers give us metadata about their VM's image but this seems kinda
Quoted 8 lines
burdensome on our customers (I assume we'll have more features which both UEFI and kernel need to both support inorder to be turned on like this one) and error-prone, if a customer incorrectly labels their image it may fail to boot.. Has there been any discussion about how to solve this? My naive thoughts were what if UEFI and Kernel had some sort of feature negotiation. Maybe that could happen via an extension to exit boot services or a UEFI runtime driver, I'm not sure what's best here just some ideas.
Quoted 3 lines
Just as an idea, we can put info into
UTS_VERSIONwhich can be read from the built bzImage. We have info on SMP and preeption there already.Instead of hacking this into the binary, couldn't we define a protocol that the kernel will call from the EFI stub (before
EBS()) to identify itself as an image that understands unaccepted memory, and knows how to deal with it?
Quoted 6 lines
Quoted 3 lines
That way, the firmware can accept all the memory on behalf of the OS at
ExitBootServices()time, unless the OS has indicated there is no need to do so.I agree it would be better. But I think it would require change to EFI spec, no?
Could this somehow be amended on to the UEFI Specification version 2.9 change which added all of the unaccepted memory features?
Quoted 3 lines
--
Kirill A. Shutemov
Ard Biesheuvel
maintainerQuoted 91 lines · Peter Gonda
Quoted 88 lines · Kirill A. Shutemov
Quoted 85 lines · Ard Biesheuvel
Quoted 71 lines · Kirill A. Shutemov
Quoted 60 lines · Peter Gonda
Quoted 35 lines · Kirill A. Shutemov
UEFI Specification version 2.9 introduces the concept of memory
acceptance: some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, requiring memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific for the Virtual Machine platform.Accepting memory is costly and it makes VMM allocate memory for the accepted guest physical address range. It's better to postpone memory acceptance until memory is needed. It lowers boot time and reduces memory overhead.
The kernel needs to know what memory has been accepted. Firmware communicates this information via memory map: a new memory type --
EFI_UNACCEPTED_MEMORY-- indicates such memory.Range-based tracking works fine for firmware, but it gets bulky for the kernel: e820 has to be modified on every page acceptance. It leads to table fragmentation, but there's a limited number of entries in the e820 table
Another option is to mark such memory as usable in e820 and track if the range has been accepted in a bitmap. One bit in the bitmap represents 2MiB in the address space: one 4k page is enough to track 64GiB or physical address space.
In the worst-case scenario -- a huge hole in the middle of the address space -- It needs 256MiB to handle 4PiB of the address space.
Any unaccepted memory that is not aligned to 2M gets accepted upfront.
The approach lowers boot time substantially. Boot to shell is ~2.5x faster for 4G TDX VM and ~4x faster for 64G.
Quoted 7 lines
TDX-specific code isolated from the core of unaccepted memory support. It supposed to help to plug-in different implementation of unaccepted memory such as SEV-SNP.
The tree can be found here:
https://github.com/intel/tdx.git guest-unaccepted-memory
Hi Kirill,
I have a couple questions about this feature mainly about how cloud customers can use this, I assume since this is a confidential compute feature a large number of the users of these patches will be cloud customers using TDX and SNP. One issue I see with these patches is how do we as a cloud provider know whether a customer's linux image supports this feature, if the image doesn't have these patches UEFI needs to fully validate the memory, if the image does we can use this new protocol. In GCE we supply our VMs with a version of the EDK2 FW and the customer doesn't input into which UEFI we run, as far as I can tell from the Azure SNP VM documentation it seems very similar. We need to somehow tell our UEFI in the VM what to do based on the image. The current way I can see to solve this issue would be to have our customers give us metadata about their VM's image but this seems kinda
Quoted 8 lines
burdensome on our customers (I assume we'll have more features which both UEFI and kernel need to both support inorder to be turned on like this one) and error-prone, if a customer incorrectly labels their image it may fail to boot.. Has there been any discussion about how to solve this? My naive thoughts were what if UEFI and Kernel had some sort of feature negotiation. Maybe that could happen via an extension to exit boot services or a UEFI runtime driver, I'm not sure what's best here just some ideas.
Quoted 3 lines
Just as an idea, we can put info into
UTS_VERSIONwhich can be read from the built bzImage. We have info on SMP and preeption there already.Instead of hacking this into the binary, couldn't we define a protocol that the kernel will call from the EFI stub (before
EBS()) to identify itself as an image that understands unaccepted memory, and knows how to deal with it?That way, the firmware can accept all the memory on behalf of the OS at
ExitBootServices()time, unless the OS has indicated there is no need to do so.
Quoted 6 lines
Quoted 2 lines
I agree it would be better. But I think it would require change to EFI spec, no?
Could this somehow be amended on to the UEFI Specification version 2.9 change which added all of the unaccepted memory features?
Why would this need a change in the EFI spec? Not every EFI protocol needs to be in the spec.
Quoted 17 lines · Ard Biesheuvel
Quoted 17 lines
Quoted 16 lines
Quoted 5 lines
Quoted 4 lines
Just as an idea, we can put info into
UTS_VERSIONwhich can be read from the built bzImage. We have info on SMP and preeption there already.Quoted 8 lines
Instead of hacking this into the binary, couldn't we define a protocol that the kernel will call from the EFI stub (before
EBS()) to identify itself as an image that understands unaccepted memory, and knows how to deal with it?That way, the firmware can accept all the memory on behalf of the OS at
ExitBootServices()time, unless the OS has indicated there is no need to do so.I agree it would be better. But I think it would require change to EFI spec, no?
Quoted 6 lines
Quoted 3 lines
Could this somehow be amended on to the UEFI Specification version 2.9 change which added all of the unaccepted memory features?
Why would this need a change in the EFI spec? Not every EFI protocol needs to be in the spec.
My EFI knowledge is shallow. Do we do this in other cases?
Signature · 1 line
Kirill A. Shutemov
Ard Biesheuvel
maintainerQuoted 19 lines · Kirill A. Shutemov
Quoted 17 lines · Ard Biesheuvel
Quoted 17 lines
Quoted 16 lines
Quoted 5 lines
Quoted 4 lines
Just as an idea, we can put info into
UTS_VERSIONwhich can be read from the built bzImage. We have info on SMP and preeption there already.Quoted 8 lines
Instead of hacking this into the binary, couldn't we define a protocol that the kernel will call from the EFI stub (before
EBS()) to identify itself as an image that understands unaccepted memory, and knows how to deal with it?That way, the firmware can accept all the memory on behalf of the OS at
ExitBootServices()time, unless the OS has indicated there is no need to do so.I agree it would be better. But I think it would require change to EFI spec, no?
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Could this somehow be amended on to the UEFI Specification version 2.9 change which added all of the unaccepted memory features?
Why would this need a change in the EFI spec? Not every EFI protocol needs to be in the spec.
My EFI knowledge is shallow. Do we do this in other cases?
The E in EFI means 'extensible' and the whole design of a protocol database using GUIDs as identifiers (which will not collide and therefore need no a priori coordination when defining them) is intended to allow extensions to be defined and implemented in a distributed manner.
Of course, it would be fantastic if we can converge on a protocol that all flavors of confidential compute can use, across different OSes, so it is generally good if a protocol is defined in *some* shared specification. But this doesn't have to be the EFI spec.
Quoted 37 lines · Ard Biesheuvel
Quoted 19 lines · Kirill A. Shutemov
Quoted 17 lines · Ard Biesheuvel
Quoted 17 lines
Quoted 16 lines
Quoted 5 lines
Quoted 4 lines
Just as an idea, we can put info into
UTS_VERSIONwhich can be read from the built bzImage. We have info on SMP and preeption there already.Quoted 8 lines
Instead of hacking this into the binary, couldn't we define a protocol that the kernel will call from the EFI stub (before
EBS()) to identify itself as an image that understands unaccepted memory, and knows how to deal with it?That way, the firmware can accept all the memory on behalf of the OS at
ExitBootServices()time, unless the OS has indicated there is no need to do so.I agree it would be better. But I think it would require change to EFI spec, no?
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Could this somehow be amended on to the UEFI Specification version 2.9 change which added all of the unaccepted memory features?
Why would this need a change in the EFI spec? Not every EFI protocol needs to be in the spec.
My EFI knowledge is shallow. Do we do this in other cases?
The E in EFI means 'extensible' and the whole design of a protocol database using GUIDs as identifiers (which will not collide and therefore need no a priori coordination when defining them) is intended to allow extensions to be defined and implemented in a distributed manner.
Quoted 4 lines
Of course, it would be fantastic if we can converge on a protocol that all flavors of confidential compute can use, across different OSes, so it is generally good if a protocol is defined in *some* shared specification. But this doesn't have to be the EFI spec.
I've talked with our firmware expert today and I think we have a problem with the approach when kernel declaries support of unaccepted memory.
This apporach doesn't work if we include bootloader into the picture: if EBS() called by bootloader we still cannot know if target kernel supports unaccepted memory and we return to the square 1.
I think we should make it obvious from a kernel image if it supports unaccepted memory (with UTS_VERSION or other way).
Any comments?
Signature · 1 line
Kirill A. Shutemov
Quoted 3 lines
I've talked with our firmware expert today and I think we have a problem with the approach when kernel declaries support of unaccepted memory.
Is this Jiewen Yao? I've been trying to design the UEFI spec change with him. The bootloader problem he commented with this morning was something I wasn't fully considering.
Quoted 8 lines
This apporach doesn't work if we include bootloader into the picture: if
EBS()called by bootloader we still cannot know if target kernel supports unaccepted memory and we return to the square 1.I think we should make it obvious from a kernel image if it supports unaccepted memory (with
UTS_VERSIONor other way).Any comments?
Is this binary parsing trick already used in EDK2? If not, I wouldn't want to introduce an ABI-solidifying requirement like that.
A bit more cumbersome, but more flexible way to enable the feature is an idea I had in a meeting today:
Make unaccepted memory support a feature-enabling EFI driver installed to the EFI system partition.
* The first time you boot (setup mode), you install an EFI driver that just sets a feature Pcd to true (using a custom protocol as Ard had suggested above).
* The second time you boot, if the feature Pcd is true, then the UEFI is free to not accept memory and use the unaccepted memory type. The bootloader will run after unaccepted memory has been allowed already, so there is no accept-all event.
The default behavior will be to accept all memory when GetMemoryMap is called unless the feature pcd is set to true.
We can then say this driver isn't needed once some new generation of this technology comes along and we can require unaccepted memory support as part of that technology's baseline, or we manage to update the UEFI spec to have GetMemoryMapEx which has unaccepted memory support baked in and the bootloaders all know to use it.
The cloud experience will be, "is boot slow? Install this EFI driver from the cloud service provider" to tell the UEFI to enable unaccepted memory.
Signature · 1 line
-Dionna Glaze, PhD (she/her)
Quoted 3 lines
Quoted 3 lines
I think we should make it obvious from a kernel image if it supports unaccepted memory (with
UTS_VERSIONor other way).
Something I didn't address in my previous email: how would the UEFI know where the kernel is to parse this UTS_VERSION out when it's booting a bootloader before Linux gets booted?
Signature · 1 line
-Dionna Glaze, PhD (she/her)
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
I think we should make it obvious from a kernel image if it supports unaccepted memory (with
UTS_VERSIONor other way).Something I didn't address in my previous email: how would the UEFI know where the kernel is to parse this
UTS_VERSIONout when it's booting a bootloader before Linux gets booted?
How about instead of the limited resource of UTS_VERSION, we add a SETUP_BOOT_FEATURES enum for setup_data in the boot header? That would be easier to parse out and more extensible in the future.
https://www.kernel.org/doc/html/latest/x86/boot.html?highlight=boot
This can contain a bitmap of a number of features that we currently need manual tagging for, such as SEV guest support, SEV-SNP guest support, TDX guest support, and (CONFIG_UNACCEPTED_MEMORY, TDX) or (CONFIG_UNACCEPTED_MEMORY, SEV-SNP).
The VMM, UEFI, or boot loader can read these from the images/kernels and have the appropriate behavior.
Signature · 1 line
-Dionna Glaze, PhD (she/her)
Borislav Petkov
maintainerQuoted 11 lines · Dionna Amalie Glaze
How about instead of the limited resource of
UTS_VERSION, we add aSETUP_BOOT_FEATURESenum forsetup_datain the boot header? That would be easier to parse out and more extensible in the future.
https://www.kernel.org/doc/html/latest/x86/boot.html?highlight=bootThis can contain a bitmap of a number of features that we currently need manual tagging for, such as SEV guest support, SEV-SNP guest support, TDX guest support, and (
CONFIG_UNACCEPTED_MEMORY, TDX) or (CONFIG_UNACCEPTED_MEMORY, SEV-SNP).
The VMM, UEFI, or boot loader can read these from the images/kernels and have the appropriate behavior.
I think for stuff like that you want loadflags or xloadflags in the setup header.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerQuoted 14 lines · Borislav Petkov
Quoted 11 lines · Dionna Amalie Glaze
How about instead of the limited resource of
UTS_VERSION, we add aSETUP_BOOT_FEATURESenum forsetup_datain the boot header? That would be easier to parse out and more extensible in the future.
https://www.kernel.org/doc/html/latest/x86/boot.html?highlight=bootThis can contain a bitmap of a number of features that we currently need manual tagging for, such as SEV guest support, SEV-SNP guest support, TDX guest support, and (
CONFIG_UNACCEPTED_MEMORY, TDX) or (CONFIG_UNACCEPTED_MEMORY, SEV-SNP).
The VMM, UEFI, or boot loader can read these from the images/kernels and have the appropriate behavior.
Quoted 3 lines
I think for stuff like that you want loadflags or xloadflags in the setup header.
Please, no. Let's not invent Linux/x86 specific hacks to infer whether or not the kernel is capable of accepting memory when it is perfectly capable of telling us directly. We will surely need something analogous on other architectures in the future as well, so the setup header is definitely not the right place for this.
The 'bootloader that calls EBS()' case does not apply to Linux, and given that we are talking specifically about confidential computing VMs here, we can afford to be normative and define something generic that works well for us.
So let's define a way for the EFI stub to signal to the firmware (before EBS()) that it will take control of accepting memory. The 'bootloader that calls EBS()' case can invent something along the lines of what has been proposed in this thread to infer the capabilities of the kernel (and decide what to signal to the firmware). But we have no need for this additional complexity on Linux.
Borislav Petkov
maintainerQuoted 7 lines · Ard Biesheuvel
So let's define a way for the EFI stub to signal to the firmware (before
EBS()) that it will take control of accepting memory. The 'bootloader that callsEBS()' case can invent something along the lines of what has been proposed in this thread to infer the capabilities of the kernel (and decide what to signal to the firmware). But we have no need for this additional complexity on Linux.
To tell you the truth, I've been perusing this thread from the sidelines and am wondering why does this need this special dance at all?
If EFI takes control of accepting memory, then when the guest kernel boots, it'll find all memory accepted and not do anything.
If EFI doesn't accept memory, then the guest kernel will boot and do the accepting itself.
So either I'm missing something or we're overengineering this for no good reason...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 11 lines · Borislav Petkov
Quoted 7 lines · Ard Biesheuvel
So let's define a way for the EFI stub to signal to the firmware (before
EBS()) that it will take control of accepting memory. The 'bootloader that callsEBS()' case can invent something along the lines of what has been proposed in this thread to infer the capabilities of the kernel (and decide what to signal to the firmware). But we have no need for this additional complexity on Linux.To tell you the truth, I've been perusing this thread from the sidelines and am wondering why does this need this special dance at all?
Quoted 8 lines
If EFI takes control of accepting memory, then when the guest kernel boots, it'll find all memory accepted and not do anything.
If EFI doesn't accept memory, then the guest kernel will boot and do the accepting itself.
So either I'm missing something or we're overengineering this for no good reason...
They're trying to design something that can (forever) handle guests that might not be able to accept memory. It's based on the idea that *something* needs to assume control and EFI doesn't have enough information to assume control.
I wish we didn't need all this complexity, though.
There are three entities that can influence how much memory is accepted:
1. The host
2. The guest firmware
3. The guest kernel (or bootloader or something after the firmware)
This whole thread is about how #2 and #3 talk to each other and make sure *someone* does it.
I kinda think we should just take the guest firmware out of the picture.
There are only going to be a few versions of the kernel that can boot
under TDX (or SEV-SNP) and *can't* handle unaccepted memory. It seems a bit silly to design this whole interface for a few versions of the OS that TDX folks tell me can't be used anyway.
I think we should just say if you want to run an OS that doesn't have unaccepted memory support, you can either:
1. Deal with that at the host level configuration
2. Boot some intermediate thing like a bootloader that does acceptance before running the stupid^Wunenlightended OS
3. Live with the 4GB of pre-accepted memory you get with no OS work.
Yeah, this isn't convenient for some hosts. But, really, this is preferable to doing an EFI/OS dance until the end of time.
Borislav Petkov
maintainerQuoted 2 lines · Dave Hansen
They're trying to design something that can (forever) handle guests that might not be able to accept memory.
Wait, what?
If you can't modify those guests to teach them to accept memory, how do you add TDX or SNP guest support to them?
I.e., you need to modify the guests and then you can add memory acceptance. Basically, your point below...
Quoted 23 lines
It's based on the idea that *something* needs to assume control and EFI doesn't have enough information to assume control.
I wish we didn't need all this complexity, though.
There are three entities that can influence how much memory is accepted:
1. The host
2. The guest firmware
3. The guest kernel (or bootloader or something after the firmware)This whole thread is about how #2 and #3 talk to each other and make sure *someone* does it.
I kinda think we should just take the guest firmware out of the picture.
There are only going to be a few versions of the kernel that can boot
under TDX (or SEV-SNP) and *can't* handle unaccepted memory. It seems a bit silly to design this whole interface for a few versions of the OS that TDX folks tell me can't be used anyway.I think we should just say if you want to run an OS that doesn't have unaccepted memory support, you can either:
Quoted 7 lines
1. Deal with that at the host level configuration
2. Boot some intermediate thing like a bootloader that does acceptance before running the stupid^Wunenlightended OS
3. Live with the 4GB of pre-accepted memory you get with no OS work.Yeah, this isn't convenient for some hosts. But, really, this is preferable to doing an EFI/OS dance until the end of time.
Ack. Definitely.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 40 lines · Borislav Petkov
Quoted 2 lines · Dave Hansen
They're trying to design something that can (forever) handle guests that might not be able to accept memory.
Wait, what?
If you can't modify those guests to teach them to accept memory, how do you add TDX or SNP guest support to them?
I.e., you need to modify the guests and then you can add memory acceptance. Basically, your point below...
Quoted 28 lines
It's based on the idea that *something* needs to assume control and EFI doesn't have enough information to assume control.
I wish we didn't need all this complexity, though.
There are three entities that can influence how much memory is accepted:
1. The host
2. The guest firmware
3. The guest kernel (or bootloader or something after the firmware)This whole thread is about how #2 and #3 talk to each other and make sure *someone* does it.
I kinda think we should just take the guest firmware out of the picture.
There are only going to be a few versions of the kernel that can boot
under TDX (or SEV-SNP) and *can't* handle unaccepted memory. It seems a bit silly to design this whole interface for a few versions of the OS that TDX folks tell me can't be used anyway.I think we should just say if you want to run an OS that doesn't have unaccepted memory support, you can either:
1. Deal with that at the host level configuration
2. Boot some intermediate thing like a bootloader that does acceptance before running the stupid^Wunenlightended OS
3. Live with the 4GB of pre-accepted memory you get with no OS work.
Quoted 4 lines
Quoted 2 lines
Yeah, this isn't convenient for some hosts. But, really, this is preferable to doing an EFI/OS dance until the end of time.
Ack. Definitely.
I like it too as it is no-code solution :P
Peter, I'm pretty sure unaccepted memory support hits upstream well before TDX get adopted widely in production. I think it is pretty reasonable to deal with it on host side in meanwhile.
Any objections?
Signature · 1 line
Kirill A. Shutemov
Quoted 7 lines · Borislav Petkov
Quoted 2 lines · Dave Hansen
They're trying to design something that can (forever) handle guests that might not be able to accept memory.
Wait, what?
If you can't modify those guests to teach them to accept memory, how do you add TDX or SNP guest support to them?
Mainline today, for instance, doesn't have unaccepted memory support for TDX or SEV-SNP guests. But, they both still boot fine because folks either configure it on the host side not to *have* any unaccepted memory. Or, they just live with the small (4GB??) amount of pre-accepted memory, which is fine for testing things.
Quoted 6 lines · Dave Hansen
Quoted 5 lines · Borislav Petkov
Quoted 2 lines · Dave Hansen
They're trying to design something that can (forever) handle guests that might not be able to accept memory.
Wait, what?
Quoted 8 lines
Quoted 2 lines
If you can't modify those guests to teach them to accept memory, how do you add TDX or SNP guest support to them?
Mainline today, for instance, doesn't have unaccepted memory support for TDX or SEV-SNP guests. But, they both still boot fine because folks either configure it on the host side not to *have* any unaccepted memory. Or, they just live with the small (4GB??) amount of pre-accepted memory, which is fine for testing things.
Today, for SEV-SNP, OVMF accepts all of the memory in advance of booting the kernel.
Thanks,
Tom
Quoted 1 line
Quoted 7 lines · Dave Hansen
Quoted 5 lines · Borislav Petkov
Quoted 2 lines · Dave Hansen
They're trying to design something that can (forever) handle guests that might not be able to accept memory.
Wait, what?
Quoted 8 lines
Quoted 2 lines
If you can't modify those guests to teach them to accept memory, how do you add TDX or SNP guest support to them?
Mainline today, for instance, doesn't have unaccepted memory support for TDX or SEV-SNP guests. But, they both still boot fine because folks either configure it on the host side not to *have* any unaccepted memory. Or, they just live with the small (4GB??) amount of pre-accepted memory, which is fine for testing things.
For us (Google cloud), "1. Deal with that at the host level configuration" looks like:
https://cloud.google.com/compute/docs/images/create-delete-deprecate-private-images#guest-os-features
In other words, we have to tag images with "feature tags" to distinguish which images have kernels that support which features.
Part of the reason we need to do it this way is that we use a single guest firmware (i.e., guest UEFI) that lives outside of the image.
These feature tags are a mess to keep track of.
All that being said, I can totally see the upstream perspective being "not our problem". It's hard to argue with that :-).
A few more thoughts:
- If the guest-side patches weren't upstream before this patch set to handle unaccepted memory, you're all definitely right, that this isn't a real issue. (Maybe it still isn't...)
- Do we anticipate (many) more features for confidential compute in the future that require code in both the guest FW and guest kernel? If yes, then designing a FW-kernel feature negotiation could be useful beyond this situation.
- Dave's suggestion to "2. Boot some intermediate thing like a bootloader that does acceptance ..." is pretty clever! So if upstream thinks this FW-kernel negotiation is not a good direction, maybe we (Google) can pursue this idea to avoid introducing yet another tag on our images.
Thank you all for this discussion.
Thanks,
Marc
Borislav Petkov
maintainerQuoted 1 line · Marc Orr
These feature tags are a mess to keep track of.
Well, looking at those tags, it doesn't look like you'll stop using them anytime soon.
And once all the required SNP/TDX features are part of the guest image,
- including unaccepted memory - if anything, you'll have less tags.
:-)
Quoted 4 lines
- Do we anticipate (many) more features for confidential compute in the future that require code in both the guest FW and guest kernel? If yes, then designing a FW-kernel feature negotiation could be useful beyond this situation.
Good question.
Quoted 5 lines
- Dave's suggestion to "2. Boot some intermediate thing like a bootloader that does acceptance ..." is pretty clever! So if upstream thinks this FW-kernel negotiation is not a good direction, maybe we (Google) can pursue this idea to avoid introducing yet another tag on our images.
Are those tags really that nasty so that you guys are looking at upstream changes just to avoid them?
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 11 lines · Borislav Petkov
Quoted 1 line · Marc Orr
These feature tags are a mess to keep track of.
Well, looking at those tags, it doesn't look like you'll stop using them anytime soon.
And once all the required SNP/TDX features are part of the guest image,
- including unaccepted memory - if anything, you'll have less tags.:-)
Yeah, once all of the features are a part of the guest image AND any older images with SNP/TDX minus the features are deprecated. I agree.
Quoted 7 lines
Quoted 4 lines
- Do we anticipate (many) more features for confidential compute in the future that require code in both the guest FW and guest kernel? If yes, then designing a FW-kernel feature negotiation could be useful beyond this situation.
Good question.
Quoted 8 lines
Quoted 5 lines
- Dave's suggestion to "2. Boot some intermediate thing like a bootloader that does acceptance ..." is pretty clever! So if upstream thinks this FW-kernel negotiation is not a good direction, maybe we (Google) can pursue this idea to avoid introducing yet another tag on our images.
Are those tags really that nasty so that you guys are looking at upstream changes just to avoid them?
Generally, no. But the problem with tags is that distros tag their images wrong sometimes. And that leads to problems. For example, I just got a bug assigned to me yesterday about some ARM image tagged as SEV_CAPABLE. Oops. Lol :-). (Though, I'm pretty sure we won't try to boot an ARM image on a non-ARM host anyway; but it's still wrong...)
That being said, this lazy accept problem is sort of a special case, since it requires deploying code to the guest FW and the guest kernel. I'm still relatively new at all of this, but other than the SNP/TDX-enlightenment patches themselves, I haven't really seen any other examples of this. So that goes back to my previous question. Is this going to happen a lot more? If not, I can definitely see value in the argument to skip the complexity of the FW/kernel feature negotiation.
Another thing I thought of since my last reply, that's mostly an internal solution to this problem on our side: Going back to Dave's 10k-foot view of the different angles of how to solve this. For "1. Deal with that at the host level configuration", I'm thinking we could tag the images with their internal guest kernel version. For example, if an image has a 5.15 kernel, then we could have a KERNEL_5_15 tag. This would then allow us to have logic in the guest FW like:
if (guest_kernel_is_at_least(/*major=*/5, /*minor=*/15)
enable_lazy_accept = true;
One detail I actually missed in all of this, is how the guest image tag gets propagated into the guest FW in this approach. (Apologies for this, as that's a pretty big oversight on my part.) Dionna: Have you thought about this? Presumably this requires some sort of paravirt for the guest to ask the host. And for any paravirt interface, now we need to think about if it degrades the security of the confidential VMs. Though, using it to get the kernel version to decide whether or not to accept the memory within the guest UEFI or mark it as unaccepted seems fine from a security angle to me.
Also, tagging images with their underlying kernel versions still seems susceptible to mis-labeling. But this seems like it can be mostly "fixed" via automation (e.g., write a tool to boot the guest and ask it what it's kernel version is and use the result to attach the tag). Also, tagging the images with their kernel version seems like a much more general solution to these sorts of issues.
Thoughts?
Borislav Petkov
maintainerQuoted 5 lines · Marc Orr
Generally, no. But the problem with tags is that distros tag their images wrong sometimes. And that leads to problems. For example, I just got a bug assigned to me yesterday about some ARM image tagged as
SEV_CAPABLE. Oops. Lol :-). (Though, I'm pretty sure we won't try to boot an ARM image on a non-ARM host anyway; but it's still wrong...)
Yeah, even if, let it crash'n'burn - people will notice pretty quickly.
Quoted 6 lines
That being said, this lazy accept problem is sort of a special case, since it requires deploying code to the guest FW and the guest kernel. I'm still relatively new at all of this, but other than the SNP/TDX-enlightenment patches themselves, I haven't really seen any other examples of this. So that goes back to my previous question. Is this going to happen a lot more?
Good question.
Unfortunately, not even the architects of coco could give you an answer because, as you see yourself, those additional features like memory acceptance, live migration, etc keep changing - the whole coco thing is pretty much a moving target.
For example, if someone comes along and says, err, see, I have this live migration helper and that thing runs as an EFI executable and it is so much better...
Not saying it'll happen but it could. I hope you're catching my drift.
Quoted 11 lines
If not, I can definitely see value in the argument to skip the complexity of the FW/kernel feature negotiation.
Another thing I thought of since my last reply, that's mostly an internal solution to this problem on our side: Going back to Dave's 10k-foot view of the different angles of how to solve this. For "1. Deal with that at the host level configuration", I'm thinking we could tag the images with their internal guest kernel version. For example, if an image has a 5.15 kernel, then we could have a
KERNEL_5_15tag. This would then allow us to have logic in the guest FW like:
Quoted 2 lines
if (
guest_kernel_is_at_least(/*major=*/5, /*minor=*/15)enable_lazy_accept = true;
Well, I don't want to spoil your idea but imagine distros like SLE or others backport features into old kernels. All of a sudden 5.14 or older can do memory acceptance too. And then that version-based scheme falls apart.
So I'm guessing it would probably be better to explicitly tag distro images. Thing is, once all needed support gets in, you can drop the tags and simply say, you don't support those old images anymore and assume all required support is there and implicit...
Quoted 4 lines
Also, tagging images with their underlying kernel versions still seems susceptible to mis-labeling. But this seems like it can be mostly "fixed" via automation (e.g., write a tool to boot the guest and ask it what it's kernel version is and use the result to attach the tag).
I'll do you one better: boot the image and check for all required features and produce tags. Or do not accept the image as a possible coco image. And so on.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 5 lines · Marc Orr
- Dave's suggestion to "2. Boot some intermediate thing like a bootloader that does acceptance ..." is pretty clever! So if upstream thinks this FW-kernel negotiation is not a good direction, maybe we (Google) can pursue this idea to avoid introducing yet another tag on our images.
I'm obviously speaking only for myself here and not for "upstream" as a whole, but I clearly don't like the FW/kernel negotiation thing. It's a permanent pain in our necks to solve a very temporary problem.
Ard Biesheuvel
maintainerQuoted 11 lines · Dave Hansen
Quoted 5 lines · Marc Orr
- Dave's suggestion to "2. Boot some intermediate thing like a bootloader that does acceptance ..." is pretty clever! So if upstream thinks this FW-kernel negotiation is not a good direction, maybe we (Google) can pursue this idea to avoid introducing yet another tag on our images.
I'm obviously speaking only for myself here and not for "upstream" as a whole, but I clearly don't like the FW/kernel negotiation thing. It's a permanent pain in our necks to solve a very temporary problem.
EFI is basically our existing embodiment of this fw/kernel negotiation thing, and iff we need it, I have no objection to using it for this purpose, i.e., to allow the firmware to infer whether or not it should accept all available memory on behalf of the OS before exiting boot services. But if we don't need this, even better.
What I strongly object to is inventing a new bespoke way for the firmware to make inferences about the capabilities of the image by inspecting fields in the file representation of the image (which is not guaranteed by EFI to be identical to its in-memory representation, as, e.g., the PE/COFF header could be omitted by a loader without violating the spec)
As for the intermediate thing: yes, that would be a valuable thing to have in OVMF (and I will gladly take EDK2 patches that implement this). However, I'm not sure how you decide whether or not this thing should be active or not, doesn't that just move the problem around?
Quoted 8 lines
What I strongly object to is inventing a new bespoke way for the firmware to make inferences about the capabilities of the image by inspecting fields in the file representation of the image (which is not guaranteed by EFI to be identical to its in-memory representation, as, e.g., the PE/COFF header could be omitted by a loader without violating the spec)
Quoted 4 lines
As for the intermediate thing: yes, that would be a valuable thing to have in OVMF (and I will gladly take EDK2 patches that implement this). However, I'm not sure how you decide whether or not this thing should be active or not, doesn't that just move the problem around?
This does just move the problem around, but it makes correct behavior the default instead of silently ignoring most of the VM's memory and booting regularly. I have the driver mostly written to change the behavior to accept all by default unless a driver has been installed to set a particular boolean to make it not. Still that's yet another thing as you say.
I agree with everyone that this situation just stinks. "Can't you just boot it?" was asked before, and yes we can, but at the scale of a CSP managing anybody's image uploads, that not-insignificant cost has to be paid by someone. It's a hard problem to route the image to the right kind of machine that's expected to be able to run it... it's a big ol' mess.
One thing is for sure: these patches shouldn't be blocked by the "how do we detect it" question. I'm glad to see so much engagement with this problem, but I fear I might have delayed its progress towards a merge. I know AMD has a follow-up to add SEV-SNP accept_memory support to finish this all up.
I'll try to get the ear of all the distributions that are tracking towards providing SEV-SNP-supported images for CSPs to get them on the release that includes these patches. I'll also see about upstreaming that EFI driver and EDK2 changes in case there's a slip in the kernel release and we need this workaround.
--
-Dionna Glaze, PhD (she/her)
Quoted 13 lines · Ard Biesheuvel
Quoted 11 lines · Dave Hansen
Quoted 5 lines · Marc Orr
- Dave's suggestion to "2. Boot some intermediate thing like a bootloader that does acceptance ..." is pretty clever! So if upstream thinks this FW-kernel negotiation is not a good direction, maybe we (Google) can pursue this idea to avoid introducing yet another tag on our images.
I'm obviously speaking only for myself here and not for "upstream" as a whole, but I clearly don't like the FW/kernel negotiation thing. It's a permanent pain in our necks to solve a very temporary problem.
Quoted 5 lines
EFI is basically our existing embodiment of this fw/kernel negotiation thing, and iff we need it, I have no objection to using it for this purpose, i.e., to allow the firmware to infer whether or not it should accept all available memory on behalf of the OS before exiting boot services. But if we don't need this, even better.
FW/kernel negotiation does not work if there's a boot loader in the middle that does ExitBootServices(). By the time kernel can announce if it supports unaccepted memory there's nobody to announce to.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Ard Biesheuvel
maintainerQuoted 21 lines · Kirill A. Shutemov
Quoted 13 lines · Ard Biesheuvel
Quoted 11 lines · Dave Hansen
Quoted 5 lines · Marc Orr
- Dave's suggestion to "2. Boot some intermediate thing like a bootloader that does acceptance ..." is pretty clever! So if upstream thinks this FW-kernel negotiation is not a good direction, maybe we (Google) can pursue this idea to avoid introducing yet another tag on our images.
I'm obviously speaking only for myself here and not for "upstream" as a whole, but I clearly don't like the FW/kernel negotiation thing. It's a permanent pain in our necks to solve a very temporary problem.
Quoted 5 lines
EFI is basically our existing embodiment of this fw/kernel negotiation thing, and iff we need it, I have no objection to using it for this purpose, i.e., to allow the firmware to infer whether or not it should accept all available memory on behalf of the OS before exiting boot services. But if we don't need this, even better.
Quoted 4 lines
FW/kernel negotiation does not work if there's a boot loader in the middle that does
ExitBootServices(). By the time kernel can announce if it supports unaccepted memory there's nobody to announce to.
Why would you want to support such bootloaders for TDX anyway? TDX heavily relies on measured boot abstractions and other things that are heavily tied to firmware.
Quoted 23 lines · Ard Biesheuvel
Quoted 21 lines · Kirill A. Shutemov
Quoted 18 lines · Ard Biesheuvel
Quoted 11 lines · Dave Hansen
Quoted 5 lines · Marc Orr
- Dave's suggestion to "2. Boot some intermediate thing like a bootloader that does acceptance ..." is pretty clever! So if upstream thinks this FW-kernel negotiation is not a good direction, maybe we (Google) can pursue this idea to avoid introducing yet another tag on our images.
I'm obviously speaking only for myself here and not for "upstream" as a whole, but I clearly don't like the FW/kernel negotiation thing. It's a permanent pain in our necks to solve a very temporary problem.
EFI is basically our existing embodiment of this fw/kernel negotiation thing, and iff we need it, I have no objection to using it for this purpose, i.e., to allow the firmware to infer whether or not it should accept all available memory on behalf of the OS before exiting boot services. But if we don't need this, even better.
Quoted 8 lines
Quoted 4 lines
FW/kernel negotiation does not work if there's a boot loader in the middle that does
ExitBootServices(). By the time kernel can announce if it supports unaccepted memory there's nobody to announce to.Why would you want to support such bootloaders for TDX anyway? TDX heavily relies on measured boot abstractions and other things that are heavily tied to firmware.
I don't understand it either. And, yet, there's demand for it.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
EFI is basically our existing embodiment of this fw/kernel negotiation thing, and iff we need it, I have no objection to using it for this purpose, i.e., to allow the firmware to infer whether or not it should accept all available memory on behalf of the OS before exiting boot services. But if we don't need this, even better.
FW/kernel negotiation does not work if there's a boot loader in the middle that does
ExitBootServices(). By the time kernel can announce if it supports unaccepted memory there's nobody to announce to.
Quoted 6 lines
Quoted 3 lines
Why would you want to support such bootloaders for TDX anyway? TDX heavily relies on measured boot abstractions and other things that are heavily tied to firmware.
I don't understand it either. And, yet, there's demand for it.
I think there's no good solution for this bad upgrade path that the UEFI spec stuck us with, so I think I'm going to stick to what many folks have suggested: just have the host require external information. What this means is that at VM creation time, the user has to specify an extra flag that all memory has to be accepted in firmware before booting the guest OS. Failure to provide the flag leads to the unfortunate outcome that the VM only has access to the lower 4GB of RAM. We can only hope that the VM OOMs shortly after they start up the machine and the user reads an FAQ that they should add this flag.
I'll do a round of appeals to distributions to include this patch set and AMD's follow-up that defines accept_memory for SEV-SNP to reduce the time that people need to know about this flag.
Signature · 1 line
-Dionna Glaze, PhD (she/her)
Hey
I posted my comment on Bugzilla https://bugzilla.tianocore.org/show_bug.cgi?id=3987
Let's achieve EDKII/UEFI related discussion there.
Thank you
Yao, Jiewen
Quoted 73 lines
-----Original Message-----
From: Dionna Amalie Glaze <[email protected]>
Sent: Tuesday, July 19, 2022 7:32 AM
To: Kirill A. Shutemov <[email protected]>
Cc: Ard Biesheuvel <[email protected]>; Peter Gonda <[email protected]>; Borislav Petkov <[email protected]>; Lutomirski, Andy <[email protected]>; Christopherson,, Sean <[email protected]>; Andrew Morton <akpm@linux- foundation.org>; Rodel, Jorg <[email protected]>; Andi Kleen <[email protected]>; Kuppuswamy Sathyanarayanan <[email protected]>; David Rientjes <[email protected]>; Vlastimil Babka <[email protected]>; Tom Lendacky <[email protected]>; Thomas Gleixner <[email protected]>; Peter Zijlstra <[email protected]>; Paolo Bonzini <[email protected]>; Ingo Molnar <[email protected]>; Varad Gautam <[email protected]>; Dario Faggioli <[email protected]>; Hansen, Dave <[email protected]>; Mike Rapoport <[email protected]>; David Hildenbrand <[email protected]>; Marcelo Cerri <[email protected]>; [email protected]; Khalid ElMously <[email protected]>; Cox, Philip <[email protected]>; thearch/x86maintainers <[email protected]>; Linux Memory Management List <[email protected]>; linux- [email protected]; linux-efi <[email protected]>; LKML <linux- [email protected]>; Yao, Jiewen <[email protected]>
Subject: Re: [PATCHv7 00/14] mm, x86/cc: Implement support for unaccepted memoryQuoted 3 lines
I've talked with our firmware expert today and I think we have a problem with the approach when kernel declaries support of unaccepted memory.
Is this Jiewen Yao? I've been trying to design the UEFI spec change with him. The bootloader problem he commented with this morning was something I wasn't fully considering.
Quoted 8 lines
This apporach doesn't work if we include bootloader into the picture: if
EBS()called by bootloader we still cannot know if target kernel supports unaccepted memory and we return to the square 1.I think we should make it obvious from a kernel image if it supports unaccepted memory (with
UTS_VERSIONor other way).Any comments?
Is this binary parsing trick already used in EDK2? If not, I wouldn't want to introduce an ABI-solidifying requirement like that.
A bit more cumbersome, but more flexible way to enable the feature is an idea I had in a meeting today:
Make unaccepted memory support a feature-enabling EFI driver installed to the EFI system partition.* The first time you boot (setup mode), you install an EFI driver that just sets a feature Pcd to true (using a custom protocol as Ard had suggested above).
* The second time you boot, if the feature Pcd is true, then the UEFI is free to not accept memory and use the unaccepted memory type. The bootloader will run after unaccepted memory has been allowed already, so there is no accept-all event.The default behavior will be to accept all memory when GetMemoryMap is called unless the feature pcd is set to true.
We can then say this driver isn't needed once some new generation of this technology comes along and we can require unaccepted memory support as part of that technology's baseline, or we manage to update the UEFI spec to have GetMemoryMapEx which has unaccepted memory support baked in and the bootloaders all know to use it.
The cloud experience will be, "is boot slow? Install this EFI driver from the cloud service provider" to tell the UEFI to enable unaccepted memory.
--
-Dionna Glaze, PhD (she/her)
cover Provide SEV-SNP support for unaccepted memory
In reply to Kirill A. ShutemovThis series adds SEV-SNP support for unaccepted memory to the patch series titled:
[PATCHv7 00/14] mm, x86/cc: Implement support for unaccepted memory
Currently, when changing the state of a page under SNP, the page state change structure is kmalloc()'d. This lead to hangs during boot when accepting memory because the allocation can trigger the need to accept more memory. So this series consists of two patches:
- A pre-patch to switch from a kmalloc()'d page state change structure to a per-CPU page state change structure.
- SNP support for unaccepted memory.
The series is based off of and tested against Kirill Shutemov's tree:
https://github.com/intel/tdx.git guest-unaccepted-memory
---
Tom Lendacky (2):
x86/sev: Use per-CPU PSC structure in prep for unaccepted memory
support
x86/sev: Add SNP-specific unaccepted memory support
arch/x86/Kconfig | 1 + arch/x86/boot/compressed/mem.c | 3 ++ arch/x86/boot/compressed/sev.c | 10 ++++- arch/x86/boot/compressed/sev.h | 23 ++++++++++ arch/x86/include/asm/sev.h | 3 ++ arch/x86/kernel/sev.c | 76 ++++++++++++++++++++++++--------- arch/x86/mm/unaccepted_memory.c | 4 ++ 7 files changed, 98 insertions(+), 22 deletions(-) create mode 100644 arch/x86/boot/compressed/sev.h
Signature · 1 line
2.36.1
PATCH 1/2 x86/sev: Use per-CPU PSC structure in prep for unaccepted memory support
In advance of providing support for unaccepted memory, switch from using kmalloc() for allocating the Page State Change structure to using a per-CPU structure. This is needed to avoid a possible recursive call into set_pages_state() if the allocation requires (more) memory to be accepted, which would result in a hang.
Protect the use of the per-CPU structure by disabling interrupts during memory acceptance. Since the set_pages_state() path is the only path into vmgexit_psc(), rename vmgexit_psc() to __vmgexit_psc() and remove the calls to disable interrupts which are now performed by set_pages_state().
Even with interrupts disabled, an NMI can be raised while performing memory acceptance. The NMI could then cause further memory acceptance to performed. To prevent corruption to the per-CPU structure, use the PSC MSR protocol in this situation.
--- arch/x86/kernel/sev.c | 60 ++++++++++++++++++++++++++++--------------- 1 file changed, 39 insertions(+), 21 deletions(-)
arch/x86/kernel/sev.c +39 -21
@@ -104,6 +104,15 @@ struct sev_es_runtime_data {* is currently unsupported in SEV-ES guests. */ unsigned long dr7; + +/* +* Page State Change structure for use when accepting memory or when +* changing page state. Interrupts are disabled when using the structure +* but an NMI could still be raised, so use a flag to indicate when the +* structure is in use and use the MSR protocol in these cases. +*/ +struct snp_psc_desc psc_desc; +bool psc_active; };struct ghcb_state {
@@ -660,7 +669,7 @@ static void pvalidate_pages(unsigned long vaddr, unsigned int npages, bool valid } } -static void __init early_set_pages_state (unsigned long paddr,unsigned int npages,enum psc_op op) +static void early_set_pages_state (unsigned long paddr,unsigned int npages,enum psc_op op) {unsigned long paddr_end;u64 val;
@@ -742,26 +751,17 @@ void __init snp_prep_memory(unsigned long paddr, unsigned int sz, enum psc_op opWARN (1 ,"invalid memory op %d\n" , op); } -static int vmgexit_psc (struct snp_psc_desc *desc) +static int __vmgexit_psc (struct snp_psc_desc *desc) {int cur_entry, end_entry, ret =0 ;struct snp_psc_desc *data;struct ghcb_state state;struct es_em_ctxt ctxt; -unsigned long flags;struct ghcb *ghcb; -/* -* __sev_get_ghcb() needs to run with IRQs disabled because it is using -* a per-CPU GHCB. -*/ -local_irq_save (flags); - ghcb =__sev_get_ghcb (&state); -if (!ghcb) { - ret =1 ; - goto out_unlock; - } +if (!ghcb) +return 1 ;/* Copy the input desc into GHCB shared buffer */ data = (struct snp_psc_desc *)ghcb->shared_buffer;
@@ -820,9 +820,6 @@ static int vmgexit_psc(struct snp_psc_desc *desc) out:__sev_put_ghcb (&state); -out_unlock: -local_irq_restore (flags); -return ret; }
@@ -861,18 +858,32 @@ static void __set_pages_state(struct snp_psc_desc *data, unsigned long vaddr, i++; } -if (vmgexit_psc (data)) +if (__vmgexit_psc (data))sev_es_terminate (SEV_TERM_SET_LINUX ,GHCB_TERM_PSC ); }static void set_pages_state (unsigned long vaddr,unsigned int npages,int op) {unsigned long vaddr_end, next_vaddr; +struct sev_es_runtime_data *data;struct snp_psc_desc *desc; +unsigned long flags; - desc =kmalloc (sizeof (*desc),GFP_KERNEL_ACCOUNT ); -if (!desc) -panic ("SNP: failed to allocate memory for PSC descriptor\n" ); +/* Disable interrupts since a per-CPU PSC and per-CPU GHCB are used. */ +local_irq_save (flags); + + data =this_cpu_read (runtime_data); +if (!data || data->psc_active) { +/* No per-CPU PSC or it is active, use the MSR protocol. */ +early_set_pages_state (__pa (vaddr), npages, op); + goto out; + } + +/* Mark the PSC in use. */ + data->psc_active = true; +barrier (); + + desc = &data->psc_desc; vaddr = vaddr &PAGE_MASK ; vaddr_end = vaddr + (npages <<PAGE_SHIFT );
@@ -887,7 +898,12 @@ static void set_pages_state(unsigned long vaddr, unsigned int npages, int op) vaddr = next_vaddr; } -kfree (desc); +/* Mark the PSC no longer in use. */ +barrier (); + data->psc_active = false; + +out: +local_irq_restore (flags); }void snp_set_memory_shared (unsigned long vaddr,unsigned int npages)
@@ -1339,6 +1355,8 @@ static void __init alloc_runtime_data(int cpu)panic ("Can't allocate SEV-ES runtime data" );per_cpu (runtime_data, cpu) = data; + + data->psc_active = false; }static void __init init_ghcb (int cpu)
Signature · 1 line
2.36.1
Re: [PATCH v1 1/2] x86/sev: Use per-CPU PSC structure in prep for unaccepted memory support
Quoted 8 lines · Tom Lendacky
arch/x86/kernel/sev.c +0
-0@@ -104,6 +104,15 @@ struct sev_es_runtime_data {* is currently unsupported in SEV-ES guests. */ unsigned long dr7;
arch/x86/kernel/sev.c struct sev_es_runtime_data { · line 116
Quoted 10 lines
arch/x86/kernel/sev.c +9
-0+ +/* +* Page State Change structure for use when accepting memory or when +* changing page state. Interrupts are disabled when using the structure +* but an NMI could still be raised, so use a flag to indicate when the +* structure is in use and use the MSR protocol in these cases. +*/ +struct snp_psc_desc psc_desc; +bool psc_active; };
This thing:
struct snp_psc_desc {
struct psc_hdr hdr;
struct psc_entry entries[VMGEXIT_PSC_MAX_ENTRY];
} __packed;is 16k, right? Being per-cpu, this might eat up a MB or two of memory on a big server?
Considering that runtime acceptance is already single-threaded[1] *and* there's a fallback method, why not just have a single copy of this guarded by a single lock?
Re: [PATCH v1 1/2] x86/sev: Use per-CPU PSC structure in prep for unaccepted memory support
Quoted 22 lines · Dave Hansen
Quoted 8 lines · Tom Lendacky
arch/x86/kernel/sev.c +0
-0@@ -104,6 +104,15 @@ struct sev_es_runtime_data {* is currently unsupported in SEV-ES guests. */ unsigned long dr7;arch/x86/kernel/sev.c struct sev_es_runtime_data { · line 116
Quoted 10 lines
arch/x86/kernel/sev.c +9
-0+ +/* +* Page State Change structure for use when accepting memory or when +* changing page state. Interrupts are disabled when using the structure +* but an NMI could still be raised, so use a flag to indicate when the +* structure is in use and use the MSR protocol in these cases. +*/ +struct snp_psc_desc psc_desc; +bool psc_active; };This thing:
Quoted 7 lines
struct snp_psc_desc { struct psc_hdr hdr; struct psc_entry entries[VMGEXIT_PSC_MAX_ENTRY]; } __packed;is 16k, right? Being per-cpu, this might eat up a MB or two of memory on a big server?
It's just under 2K, 2,032 bytes.
Quoted 4 lines
Considering that runtime acceptance is already single-threaded[1] *and* there's a fallback method, why not just have a single copy of this guarded by a single lock?
This function is called for more than just memory acceptance. It's also called for any changes from or to private or shared, which isn't single-threaded.
Thanks,
Tom
Quoted 3 lines
Re: [PATCH v1 1/2] x86/sev: Use per-CPU PSC structure in prep for unaccepted memory support
Quoted 7 lines · Tom Lendacky
Quoted 3 lines
Considering that runtime acceptance is already single-threaded[1] *and* there's a fallback method, why not just have a single copy of this guarded by a single lock?
This function is called for more than just memory acceptance. It's also called for any changes from or to private or shared, which isn't single-threaded.
I think this tidbit from the changelog threw me off:
Quoted 2 lines
Protect the use of the per-CPU structure by disabling interrupts during memory acceptance.
Could you please revise that to accurately capture the impact of this change?
Re: [PATCH v1 1/2] x86/sev: Use per-CPU PSC structure in prep for unaccepted memory support
Quoted 9 lines · Dave Hansen
Quoted 7 lines · Tom Lendacky
Quoted 3 lines
Considering that runtime acceptance is already single-threaded[1] *and* there's a fallback method, why not just have a single copy of this guarded by a single lock?
This function is called for more than just memory acceptance. It's also called for any changes from or to private or shared, which isn't single-threaded.
Quoted 7 lines
I think this tidbit from the changelog threw me off:
Quoted 2 lines
Protect the use of the per-CPU structure by disabling interrupts during memory acceptance.
Could you please revise that to accurately capture the impact of this change?
Is s/memory acceptance/page state changes/ enough of what you are looking for or something more?
Thanks,
Tom
Re: [PATCH v1 1/2] x86/sev: Use per-CPU PSC structure in prep for unaccepted memory support
Quoted 10 lines · Tom Lendacky
Quoted 7 lines
I think this tidbit from the changelog threw me off:
Quoted 2 lines
Protect the use of the per-CPU structure by disabling interrupts during memory acceptance.
Could you please revise that to accurately capture the impact of this change?
Is s/memory acceptance/page state changes/ enough of what you are looking for or something more?
That, plus a reminder of when "page state changes" are performed would be nice. How frequent are they? Are they performance sensitive? That'll help us decide if the design here is appropriate or not.
Re: [PATCH v1 1/2] x86/sev: Use per-CPU PSC structure in prep for unaccepted memory support
Quoted 9 lines · Dave Hansen
Quoted 8 lines · Tom Lendacky
Quoted 7 lines
I think this tidbit from the changelog threw me off:
Quoted 2 lines
Protect the use of the per-CPU structure by disabling interrupts during memory acceptance.
Could you please revise that to accurately capture the impact of this change?
Quoted 6 lines
Quoted 2 lines
Is s/memory acceptance/page state changes/ enough of what you are looking for or something more?
That, plus a reminder of when "page state changes" are performed would be nice. How frequent are they? Are they performance sensitive? That'll help us decide if the design here is appropriate or not.
Without submitting a v2, here's what the updated paragraph would look like:
Page state changes occur whenever DMA memory is allocated or memory needs to be shared with the hypervisor (kvmclock, attestation reports, etc.). A per-CPU structure is chosen over a single PSC structure protected with a lock because these changes can be initiated from interrupt or soft-interrupt context (e.g. the NVMe driver). Protect the use of the per-CPU structure by disabling interrupts during page state changes. Since the set_pages_state() path is the only path into vmgexit_psc(),
rename vmgexit_psc() to __vmgexit_psc() and remove the calls to disable
interrupts which are now performed by set_pages_state().
Hopefully there aren't a lot of page state changes occurring once a system has booted, so maybe a static struct with a lock would work. I am a bit worried about an NMI occurring during a page state change that requires a lock. I suppose, in_nmi() can be used to detect that and go the MSR protocol route to avoid a deadlock.
I can investigate that if the 2K-extra per-CPU is not desired.
Thanks,
Tom
cover Provide SEV-SNP support for unaccepted memory
This series adds SEV-SNP support for unaccepted memory to the patch series titled:
[PATCHv7 00/14] mm, x86/cc: Implement support for unaccepted memory
Currently, when changing the state of a page under SNP, the page state change structure is kmalloc()'d. This lead to hangs during boot when accepting memory because the allocation can trigger the need to accept more memory. So this series consists of two patches:
- A pre-patch to switch from a kmalloc()'d page state change structure to a static page state change structure proteced with access protected by a spinlock.
- SNP support for unaccepted memory.
The series is based off of and tested against Kirill Shutemov's tree:
https://github.com/intel/tdx.git guest-unaccepted-memory
---
This is what the static structure / spinlock method looks like. Let me know if this approach is preferred over the per-CPU structure. If so, I'll submit this as a v2.
Thanks,
Tom
Tom Lendacky (2):
x86/sev: Use per-CPU PSC structure in prep for unaccepted memory
support
x86/sev: Add SNP-specific unaccepted memory support
arch/x86/Kconfig | 1 + arch/x86/boot/compressed/mem.c | 3 ++ arch/x86/boot/compressed/sev.c | 10 ++++- arch/x86/boot/compressed/sev.h | 23 +++++++++++ arch/x86/include/asm/sev.h | 3 ++ arch/x86/kernel/sev.c | 71 ++++++++++++++++++++++----------- arch/x86/mm/unaccepted_memory.c | 4 ++ 7 files changed, 91 insertions(+), 24 deletions(-) create mode 100644 arch/x86/boot/compressed/sev.h
Signature · 1 line
2.36.1
PATCH 1/2 x86/sev: Use per-CPU PSC structure in prep for unaccepted memory support
In advance of providing support for unaccepted memory, switch from using kmalloc() for allocating the Page State Change (PSC) structure to using a static structure. This is needed to avoid a possible recursive call into set_pages_state() if the kmalloc() call requires (more) memory to be accepted, which would result in a hang.
Page state changes occur whenever DMA memory is allocated or memory needs to be shared with the hypervisor (kvmclock, attestation reports, etc.). Since most page state changes occur early in boot and are limited in number, a single static PSC structure is used and protected by a spin lock with interrupts disabled.
Even with interrupts disabled, an NMI can be raised while performing memory acceptance. The NMI could then cause further memory acceptance to be performed. To prevent a deadlock, use the MSR protocol if executing in an NMI context.
Since the set_pages_state() path is the only path into vmgexit_psc(), rename vmgexit_psc() to __vmgexit_psc() and remove the calls to disable interrupts which are now performed by set_pages_state().
--- arch/x86/kernel/sev.c | 55 +++++++++++++++++++++++++------------------ 1 file changed, 32 insertions(+), 23 deletions(-)
arch/x86/kernel/sev.c +32 -23
@@ -66,6 +66,9 @@ static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);*/ static struct ghcb *boot_ghcb __section(".data"); +/* Flag to indicate when the first per-CPU GHCB is registered */ +static bool ghcb_percpu_ready __section(".data"); +/* Bitmap of SEV features supported by the hypervisor */ static u64 sev_hv_features__ro_after_init ;
@@ -122,6 +125,15 @@ struct sev_config {static struct sev_config sev_cfg__read_mostly ; +/* +* Page State Change structure for use when accepting memory or when changing +* page state. Use is protected by a spinlock with interrupts disabled, but an +* NMI could still be raised, so check if running in an NMI an use the MSR +* protocol in these cases. +*/ +static struct snp_psc_desc psc_desc; +static DEFINE_SPINLOCK (psc_desc_lock); + static __always_inline bool on_vc_stack(struct pt_regs *regs) {unsigned long sp = regs->sp;
@@ -660,7 +672,7 @@ static void pvalidate_pages(unsigned long vaddr, unsigned int npages, bool valid } } -static void __init early_set_pages_state (unsigned long paddr,unsigned int npages,enum psc_op op) +static void early_set_pages_state (unsigned long paddr,unsigned int npages,enum psc_op op) {unsigned long paddr_end;u64 val;
@@ -742,26 +754,17 @@ void __init snp_prep_memory(unsigned long paddr, unsigned int sz, enum psc_op opWARN (1 ,"invalid memory op %d\n" , op); } -static int vmgexit_psc (struct snp_psc_desc *desc) +static int __vmgexit_psc (struct snp_psc_desc *desc) {int cur_entry, end_entry, ret =0 ;struct snp_psc_desc *data;struct ghcb_state state;struct es_em_ctxt ctxt; -unsigned long flags;struct ghcb *ghcb; -/* -* __sev_get_ghcb() needs to run with IRQs disabled because it is using -* a per-CPU GHCB. -*/ -local_irq_save (flags); - ghcb =__sev_get_ghcb (&state); -if (!ghcb) { - ret =1 ; - goto out_unlock; - } +if (!ghcb) +return 1 ;/* Copy the input desc into GHCB shared buffer */ data = (struct snp_psc_desc *)ghcb->shared_buffer;
@@ -820,9 +823,6 @@ static int vmgexit_psc(struct snp_psc_desc *desc) out:__sev_put_ghcb (&state); -out_unlock: -local_irq_restore (flags); -return ret; }
@@ -861,18 +861,25 @@ static void __set_pages_state(struct snp_psc_desc *data, unsigned long vaddr, i++; } -if (vmgexit_psc (data)) +if (__vmgexit_psc (data))sev_es_terminate (SEV_TERM_SET_LINUX ,GHCB_TERM_PSC ); }static void set_pages_state (unsigned long vaddr,unsigned int npages,int op) {unsigned long vaddr_end, next_vaddr; -struct snp_psc_desc *desc; +unsigned long flags; - desc =kmalloc (sizeof (*desc),GFP_KERNEL_ACCOUNT ); -if (!desc) -panic ("SNP: failed to allocate memory for PSC descriptor\n" ); +/* +* Use the MSR protocol when either: +* - executing in an NMI to avoid any possibility of a deadlock +* - per-CPU GHCBs are not yet registered, since __vmgexit_psc() +* uses the per-CPU GHCB. +*/ +if (in_nmi () || !ghcb_percpu_ready) +return early_set_pages_state (__pa (vaddr), npages, op); + +spin_lock_irqsave (&psc_desc_lock, flags); vaddr = vaddr &PAGE_MASK ; vaddr_end = vaddr + (npages <<PAGE_SHIFT );
@@ -882,12 +889,12 @@ static void set_pages_state(unsigned long vaddr, unsigned int npages, int op) next_vaddr = min_t(unsigned long, vaddr_end, (VMGEXIT_PSC_MAX_ENTRY *PAGE_SIZE ) + vaddr); -__set_pages_state (desc, vaddr, next_vaddr, op); +__set_pages_state (&psc_desc, vaddr, next_vaddr, op); vaddr = next_vaddr; } -kfree (desc); +spin_unlock_irqrestore (&psc_desc_lock, flags); }void snp_set_memory_shared (unsigned long vaddr,unsigned int npages)
@@ -1254,6 +1261,8 @@ void setup_ghcb(void)if (cc_platform_has (CC_ATTR_GUEST_SEV_SNP ))snp_register_per_cpu_ghcb (); + ghcb_percpu_ready = true; +return ; }
Signature · 1 line
2.36.1
Re: [PATCH v1.1 1/2] x86/sev: Use per-CPU PSC structure in prep for unaccepted memory support
arch/x86/kernel/sev.c static void __set_pages_state(struct snp_psc_desc *data, unsigned long vaddr, · line 882
Quoted 10 lines · Tom Lendacky
arch/x86/kernel/sev.c +10
-0+/* +* Use the MSR protocol when either: +* - executing in an NMI to avoid any possibility of a deadlock +* - per-CPU GHCBs are not yet registered, since __vmgexit_psc() +* uses the per-CPU GHCB. +*/ +if (in_nmi () || !ghcb_percpu_ready) +return early_set_pages_state (__pa (vaddr), npages, op); + +spin_lock_irqsave (&psc_desc_lock, flags);
Would it be simpler to just do a spin_trylock_irqsave()? You fall back to early_set_pages_state() whenever you can't acquire the lock.
That avoids even having to know what the situations are where you _might_ recurse. If it recurses, the trylock will just naturally fail.
You simply can't have bugs where the "(in_nmi() || !ghcb_percpu_ready)"
conditional was wrong.
Re: [PATCH v1.1 1/2] x86/sev: Use per-CPU PSC structure in prep for unaccepted memory support
Quoted 14 lines · Dave Hansen
arch/x86/kernel/sev.c static void __set_pages_state(struct snp_psc_desc *data, unsigned long vaddr, · line 882
Quoted 10 lines · Tom Lendacky
arch/x86/kernel/sev.c +10
-0+/* +* Use the MSR protocol when either: +* - executing in an NMI to avoid any possibility of a deadlock +* - per-CPU GHCBs are not yet registered, since __vmgexit_psc() +* uses the per-CPU GHCB. +*/ +if (in_nmi () || !ghcb_percpu_ready) +return early_set_pages_state (__pa (vaddr), npages, op); + +spin_lock_irqsave (&psc_desc_lock, flags);Would it be simpler to just do a
spin_trylock_irqsave()? You fall back toearly_set_pages_state()whenever you can't acquire the lock.
I was looking at that and can definitely go that route if this approach is preferred.
Thanks,
Tom
Quoted 5 lines
That avoids even having to know what the situations are where you _might_ recurse. If it recurses, the trylock will just naturally fail.
You simply can't have bugs where the "(in_nmi()|| !ghcb_percpu_ready)"
conditional was wrong.
Re: [PATCH v1.1 1/2] x86/sev: Use per-CPU PSC structure in prep for unaccepted memory support
Quoted 5 lines · Tom Lendacky
Quoted 2 lines
Would it be simpler to just do a spin_trylock_irqsave()? You fall back to
early_set_pages_state()whenever you can't acquire the lock.I was looking at that and can definitely go that route if this approach is preferred.
I prefer it for sure.
This whole iteration does look good to me versus the per-cpu version, so I say go ahead with doing this for v2 once you wait a bit for any more feedback.
Re: [PATCH v1.1 1/2] x86/sev: Use per-CPU PSC structure in prep for unaccepted memory support
Quoted 4 lines · Dave Hansen
Quoted 3 lines · Tom Lendacky
Quoted 2 lines
Would it be simpler to just do a spin_trylock_irqsave()? You fall back to
early_set_pages_state()whenever you can't acquire the lock.
Quoted 8 lines
Quoted 2 lines
I was looking at that and can definitely go that route if this approach is preferred.
I prefer it for sure.
This whole iteration does look good to me versus the per-cpu version, so I say go ahead with doing this for v2 once you wait a bit for any more feedback.
I'm still concerned about the whole spinlock and performance. What if I reduce the number of entries in the PSC structure to, say, 64, which reduces the size of the struct to 520 bytes. Any issue if that is put on the stack, instead? It definitely makes things less complicated and feels like a good compromise on the size vs the number of PSC VMGEXIT requests.
Thanks,
Tom
Re: [PATCH v1.1 1/2] x86/sev: Use per-CPU PSC structure in prep for unaccepted memory support
Quoted 10 lines · Tom Lendacky
Quoted 3 lines
This whole iteration does look good to me versus the per-cpu version, so I say go ahead with doing this for v2 once you wait a bit for any more feedback.
I'm still concerned about the whole spinlock and performance. What if I reduce the number of entries in the PSC structure to, say, 64, which reduces the size of the struct to 520 bytes. Any issue if that is put on the stack, instead? It definitely makes things less complicated and feels like a good compromise on the size vs the number of PSC VMGEXIT requests.
That would be fine too.
But, I doubt there will be any real performance issues coming out of this. As bad as this MSR thing is, I suspect it's not half as disastrous as the global spinlock in Kirill's patches.
Also, private<->shared page conversions are *NOT* common from what I can tell. There are a few pages converted at boot, but most host the guest<->host communications are through the swiotlb pages which are static.
Are there other things that SEV uses this structure for that I'm missing?
Re: [PATCH v1.1 1/2] x86/sev: Use per-CPU PSC structure in prep for unaccepted memory support
Quoted 5 lines · Dave Hansen
Quoted 4 lines · Tom Lendacky
Quoted 3 lines
This whole iteration does look good to me versus the per-cpu version, so I say go ahead with doing this for v2 once you wait a bit for any more feedback.
Quoted 8 lines
Quoted 6 lines
I'm still concerned about the whole spinlock and performance. What if I reduce the number of entries in the PSC structure to, say, 64, which reduces the size of the struct to 520 bytes. Any issue if that is put on the stack, instead? It definitely makes things less complicated and feels like a good compromise on the size vs the number of PSC VMGEXIT requests.
That would be fine too.
Ok.
Quoted 8 lines
But, I doubt there will be any real performance issues coming out of this. As bad as this MSR thing is, I suspect it's not half as disastrous as the global spinlock in Kirill's patches.
Also, private<->shared page conversions are *NOT* common from what I can tell. There are a few pages converted at boot, but most host the guest<->host communications are through the swiotlb pages which are static.
Generally, that's true. But, e.g., a dma_alloc_coherent() actually doesn't go through SWIOTLB, but instead allocates the pages and makes them shared, which results in a page state change. The NVMe driver was calling that API a lot. In this case, though, the NVMe driver was running in IRQ context and set_memory_decrypted() could sleep, so an unencrypted DMA memory pool was created to work around the sleeping issue and reduce the page state changes. It's just things like that, that make me wary.
Thanks,
Tom
Quoted 2 lines
Are there other things that SEV uses this structure for that I'm missing?
Re: [PATCH v1.1 1/2] x86/sev: Use per-CPU PSC structure in prep for unaccepted memory support
Quoted 5 lines · Tom Lendacky
Quoted 4 lines
Also, private<->shared page conversions are *NOT* common from what I can tell. There are a few pages converted at boot, but most host the guest<->host communications are through the swiotlb pages which are static.
Quoted 8 lines
Generally, that's true. But, e.g., a
dma_alloc_coherent()actually doesn't go through SWIOTLB, but instead allocates the pages and makes them shared, which results in a page state change. The NVMe driver was calling that API a lot. In this case, though, the NVMe driver was running in IRQ context andset_memory_decrypted()could sleep, so an unencrypted DMA memory pool was created to work around the sleeping issue and reduce the page state changes. It's just things like that, that make me wary.
Interesting. Is that a real passthrough NVMe device or the hypervisor presenting a virtual one that just happens to use the NVMe driver?
I'm pretty sure the TDX folks have been banking on having very few page state changes. But, part of that at least is their expectation of relying heavily on virtio.
I wonder if their expectations are accurate, or if once TDX gets out into the real world if their hopes will be dashed.
Re: [PATCH v1.1 1/2] x86/sev: Use per-CPU PSC structure in prep for unaccepted memory support
Quoted 15 lines · Dave Hansen
Quoted 5 lines · Tom Lendacky
Quoted 4 lines
Also, private<->shared page conversions are *NOT* common from what I can tell. There are a few pages converted at boot, but most host the guest<->host communications are through the swiotlb pages which are static.
Quoted 8 lines
Generally, that's true. But, e.g., a
dma_alloc_coherent()actually doesn't go through SWIOTLB, but instead allocates the pages and makes them shared, which results in a page state change. The NVMe driver was calling that API a lot. In this case, though, the NVMe driver was running in IRQ context andset_memory_decrypted()could sleep, so an unencrypted DMA memory pool was created to work around the sleeping issue and reduce the page state changes. It's just things like that, that make me wary.
Quoted 2 lines
Interesting. Is that a real passthrough NVMe device or the hypervisor presenting a virtual one that just happens to use the NVMe driver?
Hmmm... not sure, possibly the latter. I just knew that whatever it was, the NVMe driver was being used.
Thanks,
Tom
Quoted 7 lines
I'm pretty sure the TDX folks have been banking on having very few page state changes. But, part of that at least is their expectation of relying heavily on virtio.
I wonder if their expectations are accurate, or if once TDX gets out into the real world if their hopes will be dashed.
Re: [PATCH v1.1 1/2] x86/sev: Use per-CPU PSC structure in prep for unaccepted memory support
In reply to Tom LendackyOn 8/3/22 13:11, Tom Lendacky wrote:
Of course, I'll fix the subject if submitting this for real... ugh.
Thanks,
Tom
Quoted 160 lines
In advance of providing support for unaccepted memory, switch from using
kmalloc()for allocating the Page State Change (PSC) structure to using a static structure. This is needed to avoid a possible recursive call intoset_pages_state()if thekmalloc()call requires (more) memory to be accepted, which would result in a hang.Page state changes occur whenever DMA memory is allocated or memory needs to be shared with the hypervisor (kvmclock, attestation reports, etc.). Since most page state changes occur early in boot and are limited in number, a single static PSC structure is used and protected by a spin lock with interrupts disabled.
Even with interrupts disabled, an NMI can be raised while performing memory acceptance. The NMI could then cause further memory acceptance to be performed. To prevent a deadlock, use the MSR protocol if executing in an NMI context.
Since the
set_pages_state()path is the only path intovmgexit_psc(), renamevmgexit_psc()to__vmgexit_psc()and remove the calls to disable interrupts which are now performed byset_pages_state().Signed-off-by Tom Lendacky <[email protected]>--- arch/x86/kernel/sev.c | 55 +++++++++++++++++++++++++------------------ 1 file changed, 32 insertions(+), 23 deletions(-)arch/x86/kernel/sev.c +32
-23@@ -66,6 +66,9 @@ static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);*/ static struct ghcb *boot_ghcb __section(".data"); +/* Flag to indicate when the first per-CPU GHCB is registered */ +static bool ghcb_percpu_ready __section(".data"); +/* Bitmap of SEV features supported by the hypervisor */ static u64 sev_hv_features__ro_after_init ;@@ -122,6 +125,15 @@ struct sev_config {static struct sev_config sev_cfg__read_mostly ; +/* +* Page State Change structure for use when accepting memory or when changing +* page state. Use is protected by a spinlock with interrupts disabled, but an +* NMI could still be raised, so check if running in an NMI an use the MSR +* protocol in these cases. +*/ +static struct snp_psc_desc psc_desc; +static DEFINE_SPINLOCK (psc_desc_lock); + static __always_inline bool on_vc_stack(struct pt_regs *regs) {unsigned long sp = regs->sp;@@ -660,7 +672,7 @@ static void pvalidate_pages(unsigned long vaddr, unsigned int npages, bool valid } } -static void __init early_set_pages_state (unsigned long paddr,unsigned int npages,enum psc_op op) +static void early_set_pages_state (unsigned long paddr,unsigned int npages,enum psc_op op) {unsigned long paddr_end;u64 val;@@ -742,26 +754,17 @@ void __init snp_prep_memory(unsigned long paddr, unsigned int sz, enum psc_op opWARN (1 ,"invalid memory op %d\n" , op); } -static int vmgexit_psc (struct snp_psc_desc *desc) +static int __vmgexit_psc (struct snp_psc_desc *desc) {int cur_entry, end_entry, ret =0 ;struct snp_psc_desc *data;struct ghcb_state state;struct es_em_ctxt ctxt; -unsigned long flags;struct ghcb *ghcb; -/* -* __sev_get_ghcb() needs to run with IRQs disabled because it is using -* a per-CPU GHCB. -*/ -local_irq_save (flags); - ghcb =__sev_get_ghcb (&state); -if (!ghcb) { - ret =1 ; - goto out_unlock; - } +if (!ghcb) +return 1 ;/* Copy the input desc into GHCB shared buffer */ data = (struct snp_psc_desc *)ghcb->shared_buffer;@@ -820,9 +823,6 @@ static int vmgexit_psc(struct snp_psc_desc *desc) out:__sev_put_ghcb (&state); -out_unlock: -local_irq_restore (flags); -return ret; }@@ -861,18 +861,25 @@ static void __set_pages_state(struct snp_psc_desc *data, unsigned long vaddr, i++; } -if (vmgexit_psc (data)) +if (__vmgexit_psc (data))sev_es_terminate (SEV_TERM_SET_LINUX ,GHCB_TERM_PSC ); }static void set_pages_state (unsigned long vaddr,unsigned int npages,int op) {unsigned long vaddr_end, next_vaddr; -struct snp_psc_desc *desc; +unsigned long flags; - desc =kmalloc (sizeof (*desc),GFP_KERNEL_ACCOUNT ); -if (!desc) -panic ("SNP: failed to allocate memory for PSC descriptor\n" ); +/* +* Use the MSR protocol when either: +* - executing in an NMI to avoid any possibility of a deadlock +* - per-CPU GHCBs are not yet registered, since __vmgexit_psc() +* uses the per-CPU GHCB. +*/ +if (in_nmi () || !ghcb_percpu_ready) +return early_set_pages_state (__pa (vaddr), npages, op); + +spin_lock_irqsave (&psc_desc_lock, flags); vaddr = vaddr &PAGE_MASK ; vaddr_end = vaddr + (npages <<PAGE_SHIFT );@@ -882,12 +889,12 @@ static void set_pages_state(unsigned long vaddr, unsigned int npages, int op) next_vaddr = min_t(unsigned long, vaddr_end, (VMGEXIT_PSC_MAX_ENTRY *PAGE_SIZE ) + vaddr); -__set_pages_state (desc, vaddr, next_vaddr, op); +__set_pages_state (&psc_desc, vaddr, next_vaddr, op); vaddr = next_vaddr; } -kfree (desc); +spin_unlock_irqrestore (&psc_desc_lock, flags); }void snp_set_memory_shared (unsigned long vaddr,unsigned int npages)@@ -1254,6 +1261,8 @@ void setup_ghcb(void)if (cc_platform_has (CC_ATTR_GUEST_SEV_SNP ))snp_register_per_cpu_ghcb (); + ghcb_percpu_ready = true; +return ; }
PATCH 2/2 x86/sev: Add SNP-specific unaccepted memory support
In reply to Tom LendackyAdd SNP-specific hooks to the unaccepted memory support in the boot path (__accept_memory()) and the core kernel (accept_memory()) in order to support booting SNP guests when unaccepted memory is present. Without this support, SNP guests will fail to boot and/or panic() when unaccepted memory is present in the EFI memory map.
The process of accepting memory under SNP involves invoking the hypervisor to perform a page state change for the page to private memory and then issuing a PVALIDATE instruction to accept the page.
Create the new header file arch/x86/boot/compressed/sev.h because adding the function declaration to any of the existing SEV related header files pulls in too many other header files, causing the build to fail.
--- arch/x86/Kconfig | 1 + arch/x86/boot/compressed/mem.c | 3 +++ arch/x86/boot/compressed/sev.c | 10 +++++++++- arch/x86/boot/compressed/sev.h | 23 +++++++++++++++++++++++ arch/x86/include/asm/sev.h | 3 +++ arch/x86/kernel/sev.c | 16 ++++++++++++++++ arch/x86/mm/unaccepted_memory.c | 4 ++++ 7 files changed, 59 insertions(+), 1 deletion(-) create mode 100644 arch/x86/boot/compressed/sev.h
arch/x86/Kconfig +1 -0
@@ -1553,6 +1553,7 @@ config AMD_MEM_ENCRYPT select INSTRUCTION_DECODER select ARCH_HAS_CC_PLATFORM select X86_MEM_ENCRYPT + select UNACCEPTED_MEMORY help Say yes to enable support for the encryption of system memory. This requires an AMD processor that supports Secure Memory
arch/x86/boot/compressed/mem.c +3 -0
@@ -6,6 +6,7 @@#include "find.h" #include "math.h" #include "tdx.h" +#include "sev.h" #include <asm/shared/tdx.h> #define PMD_SHIFT 21
@@ -39,6 +40,8 @@ static inline void __accept_memory(phys_addr_t start, phys_addr_t end)/* Platform-specific memory-acceptance call goes here */ if (is_tdx_guest ())tdx_accept_memory (start, end); +else if (sev_snp_enabled ()) +snp_accept_memory (start, end);else error ("Cannot accept memory: unknown platform\n" ); }
arch/x86/boot/compressed/sev.c +9 -1
@@ -115,7 +115,7 @@ static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,/* Include code for early handlers */ #include "../../kernel/sev-shared.c" -static inline bool sev_snp_enabled (void ) +bool sev_snp_enabled (void ) {return sev_status &MSR_AMD64_SEV_SNP_ENABLED ; }
@@ -161,6 +161,14 @@ void snp_set_page_shared(unsigned long paddr)__page_state_change (paddr,SNP_PAGE_STATE_SHARED ); } +void snp_accept_memory (phys_addr_t start,phys_addr_t end) +{ +while (end > start) { +snp_set_page_private (start); + start +=PAGE_SIZE ; + } +} +static bool early_setup_ghcb (void ) {if (set_page_decrypted ((unsigned long )&boot_ghcb_page))
arch/x86/boot/compressed/sev.h new file +23 -0
@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* AMD SEV header for early boot related functions. +* +* Author: Tom Lendacky <[email protected]> +*/ + +#ifndef BOOT_COMPRESSED_SEV_H +#define BOOT_COMPRESSED_SEV_H + +#ifdef CONFIG_AMD_MEM_ENCRYPT + +bool sev_snp_enabled (void ); +void snp_accept_memory (phys_addr_t start,phys_addr_t end); + +#else + +static inline bool sev_snp_enabled (void ) {return false; } +static inline void snp_accept_memory (phys_addr_t start,phys_addr_t end) { } + +#endif + +#endif
arch/x86/include/asm/sev.h +3 -0
@@ -202,6 +202,7 @@ void snp_set_wakeup_secondary_cpu(void);bool snp_init (struct boot_params *bp);void snp_abort (void );int snp_issue_guest_request (u64 exit_code,struct snp_req_data *input,unsigned long *fw_err); +void snp_accept_memory (phys_addr_t start,phys_addr_t end);#else static inline void sev_es_ist_enter (struct pt_regs *regs) { }static inline void sev_es_ist_exit (void ) { }
@@ -226,6 +227,8 @@ static inline int snp_issue_guest_request(u64 exit_code, struct snp_req_data *in {return -ENOTTY ; } + +static inline void snp_accept_memory (phys_addr_t start,phys_addr_t end) { }#endif #endif
arch/x86/kernel/sev.c +16 -0
@@ -917,6 +917,22 @@ void snp_set_memory_private(unsigned long vaddr, unsigned int npages)pvalidate_pages (vaddr, npages, true); } +void snp_accept_memory (phys_addr_t start,phys_addr_t end) +{ +unsigned long vaddr; +unsigned int npages; + +if (!cc_platform_has (CC_ATTR_GUEST_SEV_SNP )) +return ; + + vaddr = (unsigned long )__va (start); + npages = (end - start) >>PAGE_SHIFT ; + +set_pages_state (vaddr, npages,SNP_PAGE_STATE_PRIVATE ); + +pvalidate_pages (vaddr, npages, true); +} +static int snp_set_vmsa (void *va,bool vmsa) {u64 attrs;
arch/x86/mm/unaccepted_memory.c +4 -0
@@ -9,6 +9,7 @@#include <asm/setup.h> #include <asm/shared/tdx.h> #include <asm/unaccepted_memory.h> +#include <asm/sev.h> /* Protects unaccepted memory bitmap */ static DEFINE_SPINLOCK (unaccepted_memory_lock);
@@ -66,6 +67,9 @@ void accept_memory(phys_addr_t start, phys_addr_t end)if (cpu_feature_enabled (X86_FEATURE_TDX_GUEST )) {tdx_accept_memory (range_start *PMD_SIZE , range_end *PMD_SIZE ); + }else if (cc_platform_has (CC_ATTR_GUEST_SEV_SNP )) { +snp_accept_memory (range_start *PMD_SIZE , + range_end *PMD_SIZE ); }else {panic ("Cannot accept memory: unknown platform\n" ); }
Signature · 1 line
2.36.1
PATCH 2/2 x86/sev: Add SNP-specific unaccepted memory support
In reply to Tom LendackyAdd SNP-specific hooks to the unaccepted memory support in the boot path (__accept_memory()) and the core kernel (accept_memory()) in order to support booting SNP guests when unaccepted memory is present. Without this support, SNP guests will fail to boot and/or panic() when unaccepted memory is present in the EFI memory map.
The process of accepting memory under SNP involves invoking the hypervisor to perform a page state change for the page to private memory and then issuing a PVALIDATE instruction to accept the page.
Create the new header file arch/x86/boot/compressed/sev.h because adding the function declaration to any of the existing SEV related header files pulls in too many other header files, causing the build to fail.
--- arch/x86/Kconfig | 1 + arch/x86/boot/compressed/mem.c | 3 +++ arch/x86/boot/compressed/sev.c | 10 +++++++++- arch/x86/boot/compressed/sev.h | 23 +++++++++++++++++++++++ arch/x86/include/asm/sev.h | 3 +++ arch/x86/kernel/sev.c | 16 ++++++++++++++++ arch/x86/mm/unaccepted_memory.c | 4 ++++ 7 files changed, 59 insertions(+), 1 deletion(-) create mode 100644 arch/x86/boot/compressed/sev.h
arch/x86/Kconfig +1 -0
@@ -1553,6 +1553,7 @@ config AMD_MEM_ENCRYPT select INSTRUCTION_DECODER select ARCH_HAS_CC_PLATFORM select X86_MEM_ENCRYPT + select UNACCEPTED_MEMORY help Say yes to enable support for the encryption of system memory. This requires an AMD processor that supports Secure Memory
arch/x86/boot/compressed/mem.c +3 -0
@@ -6,6 +6,7 @@#include "find.h" #include "math.h" #include "tdx.h" +#include "sev.h" #include <asm/shared/tdx.h> #define PMD_SHIFT 21
@@ -39,6 +40,8 @@ static inline void __accept_memory(phys_addr_t start, phys_addr_t end)/* Platform-specific memory-acceptance call goes here */ if (is_tdx_guest ())tdx_accept_memory (start, end); +else if (sev_snp_enabled ()) +snp_accept_memory (start, end);else error ("Cannot accept memory: unknown platform\n" ); }
arch/x86/boot/compressed/sev.c +9 -1
@@ -115,7 +115,7 @@ static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,/* Include code for early handlers */ #include "../../kernel/sev-shared.c" -static inline bool sev_snp_enabled (void ) +bool sev_snp_enabled (void ) {return sev_status &MSR_AMD64_SEV_SNP_ENABLED ; }
@@ -161,6 +161,14 @@ void snp_set_page_shared(unsigned long paddr)__page_state_change (paddr,SNP_PAGE_STATE_SHARED ); } +void snp_accept_memory (phys_addr_t start,phys_addr_t end) +{ +while (end > start) { +snp_set_page_private (start); + start +=PAGE_SIZE ; + } +} +static bool early_setup_ghcb (void ) {if (set_page_decrypted ((unsigned long )&boot_ghcb_page))
arch/x86/boot/compressed/sev.h new file +23 -0
@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* AMD SEV header for early boot related functions. +* +* Author: Tom Lendacky <[email protected]> +*/ + +#ifndef BOOT_COMPRESSED_SEV_H +#define BOOT_COMPRESSED_SEV_H + +#ifdef CONFIG_AMD_MEM_ENCRYPT + +bool sev_snp_enabled (void ); +void snp_accept_memory (phys_addr_t start,phys_addr_t end); + +#else + +static inline bool sev_snp_enabled (void ) {return false; } +static inline void snp_accept_memory (phys_addr_t start,phys_addr_t end) { } + +#endif + +#endif
arch/x86/include/asm/sev.h +3 -0
@@ -202,6 +202,7 @@ void snp_set_wakeup_secondary_cpu(void);bool snp_init (struct boot_params *bp);void snp_abort (void );int snp_issue_guest_request (u64 exit_code,struct snp_req_data *input,unsigned long *fw_err); +void snp_accept_memory (phys_addr_t start,phys_addr_t end);#else static inline void sev_es_ist_enter (struct pt_regs *regs) { }static inline void sev_es_ist_exit (void ) { }
@@ -226,6 +227,8 @@ static inline int snp_issue_guest_request(u64 exit_code, struct snp_req_data *in {return -ENOTTY ; } + +static inline void snp_accept_memory (phys_addr_t start,phys_addr_t end) { }#endif #endif
arch/x86/kernel/sev.c +16 -0
@@ -926,6 +926,22 @@ void snp_set_memory_private(unsigned long vaddr, unsigned int npages)pvalidate_pages (vaddr, npages, true); } +void snp_accept_memory (phys_addr_t start,phys_addr_t end) +{ +unsigned long vaddr; +unsigned int npages; + +if (!cc_platform_has (CC_ATTR_GUEST_SEV_SNP )) +return ; + + vaddr = (unsigned long )__va (start); + npages = (end - start) >>PAGE_SHIFT ; + +set_pages_state (vaddr, npages,SNP_PAGE_STATE_PRIVATE ); + +pvalidate_pages (vaddr, npages, true); +} +static int snp_set_vmsa (void *va,bool vmsa) {u64 attrs;
arch/x86/mm/unaccepted_memory.c +4 -0
@@ -9,6 +9,7 @@#include <asm/setup.h> #include <asm/shared/tdx.h> #include <asm/unaccepted_memory.h> +#include <asm/sev.h> /* Protects unaccepted memory bitmap */ static DEFINE_SPINLOCK (unaccepted_memory_lock);
@@ -66,6 +67,9 @@ void accept_memory(phys_addr_t start, phys_addr_t end)if (cpu_feature_enabled (X86_FEATURE_TDX_GUEST )) {tdx_accept_memory (range_start *PMD_SIZE , range_end *PMD_SIZE ); + }else if (cc_platform_has (CC_ATTR_GUEST_SEV_SNP )) { +snp_accept_memory (range_start *PMD_SIZE , + range_end *PMD_SIZE ); }else {panic ("Cannot accept memory: unknown platform\n" ); }
Signature · 1 line
2.36.1
Re: [PATCH v1 2/2] x86/sev: Add SNP-specific unaccepted memory support
Quoted 6 lines
arch/x86/kernel/sev.c +5
-0+void snp_accept_memory (phys_addr_t start,phys_addr_t end) +{ +unsigned long vaddr; +unsigned int npages; +
arch/x86/kernel/sev.c void snp_set_memory_private(unsigned long vaddr, unsigned int npages) · line 943
Quoted 10 lines
arch/x86/kernel/sev.c +10
-0+if (!cc_platform_has (CC_ATTR_GUEST_SEV_SNP )) +return ; + + vaddr = (unsigned long )__va (start); + npages = (end - start) >>PAGE_SHIFT ; + +set_pages_state (vaddr, npages,SNP_PAGE_STATE_PRIVATE ); + +pvalidate_pages (vaddr, npages, true); +}
My testing of this patch shows that a significant amount of time is spent using the MSR protocol to change page state, in such a significant fashion that it's slower than eagerly accepting all memory. The difference gets worse as the RAM size goes up, so I think there's some phase problem with the GHCB protocol not getting used early enough?
Signature · 1 line
-Dionna Glaze, PhD (she/her)
Re: [PATCH v1 2/2] x86/sev: Add SNP-specific unaccepted memory support
Quoted 15 lines · Dionna Amalie Glaze
Quoted 15 lines
arch/x86/kernel/sev.c +14
-0+void snp_accept_memory (phys_addr_t start,phys_addr_t end) +{ +unsigned long vaddr; +unsigned int npages; + +if (!cc_platform_has (CC_ATTR_GUEST_SEV_SNP )) +return ; + + vaddr = (unsigned long )__va (start); + npages = (end - start) >>PAGE_SHIFT ; + +set_pages_state (vaddr, npages,SNP_PAGE_STATE_PRIVATE ); + +pvalidate_pages (vaddr, npages, true);
Quoted 8 lines
Quoted 1 line
+}
My testing of this patch shows that a significant amount of time is spent using the MSR protocol to change page state, in such a significant fashion that it's slower than eagerly accepting all memory. The difference gets worse as the RAM size goes up, so I think there's some phase problem with the GHCB protocol not getting used early enough?
Thank you for testing. Let me see what I can find. I might have to rework Brijesh's original patches more to make use of the early boot GHCB in order to cut down on the number of MSR protocol requests.
Thanks,
Tom
Quoted 1 line
Re: [PATCH v1 2/2] x86/sev: Add SNP-specific unaccepted memory support
In reply to Tom Lendackyarch/x86/Kconfig config AMD_MEM_ENCRYPT · line 1559
Quoted 10 lines
arch/x86/Kconfig +1
-0@@ -1553,6 +1553,7 @@ config AMD_MEM_ENCRYPT select INSTRUCTION_DECODER select ARCH_HAS_CC_PLATFORM select X86_MEM_ENCRYPT + select UNACCEPTED_MEMORY help Say yes to enable support for the encryption of system memory. This requires an AMD processor that supports Secure Memory
At the risk of starting another centithread like on Kirill's patches for unaccepted memory, I think this needs to be brought up.
By making unaccepted_memory on option rather than a dependency, we get into an inescapable situation of always needing to know whether or not the guest OS will support unaccepted memory, from within the firmware. I think that makes a UEFI specification change necessary. If we don't make this configurable, and indeed make it a dependency, then we can say SEV-SNP implies that the firmware should create unaccepted memory. We can work around the short gap of support between kernel versions.
What are your thoughts on dependency versus UEFI spec change to allow this configuration to be negotiated with the firmware?
Signature · 1 line
-Dionna Glaze, PhD (she/her)
v2 cover Provide SEV-SNP support for unaccepted memory
In reply to Kirill A. ShutemovThis series adds SEV-SNP support for unaccepted memory to the patch series titled:
[PATCHv7 00/14] mm, x86/cc: Implement support for unaccepted memory
Currently, when changing the state of a page under SNP, the page state change structure is kmalloc()'d. This lead to hangs during boot when accepting memory because the allocation can trigger the need to accept more memory. So this series consists of two patches:
- A pre-patch to switch from a kmalloc()'d page state change structure to a (smaller) stack-based page state change structure.
- SNP support for unaccepted memory.
The series is based off of and tested against Kirill Shutemov's tree:
https://github.com/intel/tdx.git guest-unaccepted-memory
---
Changes since v1:
- Change from using a per-CPU PSC structure to a (smaller) stack PSC structure.
Tom Lendacky (2):
x86/sev: Put PSC struct on the stack in prep for unaccepted memory
support
x86/sev: Add SNP-specific unaccepted memory support
arch/x86/Kconfig | 1 + arch/x86/boot/compressed/mem.c | 3 +++ arch/x86/boot/compressed/sev.c | 10 +++++++- arch/x86/boot/compressed/sev.h | 23 ++++++++++++++++++ arch/x86/include/asm/sev-common.h | 2 +- arch/x86/include/asm/sev.h | 3 +++ arch/x86/kernel/sev.c | 40 ++++++++++++++++++++++++------- arch/x86/mm/unaccepted_memory.c | 4 ++++ 8 files changed, 76 insertions(+), 10 deletions(-) create mode 100644 arch/x86/boot/compressed/sev.h
Signature · 1 line
2.36.1
PATCH v2 1/2 x86/sev: Put PSC struct on the stack in prep for unaccepted memory support
In advance of providing support for unaccepted memory, switch from using kmalloc() for allocating the Page State Change (PSC) structure to using a local variable that lives on the stack. This is needed to avoid a possible recursive call into set_pages_state() if the kmalloc() call requires (more) memory to be accepted, which would result in a hang.
The current size of the PSC struct is 2,032 bytes. To make the struct more stack friendly, reduce the number of PSC entries from 253 down to 64, resulting in a size of 520 bytes. This is a nice compromise on struct size and total PSC requests.
Also, since set_pages_state() uses the per-CPU GHCB, add a static variable that indicates when per-CPU GHCBs are available. Until they are available, the GHCB MSR protocol is used to perform page state changes.
If either the reduction in PSC entries or the use of the MSR protocol until the per-CPU GHCBs are available, result in any kind of performance issue (that is not seen at the moment), use of a larger static PSC struct, with fallback to the smaller stack version, or use the boot GHCB prior to per-CPU GHCBs, can be investigated.
--- arch/x86/include/asm/sev-common.h | 2 +- arch/x86/kernel/sev.c | 24 ++++++++++++++++-------- 2 files changed, 17 insertions(+), 9 deletions(-)
arch/x86/include/asm/sev-common.h +1 -1
@@ -107,7 +107,7 @@ enum psc_op {#define GHCB_HV_FT_SNP_AP_CREATION BIT_ULL(1)/* SNP Page State Change NAE event */ -#define VMGEXIT_PSC_MAX_ENTRY 253 +#define VMGEXIT_PSC_MAX_ENTRY 64struct psc_hdr {u16 cur_entry;
arch/x86/kernel/sev.c +16 -8
@@ -66,6 +66,9 @@ static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);*/ static struct ghcb *boot_ghcb __section(".data"); +/* Flag to indicate when the first per-CPU GHCB is registered */ +static bool ghcb_percpu_ready __section(".data"); +/* Bitmap of SEV features supported by the hypervisor */ static u64 sev_hv_features__ro_after_init ;
@@ -660,7 +663,7 @@ static void pvalidate_pages(unsigned long vaddr, unsigned int npages, bool valid } } -static void __init early_set_pages_state (unsigned long paddr,unsigned int npages,enum psc_op op) +static void early_set_pages_state (unsigned long paddr,unsigned int npages,enum psc_op op) {unsigned long paddr_end;u64 val;
@@ -868,11 +871,16 @@ static void __set_pages_state(struct snp_psc_desc *data, unsigned long vaddr,static void set_pages_state (unsigned long vaddr,unsigned int npages,int op) {unsigned long vaddr_end, next_vaddr; -struct snp_psc_desc *desc; +struct snp_psc_desc desc; + +/* +* Use the MSR protocol when the per-CPU GHCBs are not yet registered, +* since vmgexit_psc() uses the per-CPU GHCB. +*/ +if (!ghcb_percpu_ready) +return early_set_pages_state (__pa (vaddr), npages, op); - desc =kmalloc (sizeof (*desc),GFP_KERNEL_ACCOUNT ); -if (!desc) -panic ("SNP: failed to allocate memory for PSC descriptor\n" ); +memset (&desc,0 ,sizeof (desc)); vaddr = vaddr &PAGE_MASK ; vaddr_end = vaddr + (npages <<PAGE_SHIFT );
@@ -882,12 +890,10 @@ static void set_pages_state(unsigned long vaddr, unsigned int npages, int op) next_vaddr = min_t(unsigned long, vaddr_end, (VMGEXIT_PSC_MAX_ENTRY *PAGE_SIZE ) + vaddr); -__set_pages_state (desc, vaddr, next_vaddr, op); +__set_pages_state (&desc, vaddr, next_vaddr, op); vaddr = next_vaddr; } - -kfree (desc); }void snp_set_memory_shared (unsigned long vaddr,unsigned int npages)
@@ -1254,6 +1260,8 @@ void setup_ghcb(void)if (cc_platform_has (CC_ATTR_GUEST_SEV_SNP ))snp_register_per_cpu_ghcb (); + ghcb_percpu_ready = true; +return ; }
Signature · 1 line
2.36.1
Re: [PATCH v2 1/2] x86/sev: Put PSC struct on the stack in prep for unaccepted memory support
On 8/8/22 10:16, Tom Lendacky wrote:
...
arch/x86/include/asm/sev-common.h enum psc_op { · line 110
Quoted 10 lines
arch/x86/include/asm/sev-common.h +1
-1@@ -107,7 +107,7 @@ enum psc_op {#define GHCB_HV_FT_SNP_AP_CREATION BIT_ULL(1)/* SNP Page State Change NAE event */ -#define VMGEXIT_PSC_MAX_ENTRY 253 +#define VMGEXIT_PSC_MAX_ENTRY 64
In general, the stack-based allocation looks fine. It might be worth a comment in there to make it clear that this can consume stack space.
arch/x86/kernel/sev.c static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE); · line 70
Quoted 12 lines
arch/x86/include/asm/sev-common.h +0
-0struct psc_hdr {u16 cur_entry;arch/x86/kernel/sev.c +2
-0@@ -66,6 +66,9 @@ static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);*/ static struct ghcb *boot_ghcb __section(".data"); +/* Flag to indicate when the first per-CPU GHCB is registered */ +static bool ghcb_percpu_ready __section(".data");
So, there's a code path that can't be entered until this is set? Seems like the least we can do it annotate that path with a WARN_ON_ONCE(!ghcb_percpu_ready).
Also, how does having _one_ global variable work for indicating the state of multiple per-cpu structures? The code doesn't seem to delay setting this variable until after _all_ of the per-cpu state is ready.
Quoted 20 lines
arch/x86/kernel/sev.c +4
-2/* Bitmap of SEV features supported by the hypervisor */ static u64 sev_hv_features__ro_after_init ;@@ -660,7 +663,7 @@ static void pvalidate_pages(unsigned long vaddr, unsigned int npages, bool valid } } -static void __init early_set_pages_state (unsigned long paddr,unsigned int npages,enum psc_op op) +static void early_set_pages_state (unsigned long paddr,unsigned int npages,enum psc_op op) {unsigned long paddr_end;u64 val;@@ -868,11 +871,16 @@ static void __set_pages_state(struct snp_psc_desc *data, unsigned long vaddr,static void set_pages_state (unsigned long vaddr,unsigned int npages,int op) {unsigned long vaddr_end, next_vaddr; -struct snp_psc_desc *desc; +struct snp_psc_desc desc; + +/*
arch/x86/kernel/sev.c static void __set_pages_state(struct snp_psc_desc *data, unsigned long vaddr, · line 883
Quoted 10 lines
arch/x86/kernel/sev.c +6
-3+* Use the MSR protocol when the per-CPU GHCBs are not yet registered, +* since vmgexit_psc() uses the per-CPU GHCB. +*/ +if (!ghcb_percpu_ready) +return early_set_pages_state (__pa (vaddr), npages, op); - desc =kmalloc (sizeof (*desc),GFP_KERNEL_ACCOUNT ); -if (!desc) -panic ("SNP: failed to allocate memory for PSC descriptor\n" ); +memset (&desc,0 ,sizeof (desc));
Why is this using memset()? The compiler should be smart enough to delay initializing 'desc' until after the return with this kind of construct:
struct snp_psc_desc desc = {};
if (foo)
return;
// use 'desc' hereThe compiler *knows* there is no access to 'desc' before the if().
Quoted 25 lines
arch/x86/kernel/sev.c +3
-3vaddr = vaddr &PAGE_MASK ; vaddr_end = vaddr + (npages <<PAGE_SHIFT );@@ -882,12 +890,10 @@ static void set_pages_state(unsigned long vaddr, unsigned int npages, int op) next_vaddr = min_t(unsigned long, vaddr_end, (VMGEXIT_PSC_MAX_ENTRY *PAGE_SIZE ) + vaddr); -__set_pages_state (desc, vaddr, next_vaddr, op); +__set_pages_state (&desc, vaddr, next_vaddr, op); vaddr = next_vaddr; } - -kfree (desc); }void snp_set_memory_shared (unsigned long vaddr,unsigned int npages)@@ -1254,6 +1260,8 @@ void setup_ghcb(void)if (cc_platform_has (CC_ATTR_GUEST_SEV_SNP ))snp_register_per_cpu_ghcb (); + ghcb_percpu_ready = true; +return ; }
Re: [PATCH v2 1/2] x86/sev: Put PSC struct on the stack in prep for unaccepted memory support
Quoted 9 lines · Dave Hansen
On 8/8/22 10:16, Tom Lendacky wrote:
...Quoted 7 lines
arch/x86/include/asm/sev-common.h +0
-0@@ -107,7 +107,7 @@ enum psc_op {#define GHCB_HV_FT_SNP_AP_CREATION BIT_ULL(1)
Quoted 6 lines
arch/x86/include/asm/sev-common.h enum psc_op { · line 110
Quoted 3 lines
arch/x86/include/asm/sev-common.h +1
-1/* SNP Page State Change NAE event */ -#define VMGEXIT_PSC_MAX_ENTRY 253 +#define VMGEXIT_PSC_MAX_ENTRY 64In general, the stack-based allocation looks fine. It might be worth a comment in there to make it clear that this can consume stack space.
I'll add that.
Quoted 11 lines
Quoted 11 lines
arch/x86/include/asm/sev-common.h +0
-0struct psc_hdr {u16 cur_entry;arch/x86/kernel/sev.c +0
-0@@ -66,6 +66,9 @@ static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);*/ static struct ghcb *boot_ghcb __section(".data");
Quoted 6 lines
arch/x86/kernel/sev.c static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE); · line 70
Quoted 2 lines
arch/x86/kernel/sev.c +2
-0+/* Flag to indicate when the first per-CPU GHCB is registered */ +static bool ghcb_percpu_ready __section(".data");So, there's a code path that can't be entered until this is set? Seems like the least we can do it annotate that path with a
WARN_ON_ONCE(!ghcb_percpu_ready).
Sure, that can be added. Right now the only function that calls vmgexit_psc() is covered (set_pages_state()/__set_pages_state()) and is doing the right thing. But I guess if anything is added in the future, that will provide details on what happened.
Quoted 4 lines
Also, how does having _one_ global variable work for indicating the state of multiple per-cpu structures? The code doesn't seem to delay setting this variable until after _all_ of the per-cpu state is ready.
All of the per-CPU GHCBs are allocated during the BSP boot, before any AP is started. The APs only ever run the kernel_exc_vmm_communication() #VC handler and only ever use the per-CPU version of the GHCB, never the early boot version. This is based on the initial_vc_handler being switched to the runtime #VC handler, kernel_exc_vmm_communication.
The trigger for the switch over for the BSP from the early boot GHCB to the per-CPU GHCB is during setup_ghcb() after the initial_vc_handler has been switched to kernel_exc_vmm_communication, which is just after the per-CPU allocations. By putting the setting of the ghcb_percpu_ready in setup_ghcb(), it indicates that the BSP per-CPU GHCB has been registered and can be used.
Quoted 36 lines
Quoted 21 lines
arch/x86/kernel/sev.c +4
-2/* Bitmap of SEV features supported by the hypervisor */ static u64 sev_hv_features__ro_after_init ;@@ -660,7 +663,7 @@ static void pvalidate_pages(unsigned long vaddr, unsigned int npages, bool valid } } -static void __init early_set_pages_state (unsigned long paddr,unsigned int npages,enum psc_op op) +static void early_set_pages_state (unsigned long paddr,unsigned int npages,enum psc_op op) {unsigned long paddr_end;u64 val;@@ -868,11 +871,16 @@ static void __set_pages_state(struct snp_psc_desc *data, unsigned long vaddr,static void set_pages_state (unsigned long vaddr,unsigned int npages,int op) {unsigned long vaddr_end, next_vaddr; -struct snp_psc_desc *desc; +struct snp_psc_desc desc; + +/* arch/x86/kernel/sev.c static void __set_pages_state(struct snp_psc_desc *data, unsigned long vaddr, · line 883
Quoted 10 lines
arch/x86/kernel/sev.c +6
-3+* Use the MSR protocol when the per-CPU GHCBs are not yet registered, +* since vmgexit_psc() uses the per-CPU GHCB. +*/ +if (!ghcb_percpu_ready) +return early_set_pages_state (__pa (vaddr), npages, op); - desc =kmalloc (sizeof (*desc),GFP_KERNEL_ACCOUNT ); -if (!desc) -panic ("SNP: failed to allocate memory for PSC descriptor\n" ); +memset (&desc,0 ,sizeof (desc));Why is this using
memset()? The compiler should be smart enough to delay initializing 'desc' until after the return with this kind of construct:
Quoted 6 lines
struct snp_psc_desc desc = {}; if (foo) return; // use 'desc' hereThe compiler *knows* there is no access to 'desc' before the
if().
Yup, I can change that.
Thanks,
Tom
Quoted 28 lines
Quoted 27 lines
arch/x86/kernel/sev.c +3
-3vaddr = vaddr &PAGE_MASK ; vaddr_end = vaddr + (npages <<PAGE_SHIFT );@@ -882,12 +890,10 @@ static void set_pages_state(unsigned long vaddr, unsigned int npages, int op) next_vaddr = min_t(unsigned long, vaddr_end, (VMGEXIT_PSC_MAX_ENTRY *PAGE_SIZE ) + vaddr); -__set_pages_state (desc, vaddr, next_vaddr, op); +__set_pages_state (&desc, vaddr, next_vaddr, op); vaddr = next_vaddr; } - -kfree (desc); }void snp_set_memory_shared (unsigned long vaddr,unsigned int npages)@@ -1254,6 +1260,8 @@ void setup_ghcb(void)if (cc_platform_has (CC_ATTR_GUEST_SEV_SNP ))snp_register_per_cpu_ghcb (); + ghcb_percpu_ready = true; +return ; }
Re: [PATCH v2 1/2] x86/sev: Put PSC struct on the stack in prep for unaccepted memory support
Quoted 23 lines · Tom Lendacky
Quoted 6 lines
Quoted 2 lines
+/* Flag to indicate when the first per-CPU GHCB is registered */ +static bool
ghcb_percpu_ready__section(".data");So, there's a code path that can't be entered until this is set? Seems like the least we can do it annotate that path with a
WARN_ON_ONCE(!ghcb_percpu_ready).Sure, that can be added. Right now the only function that calls
vmgexit_psc()is covered (set_pages_state()/__set_pages_state()) and is doing the right thing. But I guess if anything is added in the future, that will provide details on what happened.Quoted 4 lines
Also, how does having _one_ global variable work for indicating the state of multiple per-cpu structures? The code doesn't seem to delay setting this variable until after _all_ of the per-cpu state is ready.
All of the per-CPU GHCBs are allocated during the BSP boot, before any AP is started. The APs only ever run the
kernel_exc_vmm_communication()#VC handler and only ever use the per-CPU version of the GHCB, never the early boot version. This is based on theinitial_vc_handlerbeing switched to the runtime #VC handler,kernel_exc_vmm_communication.
Quoted 6 lines
The trigger for the switch over for the BSP from the early boot GHCB to the per-CPU GHCB is during
setup_ghcb()after theinitial_vc_handlerhas been switched tokernel_exc_vmm_communication, which is just after the per-CPU allocations. By putting the setting of theghcb_percpu_readyinsetup_ghcb(), it indicates that the BSP per-CPU GHCB has been registered and can be used.
That description makes the proposed comment even more confusing:
/* Flag to indicate when the first per-CPU GHCB is registered */
The important thing is that this variable is only _useful_ for the boot CPU. After the boot CPU has allocated space for _itself_, it can then go and stop using the MSR-based method.
The reason it's set after "the first" is because "the first" is also the boot CPU, but referring to it as the "the first" is a bit oblique. Maybe something like this:
/* * Set after the boot CPU's GHCB is registered. At that point, * it can be used for calls instead of MSRs. */
Re: [PATCH v2 1/2] x86/sev: Put PSC struct on the stack in prep for unaccepted memory support
Quoted 39 lines · Dave Hansen
Quoted 23 lines · Tom Lendacky
Quoted 6 lines
Quoted 2 lines
+/* Flag to indicate when the first per-CPU GHCB is registered */
+static boolghcb_percpu_ready__section(".data");So, there's a code path that can't be entered until this is set? Seems like the least we can do it annotate that path with a
WARN_ON_ONCE(!ghcb_percpu_ready).Sure, that can be added. Right now the only function that calls
vmgexit_psc()is covered (set_pages_state()/__set_pages_state()) and is doing the right thing. But I guess if anything is added in the future, that will provide details on what happened.Quoted 4 lines
Also, how does having _one_ global variable work for indicating the state of multiple per-cpu structures? The code doesn't seem to delay setting this variable until after _all_ of the per-cpu state is ready.
All of the per-CPU GHCBs are allocated during the BSP boot, before any AP is started. The APs only ever run the
kernel_exc_vmm_communication()#VC handler and only ever use the per-CPU version of the GHCB, never the early boot version. This is based on theinitial_vc_handlerbeing switched to the runtime #VC handler,kernel_exc_vmm_communication.Quoted 6 lines
The trigger for the switch over for the BSP from the early boot GHCB to the per-CPU GHCB is during
setup_ghcb()after theinitial_vc_handlerhas been switched tokernel_exc_vmm_communication, which is just after the per-CPU allocations. By putting the setting of theghcb_percpu_readyinsetup_ghcb(), it indicates that the BSP per-CPU GHCB has been registered and can be used.That description makes the proposed comment even more confusing:
/* Flag to indicate when the first per-CPU GHCB is registered */The important thing is that this variable is only _useful_ for the boot CPU. After the boot CPU has allocated space for _itself_, it can then go and stop using the MSR-based method.
Quoted 8 lines
The reason it's set after "the first" is because "the first" is also the boot CPU, but referring to it as the "the first" is a bit oblique. Maybe something like this:
/* * Set after the boot CPU's GHCB is registered. At that point, * it can be used for calls instead of MSRs. */
Sure, I'll work on the wording.
Thanks,
Tom
Borislav Petkov
maintainerRe: [PATCH v2 1/2] x86/sev: Put PSC struct on the stack in prep for unaccepted memory support
In reply to Tom LendackyQuoted 5 lines · Tom Lendacky
In advance of providing support for unaccepted memory, switch from using
kmalloc()for allocating the Page State Change (PSC) structure to using a local variable that lives on the stack. This is needed to avoid a possible recursive call intoset_pages_state()if thekmalloc()call requires (more) memory to be accepted, which would result in a hang.
I don't understand: kmalloc() allocates memory which is unaccepted?
Quoted 4 lines
The current size of the PSC struct is 2,032 bytes. To make the struct more stack friendly, reduce the number of PSC entries from 253 down to 64, resulting in a size of 520 bytes. This is a nice compromise on struct size and total PSC requests.
Why can't you simply allocate that one PSC page once at boot, accept the memory for it and use it throughout? Under locking, ofc, if multiple PSC calls need to happen in parallel...
Instead of limiting the PSC req size.
arch/x86/kernel/sev.c void setup_ghcb(void) · line 1263
Quoted 5 lines
arch/x86/kernel/sev.c +1
-0@@ -1254,6 +1260,8 @@ void setup_ghcb(void)if (cc_platform_has (CC_ATTR_GUEST_SEV_SNP ))snp_register_per_cpu_ghcb (); + ghcb_percpu_ready = true;
You know how I can't stand those random boolean vars stating something has been initialized?
Can't you at least use some field in struct ghcb.reserved_1[] or so which the spec can provide to OS use so that FW doesn't touch it?
And then stick a "percpu_ready" bit there.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v2 1/2] x86/sev: Put PSC struct on the stack in prep for unaccepted memory support
Quoted 8 lines · Borislav Petkov
Quoted 5 lines · Tom Lendacky
In advance of providing support for unaccepted memory, switch from using
kmalloc()for allocating the Page State Change (PSC) structure to using a local variable that lives on the stack. This is needed to avoid a possible recursive call intoset_pages_state()if thekmalloc()call requires (more) memory to be accepted, which would result in a hang.I don't understand:
kmalloc()allocates memory which is unaccepted?
In order to satisfy the kmalloc() some memory has to be accepted. So it tries to accept some additional memory, but we're already in the accept memory path... deadlock.
Quoted 11 lines
Quoted 5 lines
The current size of the PSC struct is 2,032 bytes. To make the struct more stack friendly, reduce the number of PSC entries from 253 down to 64, resulting in a size of 520 bytes. This is a nice compromise on struct size and total PSC requests.
Why can't you simply allocate that one PSC page once at boot, accept the memory for it and use it throughout? Under locking, ofc, if multiple PSC calls need to happen in parallel...
Instead of limiting the PSC req size.
There was a whole discussion on this and I would prefer to keep the ability to parallelize PSC without locking.
Quoted 5 lines
Quoted 5 lines
arch/x86/kernel/sev.c +0
-0@@ -1254,6 +1260,8 @@ void setup_ghcb(void)if (cc_platform_has (CC_ATTR_GUEST_SEV_SNP ))snp_register_per_cpu_ghcb ();
Quoted 7 lines
arch/x86/kernel/sev.c void setup_ghcb(void) · line 1263
Quoted 1 line
arch/x86/kernel/sev.c +1
-0+ ghcb_percpu_ready = true;You know how I can't stand those random boolean vars stating something has been initialized?
Can't you at least use some field in struct ghcb.reserved_1[] or so which the spec can provide to OS use so that FW doesn't touch it?
Well when we don't know which GHCB is in use, using that reserved area in the GHCB doesn't help. Also, I don't want to update the GHCB specification for a single bit that is only required because of the way Linux went about establishing the GHCB usage.
Thanks,
Tom
Quoted 5 lines
And then stick a "
percpu_ready" bit there.Thx.
Borislav Petkov
maintainerRe: [PATCH v2 1/2] x86/sev: Put PSC struct on the stack in prep for unaccepted memory support
Quoted 1 line · Tom Lendacky
There was a whole discussion on this
Pointer to it?
Quoted 2 lines
and I would prefer to keep the ability to parallelize PSC without locking.
So smaller, on-stack PSC but lockless is still better than a bigger one but with synchronized accesses to it?
Quoted 2 lines
Well when we don't know which GHCB is in use, using that reserved area in the GHCB doesn't help.
What do you mean?
The one which you read with
data = this_cpu_read(runtime_data);
in snp_register_per_cpu_ghcb() is the one you register.
Quoted 3 lines
Also, I don't want to update the GHCB specification for a single bit that is only required because of the way Linux went about establishing the GHCB usage.
Linux?
You mean, you did it this way: 885689e47dfa1499b756a07237eb645234d93cf9
:-)
"The runtime handler needs one GHCB per-CPU. Set them up and map them unencrypted."
Why does that handler need one GHCB per CPU?
As to the field, I was thinking along the lines of
struct ghcb.vendor_flags
field which each virt vendor can use however they like.
It might be overkill but a random bool ain't pretty either. Especially if those things start getting added for all kinds of other things.
If anything, you could make this a single u64 sev_flags which can at least collect all that gunk in one variable ... at least...
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v2 1/2] x86/sev: Put PSC struct on the stack in prep for unaccepted memory support
Quoted 4 lines · Borislav Petkov
Quoted 1 line · Tom Lendacky
There was a whole discussion on this
Pointer to it?
It starts here: https://lore.kernel.org/lkml/658c455c40e8950cb046dd885dd19dc1c52d060a.1659103274.git.thomas.lendacky@amd.com/
Quoted 7 lines
Quoted 3 lines
and I would prefer to keep the ability to parallelize PSC without locking.
So smaller, on-stack PSC but lockless is still better than a bigger one but with synchronized accesses to it?
Quoted 8 lines
Quoted 2 lines
Well when we don't know which GHCB is in use, using that reserved area in the GHCB doesn't help.
What do you mean?
The one which you read with
data = this_cpu_read(runtime_data);
Memory acceptance is called before the per-CPU GHCBs have been allocated and so you would be actually be using early boot GHCB. And that is decided based on the #VC handler that is invoked - but in this case we're not coming through the #VC handler to accept memory.
Quoted 7 lines
in
snp_register_per_cpu_ghcb()is the one you register.Quoted 3 lines
Also, I don't want to update the GHCB specification for a single bit that is only required because of the way Linux went about establishing the GHCB usage.
Quoted 5 lines
Linux?
You mean, you did it this way:
885689e47dfa1499b756a07237eb645234d93cf9:-)
Well Joerg re-worked all that quite a bit. And with the SNP support, the added requirement of registering the GHCB changed which GHCB could be used. So even when the per-CPU GHCB is allocated, it can't be used until it is registered, which depends on when the #VC handler is changed from the boot #VC handler to the runtime #VC handler.
Quoted 5 lines
"The runtime handler needs one GHCB per-CPU. Set them up and map them unencrypted."
Why does that handler need one GHCB per CPU?
Each vCPU can be handling a #VC and you don't want to be serializing on a single GHCB.
Thanks,
Tom
Quoted 15 lines
As to the field, I was thinking along the lines of
struct ghcb.vendor_flags
field which each virt vendor can use however they like.
It might be overkill but a random bool ain't pretty either. Especially if those things start getting added for all kinds of other things.
If anything, you could make this a single u64
sev_flagswhich can at least collect all that gunk in one variable ... at least...Thx.
Borislav Petkov
maintainerRe: [PATCH v2 1/2] x86/sev: Put PSC struct on the stack in prep for unaccepted memory support
Quoted 7 lines · Tom Lendacky
Quoted 4 lines · Borislav Petkov
Quoted 1 line · Tom Lendacky
There was a whole discussion on this
Pointer to it?
It starts here: https://lore.kernel.org/lkml/658c455c40e8950cb046dd885dd19dc1c52d060a.1659103274.git.thomas.lendacky@amd.com/
So how come none of the rationale for the on-stack decision vs a single buffer with a spinlock protection hasn't made it to this patch?
We need to have the reason why this thing is changed documented somewhere.
Quoted 2 lines
Quoted 2 lines
So smaller, on-stack PSC but lockless is still better than a bigger one but with synchronized accesses to it?
That thing.
That decision for on-stack buffer needs explaining why.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Well when we don't know which GHCB is in use, using that reserved area in the GHCB doesn't help.
What do you mean?
Quoted 8 lines
Quoted 3 lines
The one which you read with
data = this_cpu_read(runtime_data);Memory acceptance is called before the per-CPU GHCBs have been allocated and so you would be actually be using early boot GHCB. And that is decided based on the #VC handler that is invoked - but in this case we're not coming through the #VC handler to accept memory.
But then ghcb_percpu_ready needs to be a per-CPU variable too! Because it is set right after snp_register_per_cpu_ghcb() which works on the *per-CPU* GHCB.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v2 1/2] x86/sev: Put PSC struct on the stack in prep for unaccepted memory support
Quoted 7 lines · Borislav Petkov
Quoted 6 lines · Tom Lendacky
Quoted 4 lines · Borislav Petkov
Quoted 1 line · Tom Lendacky
There was a whole discussion on this
Pointer to it?
Quoted 7 lines
Quoted 1 line
It starts here: https://lore.kernel.org/lkml/658c455c40e8950cb046dd885dd19dc1c52d060a.1659103274.git.thomas.lendacky@amd.com/
So how come none of the rationale for the on-stack decision vs a single buffer with a spinlock protection hasn't made it to this patch?
We need to have the reason why this thing is changed documented somewhere.
Yup, was all being addressed in v3 based on Dave's comments.
Quoted 17 lines
Quoted 3 lines
Quoted 3 lines
So smaller, on-stack PSC but lockless is still better than a bigger one but with synchronized accesses to it?
That thing.
That decision for on-stack buffer needs explaining why.
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines
Well when we don't know which GHCB is in use, using that reserved area in the GHCB doesn't help.
What do you mean?
The one which you read with
data = this_cpu_read(runtime_data);
Quoted 8 lines
Quoted 4 lines
Memory acceptance is called before the per-CPU GHCBs have been allocated and so you would be actually be using early boot GHCB. And that is decided based on the #VC handler that is invoked - but in this case we're not coming through the #VC handler to accept memory.
But then
ghcb_percpu_readyneeds to be a per-CPU variable too! Because it is set right aftersnp_register_per_cpu_ghcb()which works on the *per-CPU* GHCB.
No, and the code comment will explain this. Since the APs only ever use the per-CPU GHCB there is no concern as to when there is a switch over from the early boot GHCB to the per-CPU GHCB, so a single global variable is all that is needed.
I'll send out v3 soon.
Thanks,
Tom
Quoted 1 line
PATCH v2 2/2 x86/sev: Add SNP-specific unaccepted memory support
In reply to Tom LendackyAdd SNP-specific hooks to the unaccepted memory support in the boot path (__accept_memory()) and the core kernel (accept_memory()) in order to support booting SNP guests when unaccepted memory is present. Without this support, SNP guests will fail to boot and/or panic() when unaccepted memory is present in the EFI memory map.
The process of accepting memory under SNP involves invoking the hypervisor to perform a page state change for the page to private memory and then issuing a PVALIDATE instruction to accept the page.
Create the new header file arch/x86/boot/compressed/sev.h because adding the function declaration to any of the existing SEV related header files pulls in too many other header files, causing the build to fail.
--- arch/x86/Kconfig | 1 + arch/x86/boot/compressed/mem.c | 3 +++ arch/x86/boot/compressed/sev.c | 10 +++++++++- arch/x86/boot/compressed/sev.h | 23 +++++++++++++++++++++++ arch/x86/include/asm/sev.h | 3 +++ arch/x86/kernel/sev.c | 16 ++++++++++++++++ arch/x86/mm/unaccepted_memory.c | 4 ++++ 7 files changed, 59 insertions(+), 1 deletion(-) create mode 100644 arch/x86/boot/compressed/sev.h
arch/x86/Kconfig +1 -0
@@ -1553,6 +1553,7 @@ config AMD_MEM_ENCRYPT select INSTRUCTION_DECODER select ARCH_HAS_CC_PLATFORM select X86_MEM_ENCRYPT + select UNACCEPTED_MEMORY help Say yes to enable support for the encryption of system memory. This requires an AMD processor that supports Secure Memory
arch/x86/boot/compressed/mem.c +3 -0
@@ -6,6 +6,7 @@#include "find.h" #include "math.h" #include "tdx.h" +#include "sev.h" #include <asm/shared/tdx.h> #define PMD_SHIFT 21
@@ -39,6 +40,8 @@ static inline void __accept_memory(phys_addr_t start, phys_addr_t end)/* Platform-specific memory-acceptance call goes here */ if (is_tdx_guest ())tdx_accept_memory (start, end); +else if (sev_snp_enabled ()) +snp_accept_memory (start, end);else error ("Cannot accept memory: unknown platform\n" ); }
arch/x86/boot/compressed/sev.c +9 -1
@@ -115,7 +115,7 @@ static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,/* Include code for early handlers */ #include "../../kernel/sev-shared.c" -static inline bool sev_snp_enabled (void ) +bool sev_snp_enabled (void ) {return sev_status &MSR_AMD64_SEV_SNP_ENABLED ; }
@@ -161,6 +161,14 @@ void snp_set_page_shared(unsigned long paddr)__page_state_change (paddr,SNP_PAGE_STATE_SHARED ); } +void snp_accept_memory (phys_addr_t start,phys_addr_t end) +{ +while (end > start) { +snp_set_page_private (start); + start +=PAGE_SIZE ; + } +} +static bool early_setup_ghcb (void ) {if (set_page_decrypted ((unsigned long )&boot_ghcb_page))
arch/x86/boot/compressed/sev.h new file +23 -0
@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* AMD SEV header for early boot related functions. +* +* Author: Tom Lendacky <[email protected]> +*/ + +#ifndef BOOT_COMPRESSED_SEV_H +#define BOOT_COMPRESSED_SEV_H + +#ifdef CONFIG_AMD_MEM_ENCRYPT + +bool sev_snp_enabled (void ); +void snp_accept_memory (phys_addr_t start,phys_addr_t end); + +#else + +static inline bool sev_snp_enabled (void ) {return false; } +static inline void snp_accept_memory (phys_addr_t start,phys_addr_t end) { } + +#endif + +#endif
arch/x86/include/asm/sev.h +3 -0
@@ -202,6 +202,7 @@ void snp_set_wakeup_secondary_cpu(void);bool snp_init (struct boot_params *bp);void snp_abort (void );int snp_issue_guest_request (u64 exit_code,struct snp_req_data *input,unsigned long *fw_err); +void snp_accept_memory (phys_addr_t start,phys_addr_t end);#else static inline void sev_es_ist_enter (struct pt_regs *regs) { }static inline void sev_es_ist_exit (void ) { }
@@ -226,6 +227,8 @@ static inline int snp_issue_guest_request(u64 exit_code, struct snp_req_data *in {return -ENOTTY ; } + +static inline void snp_accept_memory (phys_addr_t start,phys_addr_t end) { }#endif #endif
arch/x86/kernel/sev.c +16 -0
@@ -916,6 +916,22 @@ void snp_set_memory_private(unsigned long vaddr, unsigned int npages)pvalidate_pages (vaddr, npages, true); } +void snp_accept_memory (phys_addr_t start,phys_addr_t end) +{ +unsigned long vaddr; +unsigned int npages; + +if (!cc_platform_has (CC_ATTR_GUEST_SEV_SNP )) +return ; + + vaddr = (unsigned long )__va (start); + npages = (end - start) >>PAGE_SHIFT ; + +set_pages_state (vaddr, npages,SNP_PAGE_STATE_PRIVATE ); + +pvalidate_pages (vaddr, npages, true); +} +static int snp_set_vmsa (void *va,bool vmsa) {u64 attrs;
arch/x86/mm/unaccepted_memory.c +4 -0
@@ -9,6 +9,7 @@#include <asm/setup.h> #include <asm/shared/tdx.h> #include <asm/unaccepted_memory.h> +#include <asm/sev.h> /* Protects unaccepted memory bitmap */ static DEFINE_SPINLOCK (unaccepted_memory_lock);
@@ -66,6 +67,9 @@ void accept_memory(phys_addr_t start, phys_addr_t end)if (cpu_feature_enabled (X86_FEATURE_TDX_GUEST )) {tdx_accept_memory (range_start *PMD_SIZE , range_end *PMD_SIZE ); + }else if (cc_platform_has (CC_ATTR_GUEST_SEV_SNP )) { +snp_accept_memory (range_start *PMD_SIZE , + range_end *PMD_SIZE ); }else {panic ("Cannot accept memory: unknown platform\n" ); }
Signature · 1 line
2.36.1
v3 cover Provide SEV-SNP support for unaccepted memory
In reply to Kirill A. ShutemovThis series adds SEV-SNP support for unaccepted memory to the patch series titled:
[PATCHv7 00/14] mm, x86/cc: Implement support for unaccepted memory
Currently, when changing the state of a page under SNP, the page state change structure is kmalloc()'d. This lead to hangs during boot when accepting memory because the allocation can trigger the need to accept more memory. So this series consists of two patches:
- A pre-patch to switch from a kmalloc()'d page state change structure to a (smaller) stack-based page state change structure.
- SNP support for unaccepted memory.
The series is based off of and tested against Kirill Shutemov's tree:
https://github.com/intel/tdx.git guest-unaccepted-memory
---
Changes since v2:
- Improve code comments in regards to when to use the per-CPU GHCB vs the MSR protocol and why a single global value is valid for both the BSP and APs.
- Add a comment related to the number of PSC entries and how it can impact the size of the struct and, therefore, stack usage.
- Add a WARN_ON_ONCE() for invoking vmgexit_psc() when per-CPU GHCBs haven't been created or registered, yet.
- Use the compiler support for clearing the PSC struct instead of issuing memset().
Changes since v1:
- Change from using a per-CPU PSC structure to a (smaller) stack PSC structure.
Tom Lendacky (2):
x86/sev: Put PSC struct on the stack in prep for unaccepted memory
support
x86/sev: Add SNP-specific unaccepted memory support
arch/x86/Kconfig | 1 + arch/x86/boot/compressed/mem.c | 3 ++ arch/x86/boot/compressed/sev.c | 10 ++++++- arch/x86/boot/compressed/sev.h | 23 +++++++++++++++ arch/x86/include/asm/sev-common.h | 9 ++++-- arch/x86/include/asm/sev.h | 3 ++ arch/x86/kernel/sev.c | 48 +++++++++++++++++++++++++------ arch/x86/mm/unaccepted_memory.c | 4 +++ 8 files changed, 90 insertions(+), 11 deletions(-) create mode 100644 arch/x86/boot/compressed/sev.h
Signature · 1 line
2.36.1
PATCH v3 1/2 x86/sev: Put PSC struct on the stack in prep for unaccepted memory support
In advance of providing support for unaccepted memory, switch from using kmalloc() for allocating the Page State Change (PSC) structure to using a local variable that lives on the stack. This is needed to avoid a possible recursive call into set_pages_state() if the kmalloc() call requires (more) memory to be accepted, which would result in a hang.
The current size of the PSC struct is 2,032 bytes. To make the struct more stack friendly, reduce the number of PSC entries from 253 down to 64, resulting in a size of 520 bytes. This is a nice compromise on struct size and total PSC requests while still allowing parallel PSC operations across vCPUs.
Also, since set_pages_state() uses the per-CPU GHCB, add a static variable that indicates when per-CPU GHCBs are available. Until they are available, the GHCB MSR protocol is used to perform page state changes. For APs, the per-CPU GHCB is created before they are started and registered upon startup, so this flag can be used globally for the BSP and APs instead of creating a per-CPU flag.
If either the reduction in PSC entries or the use of the MSR protocol until the per-CPU GHCBs are available, result in any kind of performance issue (that is not seen at the moment), use of a larger static PSC struct, with fallback to the smaller stack version, or use the boot GHCB prior to per-CPU GHCBs, can be investigated.
--- arch/x86/include/asm/sev-common.h | 9 +++++++-- arch/x86/kernel/sev.c | 32 +++++++++++++++++++++++-------- 2 files changed, 31 insertions(+), 10 deletions(-)
arch/x86/include/asm/sev-common.h +7 -2
@@ -106,8 +106,13 @@ enum psc_op {#define GHCB_HV_FT_SNP BIT_ULL(0)#define GHCB_HV_FT_SNP_AP_CREATION BIT_ULL(1) -/* SNP Page State Change NAE event */ -#define VMGEXIT_PSC_MAX_ENTRY 253 +/* +* SNP Page State Change NAE event +* The VMGEXIT_PSC_MAX_ENTRY determines the size of the PSC structure, +* which is a local variable (stack usage) in set_pages_state(). Do not +* increase this value without evaluating the impact to stack usage. +*/ +#define VMGEXIT_PSC_MAX_ENTRY 64struct psc_hdr {u16 cur_entry;
arch/x86/kernel/sev.c +24 -8
@@ -66,6 +66,17 @@ static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);*/ static struct ghcb *boot_ghcb __section(".data"); +/* +* A flag used by set_pages_state() that indicates when the per-CPU GHCB has +* been created and registered and thus can be used instead of using the MSR +* protocol. The set_pages_state() function eventually invokes vmgexit_psc(), +* which only works with a per-CPU GHCB. +* +* For APs, the per-CPU GHCB is created before they are started and registered +* upon startup, so this flag can be used globally for the BSP and APs. +*/ +static bool ghcb_percpu_ready __section(".data"); +/* Bitmap of SEV features supported by the hypervisor */ static u64 sev_hv_features__ro_after_init ;
@@ -660,7 +671,7 @@ static void pvalidate_pages(unsigned long vaddr, unsigned int npages, bool valid } } -static void __init early_set_pages_state (unsigned long paddr,unsigned int npages,enum psc_op op) +static void early_set_pages_state (unsigned long paddr,unsigned int npages,enum psc_op op) {unsigned long paddr_end;u64 val;
@@ -751,6 +762,8 @@ static int vmgexit_psc(struct snp_psc_desc *desc)unsigned long flags;struct ghcb *ghcb; +WARN_ON_ONCE (!ghcb_percpu_ready); +/* * __sev_get_ghcb() needs to run with IRQs disabled because it is using * a per-CPU GHCB.
@@ -868,11 +881,14 @@ static void __set_pages_state(struct snp_psc_desc *data, unsigned long vaddr,static void set_pages_state (unsigned long vaddr,unsigned int npages,int op) {unsigned long vaddr_end, next_vaddr; -struct snp_psc_desc *desc; +struct snp_psc_desc desc = {}; - desc =kmalloc (sizeof (*desc),GFP_KERNEL_ACCOUNT ); -if (!desc) -panic ("SNP: failed to allocate memory for PSC descriptor\n" ); +/* +* Use the MSR protocol when the per-CPU GHCBs are not yet registered, +* since vmgexit_psc() uses the per-CPU GHCB. +*/ +if (!ghcb_percpu_ready) +return early_set_pages_state (__pa (vaddr), npages, op); vaddr = vaddr &PAGE_MASK ; vaddr_end = vaddr + (npages <<PAGE_SHIFT );
@@ -882,12 +898,10 @@ static void set_pages_state(unsigned long vaddr, unsigned int npages, int op) next_vaddr = min_t(unsigned long, vaddr_end, (VMGEXIT_PSC_MAX_ENTRY *PAGE_SIZE ) + vaddr); -__set_pages_state (desc, vaddr, next_vaddr, op); +__set_pages_state (&desc, vaddr, next_vaddr, op); vaddr = next_vaddr; } - -kfree (desc); }void snp_set_memory_shared (unsigned long vaddr,unsigned int npages)
@@ -1254,6 +1268,8 @@ void setup_ghcb(void)if (cc_platform_has (CC_ATTR_GUEST_SEV_SNP ))snp_register_per_cpu_ghcb (); + ghcb_percpu_ready = true; +return ; }
Signature · 1 line
2.36.1
Borislav Petkov
maintainerRe: [PATCH v3 1/2] x86/sev: Put PSC struct on the stack in prep for unaccepted memory support
Quoted 7 lines · Tom Lendacky
arch/x86/kernel/sev.c +0
-0@@ -66,6 +66,17 @@ static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);*/ static struct ghcb *boot_ghcb __section(".data");
arch/x86/kernel/sev.c static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE); · line 77
Quoted 10 lines
arch/x86/kernel/sev.c +9
-0+/* +* A flag used by set_pages_state() that indicates when the per-CPU GHCB has +* been created and registered and thus can be used instead of using the MSR +* protocol. The set_pages_state() function eventually invokes vmgexit_psc(), +* which only works with a per-CPU GHCB. +* +* For APs, the per-CPU GHCB is created before they are started and registered +* upon startup, so this flag can be used globally for the BSP and APs. +*/
Ok, better, thanks!
arch/x86/kernel/sev.c static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE); · line 78
Quoted 1 line
arch/x86/kernel/sev.c +1
-0+static bool ghcb_percpu_ready __section(".data");
However, it reads really weird if you have "percpu" in the name of a variable which is not per CPU...
Let's just call it "ghcbs_initialized" and be done with it.
And I still hate the whole thing ofc.
Do this ontop (and I knew we had a flags thing already):
(And yes, __read_mostly is in the .data section too).
---
arch/x86/kernel/sev.c +15 -15
@@ -66,17 +66,6 @@ static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);*/ static struct ghcb *boot_ghcb __section(".data"); -/* -* A flag used by set_pages_state() that indicates when the per-CPU GHCB has -* been created and registered and thus can be used instead of using the MSR -* protocol. The set_pages_state() function eventually invokes vmgexit_psc(), -* which only works with a per-CPU GHCB. -* -* For APs, the per-CPU GHCB is created before they are started and registered -* upon startup, so this flag can be used globally for the BSP and APs. -*/ -static bool ghcb_percpu_ready __section(".data"); -/* Bitmap of SEV features supported by the hypervisor */ static u64 sev_hv_features__ro_after_init ;
@@ -128,7 +117,18 @@ static DEFINE_PER_CPU(struct sev_es_save_area *, sev_vmsa);struct sev_config {__u64 debug :1 , - __reserved :63 ; + +/* +* A flag used by set_pages_state() that indicates when the per-CPU GHCB has +* been created and registered and thus can be used instead of using the MSR +* protocol. The set_pages_state() function eventually invokes vmgexit_psc(), +* which only works with a per-CPU GHCB. +* +* For APs, the per-CPU GHCB is created before they are started and registered +* upon startup, so this flag can be used globally for the BSP and APs. +*/ + ghcbs_initialized :1 , + __reserved :62 ; };static struct sev_config sev_cfg__read_mostly ;
@@ -762,7 +762,7 @@ static int vmgexit_psc(struct snp_psc_desc *desc)unsigned long flags;struct ghcb *ghcb; -WARN_ON_ONCE (!ghcb_percpu_ready); +WARN_ON_ONCE (!sev_cfg.ghcbs_initialized);/* * __sev_get_ghcb() needs to run with IRQs disabled because it is using
@@ -887,7 +887,7 @@ static void set_pages_state(unsigned long vaddr, unsigned int npages, int op)* Use the MSR protocol when the per-CPU GHCBs are not yet registered, * since vmgexit_psc() uses the per-CPU GHCB. */ -if (!ghcb_percpu_ready) +if (!sev_cfg.ghcbs_initialized)return early_set_pages_state (__pa (vaddr), npages, op); vaddr = vaddr &PAGE_MASK ;
@@ -1268,7 +1268,7 @@ void setup_ghcb(void)if (cc_platform_has (CC_ATTR_GUEST_SEV_SNP ))snp_register_per_cpu_ghcb (); - ghcb_percpu_ready = true; + sev_cfg.ghcbs_initialized = true;return ; }
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v3 1/2] x86/sev: Put PSC struct on the stack in prep for unaccepted memory support
Quoted 26 lines · Borislav Petkov
Quoted 7 lines · Tom Lendacky
arch/x86/kernel/sev.c +0
-0@@ -66,6 +66,17 @@ static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);*/ static struct ghcb *boot_ghcb __section(".data");arch/x86/kernel/sev.c static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE); · line 77
Quoted 10 lines
arch/x86/kernel/sev.c +9
-0+/* +* A flag used by set_pages_state() that indicates when the per-CPU GHCB has +* been created and registered and thus can be used instead of using the MSR +* protocol. The set_pages_state() function eventually invokes vmgexit_psc(), +* which only works with a per-CPU GHCB. +* +* For APs, the per-CPU GHCB is created before they are started and registered +* upon startup, so this flag can be used globally for the BSP and APs. +*/ Ok, better, thanks!
arch/x86/kernel/sev.c static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE); · line 78
Quoted 1 line
arch/x86/kernel/sev.c +1
-0+static bool ghcb_percpu_ready __section(".data");However, it reads really weird if you have "percpu" in the name of a variable which is not per CPU...
Quoted 7 lines
Let's just call it "
ghcbs_initialized" and be done with it.And I still hate the whole thing ofc.
Do this ontop (and I knew we had a flags thing already):
(And yes,
__read_mostlyis in the .data section too).
Cool, will do.
Thanks,
Tom
Quoted 72 lines
---
arch/x86/kernel/sev.c +15
-15@@ -66,17 +66,6 @@ static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);*/ static struct ghcb *boot_ghcb __section(".data"); -/* -* A flag used by set_pages_state() that indicates when the per-CPU GHCB has -* been created and registered and thus can be used instead of using the MSR -* protocol. The set_pages_state() function eventually invokes vmgexit_psc(), -* which only works with a per-CPU GHCB. -* -* For APs, the per-CPU GHCB is created before they are started and registered -* upon startup, so this flag can be used globally for the BSP and APs. -*/ -static bool ghcb_percpu_ready __section(".data"); -/* Bitmap of SEV features supported by the hypervisor */ static u64 sev_hv_features__ro_after_init ;@@ -128,7 +117,18 @@ static DEFINE_PER_CPU(struct sev_es_save_area *, sev_vmsa);struct sev_config {__u64 debug :1 , - __reserved :63 ; + +/* +* A flag used by set_pages_state() that indicates when the per-CPU GHCB has +* been created and registered and thus can be used instead of using the MSR +* protocol. The set_pages_state() function eventually invokes vmgexit_psc(), +* which only works with a per-CPU GHCB. +* +* For APs, the per-CPU GHCB is created before they are started and registered +* upon startup, so this flag can be used globally for the BSP and APs. +*/ + ghcbs_initialized :1 , + __reserved :62 ; };static struct sev_config sev_cfg__read_mostly ;@@ -762,7 +762,7 @@ static int vmgexit_psc(struct snp_psc_desc *desc)unsigned long flags;struct ghcb *ghcb; -WARN_ON_ONCE (!ghcb_percpu_ready); +WARN_ON_ONCE (!sev_cfg.ghcbs_initialized);/* * __sev_get_ghcb() needs to run with IRQs disabled because it is using @@ -887,7 +887,7 @@ static void set_pages_state(unsigned long vaddr, unsigned int npages, int op)* Use the MSR protocol when the per-CPU GHCBs are not yet registered, * since vmgexit_psc() uses the per-CPU GHCB. */ -if (!ghcb_percpu_ready) +if (!sev_cfg.ghcbs_initialized)return early_set_pages_state (__pa (vaddr), npages, op); vaddr = vaddr &PAGE_MASK ;@@ -1268,7 +1268,7 @@ void setup_ghcb(void)if (cc_platform_has (CC_ATTR_GUEST_SEV_SNP ))snp_register_per_cpu_ghcb (); - ghcb_percpu_ready = true; + sev_cfg.ghcbs_initialized = true;return ; }
PATCH v3 2/2 x86/sev: Add SNP-specific unaccepted memory support
In reply to Tom LendackyAdd SNP-specific hooks to the unaccepted memory support in the boot path (__accept_memory()) and the core kernel (accept_memory()) in order to support booting SNP guests when unaccepted memory is present. Without this support, SNP guests will fail to boot and/or panic() when unaccepted memory is present in the EFI memory map.
The process of accepting memory under SNP involves invoking the hypervisor to perform a page state change for the page to private memory and then issuing a PVALIDATE instruction to accept the page.
Create the new header file arch/x86/boot/compressed/sev.h because adding the function declaration to any of the existing SEV related header files pulls in too many other header files, causing the build to fail.
--- arch/x86/Kconfig | 1 + arch/x86/boot/compressed/mem.c | 3 +++ arch/x86/boot/compressed/sev.c | 10 +++++++++- arch/x86/boot/compressed/sev.h | 23 +++++++++++++++++++++++ arch/x86/include/asm/sev.h | 3 +++ arch/x86/kernel/sev.c | 16 ++++++++++++++++ arch/x86/mm/unaccepted_memory.c | 4 ++++ 7 files changed, 59 insertions(+), 1 deletion(-) create mode 100644 arch/x86/boot/compressed/sev.h
arch/x86/Kconfig +1 -0
@@ -1553,6 +1553,7 @@ config AMD_MEM_ENCRYPT select INSTRUCTION_DECODER select ARCH_HAS_CC_PLATFORM select X86_MEM_ENCRYPT + select UNACCEPTED_MEMORY help Say yes to enable support for the encryption of system memory. This requires an AMD processor that supports Secure Memory
arch/x86/boot/compressed/mem.c +3 -0
@@ -6,6 +6,7 @@#include "find.h" #include "math.h" #include "tdx.h" +#include "sev.h" #include <asm/shared/tdx.h> #define PMD_SHIFT 21
@@ -39,6 +40,8 @@ static inline void __accept_memory(phys_addr_t start, phys_addr_t end)/* Platform-specific memory-acceptance call goes here */ if (is_tdx_guest ())tdx_accept_memory (start, end); +else if (sev_snp_enabled ()) +snp_accept_memory (start, end);else error ("Cannot accept memory: unknown platform\n" ); }
arch/x86/boot/compressed/sev.c +9 -1
@@ -115,7 +115,7 @@ static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,/* Include code for early handlers */ #include "../../kernel/sev-shared.c" -static inline bool sev_snp_enabled (void ) +bool sev_snp_enabled (void ) {return sev_status &MSR_AMD64_SEV_SNP_ENABLED ; }
@@ -161,6 +161,14 @@ void snp_set_page_shared(unsigned long paddr)__page_state_change (paddr,SNP_PAGE_STATE_SHARED ); } +void snp_accept_memory (phys_addr_t start,phys_addr_t end) +{ +while (end > start) { +snp_set_page_private (start); + start +=PAGE_SIZE ; + } +} +static bool early_setup_ghcb (void ) {if (set_page_decrypted ((unsigned long )&boot_ghcb_page))
arch/x86/boot/compressed/sev.h new file +23 -0
@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* AMD SEV header for early boot related functions. +* +* Author: Tom Lendacky <[email protected]> +*/ + +#ifndef BOOT_COMPRESSED_SEV_H +#define BOOT_COMPRESSED_SEV_H + +#ifdef CONFIG_AMD_MEM_ENCRYPT + +bool sev_snp_enabled (void ); +void snp_accept_memory (phys_addr_t start,phys_addr_t end); + +#else + +static inline bool sev_snp_enabled (void ) {return false; } +static inline void snp_accept_memory (phys_addr_t start,phys_addr_t end) { } + +#endif + +#endif
arch/x86/include/asm/sev.h +3 -0
@@ -202,6 +202,7 @@ void snp_set_wakeup_secondary_cpu(void);bool snp_init (struct boot_params *bp);void snp_abort (void );int snp_issue_guest_request (u64 exit_code,struct snp_req_data *input,unsigned long *fw_err); +void snp_accept_memory (phys_addr_t start,phys_addr_t end);#else static inline void sev_es_ist_enter (struct pt_regs *regs) { }static inline void sev_es_ist_exit (void ) { }
@@ -226,6 +227,8 @@ static inline int snp_issue_guest_request(u64 exit_code, struct snp_req_data *in {return -ENOTTY ; } + +static inline void snp_accept_memory (phys_addr_t start,phys_addr_t end) { }#endif #endif
arch/x86/kernel/sev.c +16 -0
@@ -924,6 +924,22 @@ void snp_set_memory_private(unsigned long vaddr, unsigned int npages)pvalidate_pages (vaddr, npages, true); } +void snp_accept_memory (phys_addr_t start,phys_addr_t end) +{ +unsigned long vaddr; +unsigned int npages; + +if (!cc_platform_has (CC_ATTR_GUEST_SEV_SNP )) +return ; + + vaddr = (unsigned long )__va (start); + npages = (end - start) >>PAGE_SHIFT ; + +set_pages_state (vaddr, npages,SNP_PAGE_STATE_PRIVATE ); + +pvalidate_pages (vaddr, npages, true); +} +static int snp_set_vmsa (void *va,bool vmsa) {u64 attrs;
arch/x86/mm/unaccepted_memory.c +4 -0
@@ -9,6 +9,7 @@#include <asm/setup.h> #include <asm/shared/tdx.h> #include <asm/unaccepted_memory.h> +#include <asm/sev.h> /* Protects unaccepted memory bitmap */ static DEFINE_SPINLOCK (unaccepted_memory_lock);
@@ -66,6 +67,9 @@ void accept_memory(phys_addr_t start, phys_addr_t end)if (cpu_feature_enabled (X86_FEATURE_TDX_GUEST )) {tdx_accept_memory (range_start *PMD_SIZE , range_end *PMD_SIZE ); + }else if (cc_platform_has (CC_ATTR_GUEST_SEV_SNP )) { +snp_accept_memory (range_start *PMD_SIZE , + range_end *PMD_SIZE ); }else {panic ("Cannot accept memory: unknown platform\n" ); }
Signature · 1 line
2.36.1
Borislav Petkov
maintainerRe: [PATCH v3 2/2] x86/sev: Add SNP-specific unaccepted memory support
Quoted 15 lines · Tom Lendacky
Add SNP-specific hooks to the unaccepted memory support in the boot path (
__accept_memory()) and the core kernel (accept_memory()) in order to support booting SNP guests when unaccepted memory is present. Without this support, SNP guests will fail to boot and/orpanic()when unaccepted memory is present in the EFI memory map.The process of accepting memory under SNP involves invoking the hypervisor to perform a page state change for the page to private memory and then issuing a PVALIDATE instruction to accept the page.
Create the new header file
arch/x86/boot/compressed/sev.hbecause adding the function declaration to any of the existing SEV related header files pulls in too many other header files, causing the build to fail.Signed-off-by Tom Lendacky <[email protected]>
Quoted 10 lines
--- arch/x86/Kconfig | 1 + arch/x86/boot/compressed/mem.c | 3 +++ arch/x86/boot/compressed/sev.c | 10 +++++++++- arch/x86/boot/compressed/sev.h | 23 +++++++++++++++++++++++ arch/x86/include/asm/sev.h | 3 +++ arch/x86/kernel/sev.c | 16 ++++++++++++++++ arch/x86/mm/unaccepted_memory.c | 4 ++++ 7 files changed, 59 insertions(+), 1 deletion(-) create mode 100644 arch/x86/boot/compressed/sev.h
Looks mostly ok to me...
arch/x86/boot/compressed/sev.c static enum es_result vc_read_mem(struct es_em_ctxt *ctxt, · line 121
Quoted 13 lines
arch/x86/boot/compressed/sev.c +1
-1@@ -115,7 +115,7 @@ static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,/* Include code for early handlers */ #include "../../kernel/sev-shared.c" -static inline bool sev_snp_enabled (void ) +bool sev_snp_enabled (void ) {return sev_status &MSR_AMD64_SEV_SNP_ENABLED ; }
This is another one of my pet peeves and now it even gets exported but it is the early decompressor crap so I won't even try to mention cc_* helpers...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettev4 cover Provide SEV-SNP support for unaccepted memory
In reply to Kirill A. ShutemovThis series adds SEV-SNP support for unaccepted memory to the patch series titled:
[PATCHv7 00/14] mm, x86/cc: Implement support for unaccepted memory
Currently, when changing the state of a page under SNP, the page state change structure is kmalloc()'d. This lead to hangs during boot when accepting memory because the allocation can trigger the need to accept more memory. Additionally, the page state change operations are not optimized under Linux since it was expected that all memory has been validated already, resulting in poor performance when adding basic support for unaccepted memory.
So this series consists of four patches:
- A pre-patch to switch from a kmalloc()'d page state change structure to a (smaller) stack-based page state change structure.
- A pre-patch to allow the use of the early boot GHCB in the core kernel path.
- A pre-patch to allow for use of 2M page state change requests and 2M page validation.
- SNP support for unaccepted memory.
The series is based off of and tested against Kirill Shutemov's tree:
https://github.com/intel/tdx.git guest-unaccepted-memory
---
Changes since v3:
- Reworks the PSC process to greatly improve performance:
- Optimize the PSC process to use 2M pages when applicable.
- Optimize the page validation process to use 2M pages when applicable.
- Use the early GHCB in both the decompression phase and core kernel boot phase in order to minimize the use of the MSR protocol. The MSR protocol only allows for a single 4K page to be updated at a time.
- Move the ghcb_percpu_ready flag into the sev_config structure and rename it to ghcbs_initialized.
Changes since v2:
- Improve code comments in regards to when to use the per-CPU GHCB vs the MSR protocol and why a single global value is valid for both the BSP and APs.
- Add a comment related to the number of PSC entries and how it can impact the size of the struct and, therefore, stack usage.
- Add a WARN_ON_ONCE() for invoking vmgexit_psc() when per-CPU GHCBs haven't been created or registered, yet.
- Use the compiler support for clearing the PSC struct instead of issuing memset().
Changes since v1:
- Change from using a per-CPU PSC structure to a (smaller) stack PSC structure.
Tom Lendacky (4):
x86/sev: Put PSC struct on the stack in prep for unaccepted memory
support
x86/sev: Allow for use of the early boot GHCB for PSC requests x86/sev: Use large PSC requests if applicable
x86/sev: Add SNP-specific unaccepted memory support
arch/x86/Kconfig | 1 + arch/x86/boot/compressed/mem.c | 3 + arch/x86/boot/compressed/sev.c | 54 ++++++- arch/x86/boot/compressed/sev.h | 23 +++ arch/x86/include/asm/sev-common.h | 9 +- arch/x86/include/asm/sev.h | 7 + arch/x86/kernel/sev-shared.c | 104 +++++++++++++ arch/x86/kernel/sev.c | 246 +++++++++++++----------------- arch/x86/mm/unaccepted_memory.c | 4 + 9 files changed, 305 insertions(+), 146 deletions(-) create mode 100644 arch/x86/boot/compressed/sev.h
Signature · 1 line
2.37.2
PATCH v4 1/4 x86/sev: Put PSC struct on the stack in prep for unaccepted memory support
In advance of providing support for unaccepted memory, switch from using kmalloc() for allocating the Page State Change (PSC) structure to using a local variable that lives on the stack. This is needed to avoid a possible recursive call into set_pages_state() if the kmalloc() call requires (more) memory to be accepted, which would result in a hang.
The current size of the PSC struct is 2,032 bytes. To make the struct more stack friendly, reduce the number of PSC entries from 253 down to 64, resulting in a size of 520 bytes. This is a nice compromise on struct size and total PSC requests while still allowing parallel PSC operations across vCPUs.
If the reduction in PSC entries results in any kind of performance issue (that is not seen at the moment), use of a larger static PSC struct, with fallback to the smaller stack version, can be investigated.
--- arch/x86/include/asm/sev-common.h | 9 +++++++-- arch/x86/kernel/sev.c | 10 ++-------- 2 files changed, 9 insertions(+), 10 deletions(-)
arch/x86/include/asm/sev-common.h +7 -2
@@ -106,8 +106,13 @@ enum psc_op {#define GHCB_HV_FT_SNP BIT_ULL(0)#define GHCB_HV_FT_SNP_AP_CREATION BIT_ULL(1) -/* SNP Page State Change NAE event */ -#define VMGEXIT_PSC_MAX_ENTRY 253 +/* +* SNP Page State Change NAE event +* The VMGEXIT_PSC_MAX_ENTRY determines the size of the PSC structure, +* which is a local variable (stack usage) in set_pages_state(). Do not +* increase this value without evaluating the impact to stack usage. +*/ +#define VMGEXIT_PSC_MAX_ENTRY 64struct psc_hdr {u16 cur_entry;
arch/x86/kernel/sev.c +2 -8
@@ -868,11 +868,7 @@ static void __set_pages_state(struct snp_psc_desc *data, unsigned long vaddr,static void set_pages_state (unsigned long vaddr,unsigned int npages,int op) {unsigned long vaddr_end, next_vaddr; -struct snp_psc_desc *desc; - - desc =kmalloc (sizeof (*desc),GFP_KERNEL_ACCOUNT ); -if (!desc) -panic ("SNP: failed to allocate memory for PSC descriptor\n" ); +struct snp_psc_desc desc; vaddr = vaddr &PAGE_MASK ; vaddr_end = vaddr + (npages <<PAGE_SHIFT );
@@ -882,12 +878,10 @@ static void set_pages_state(unsigned long vaddr, unsigned int npages, int op) next_vaddr = min_t(unsigned long, vaddr_end, (VMGEXIT_PSC_MAX_ENTRY *PAGE_SIZE ) + vaddr); -__set_pages_state (desc, vaddr, next_vaddr, op); +__set_pages_state (&desc, vaddr, next_vaddr, op); vaddr = next_vaddr; } - -kfree (desc); }void snp_set_memory_shared (unsigned long vaddr,unsigned int npages)
Signature · 1 line
2.37.2
Borislav Petkov
maintainerRe: [PATCH v4 1/4] x86/sev: Put PSC struct on the stack in prep for unaccepted memory support
Quoted 12 lines · Tom Lendacky
In advance of providing support for unaccepted memory, switch from using
kmalloc()for allocating the Page State Change (PSC) structure to using a local variable that lives on the stack. This is needed to avoid a possible recursive call intoset_pages_state()if thekmalloc()call requires (more) memory to be accepted, which would result in a hang.The current size of the PSC struct is 2,032 bytes. To make the struct more stack friendly, reduce the number of PSC entries from 253 down to 64, resulting in a size of 520 bytes. This is a nice compromise on struct size and total PSC requests while still allowing parallel PSC operations across vCPUs.
Quoted 3 lines
If the reduction in PSC entries results in any kind of performance issue (that is not seen at the moment), use of a larger static PSC struct, with fallback to the smaller stack version, can be investigated.
"For more background info on this decision see the subthread in the Link tag below."
Quoted 1 line
Signed-off-by Tom Lendacky <[email protected]>
Quoted 10 lines
--- arch/x86/include/asm/sev-common.h | 9 +++++++-- arch/x86/kernel/sev.c | 10 ++-------- 2 files changed, 9 insertions(+), 10 deletions(-)arch/x86/include/asm/sev-common.h +0
-0
arch/x86/include/asm/sev-common.h enum psc_op { · line 112
Quoted 9 lines
arch/x86/include/asm/sev-common.h +4
-2@@ -106,8 +106,13 @@ enum psc_op {#define GHCB_HV_FT_SNP BIT_ULL(0)#define GHCB_HV_FT_SNP_AP_CREATION BIT_ULL(1) -/* SNP Page State Change NAE event */ -#define VMGEXIT_PSC_MAX_ENTRY 253 +/* +* SNP Page State Change NAE event +* The VMGEXIT_PSC_MAX_ENTRY determines the size of the PSC structure, +* which is a local variable (stack usage) in set_pages_state(). Do not
... which is a local stack variable...
Quoted 2 lines
arch/x86/include/asm/sev-common.h +2
-0+* increase this value without evaluating the impact to stack usage. +*/
...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v4 2/4 x86/sev: Allow for use of the early boot GHCB for PSC requests
In reply to Tom LendackyUsing a GHCB for a page stage change (as opposed to the MSR protocol) allows for multiple pages to be processed in a single request. In prep for early PSC requests in support of unaccepted memory, update the invocation of vmgexit_psc() to be able to use the early boot GHCB and not just the per-CPU GHCB structure.
In order to use the proper GHCB (early boot vs per-CPU), set a flag that indicates when the per-CPU GHCBs are available and registered. For APs, the per-CPU GHCBs are created before they are started and registered upon startup, so this flag can be used globally for the BSP and APs instead of creating a per-CPU flag. This will allow for a significant reduction in the number of MSR protocol page state change requests when accepting memory.
--- arch/x86/kernel/sev.c | 61 +++++++++++++++++++++++++++---------------- 1 file changed, 38 insertions(+), 23 deletions(-)
arch/x86/kernel/sev.c +38 -23
@@ -117,7 +117,19 @@ static DEFINE_PER_CPU(struct sev_es_save_area *, sev_vmsa);struct sev_config {__u64 debug :1 , - __reserved :63 ; + +/* +* A flag used by __set_pages_state() that indicates when the +* per-CPU GHCB has been created and registered and thus can be +* used by the BSP instead of the early boot GHCB. +* +* For APs, the per-CPU GHCB is created before they are started +* and registered upon startup, so this flag can be used globally +* for the BSP and APs. +*/ + ghcbs_initialized :1 , + + __reserved :62 ; };static struct sev_config sev_cfg__read_mostly ;
@@ -660,7 +672,7 @@ static void pvalidate_pages(unsigned long vaddr, unsigned int npages, bool valid } } -static void __init early_set_pages_state (unsigned long paddr,unsigned int npages,enum psc_op op) +static void early_set_pages_state (unsigned long paddr,unsigned int npages,enum psc_op op) {unsigned long paddr_end;u64 val;
@@ -742,26 +754,13 @@ void __init snp_prep_memory(unsigned long paddr, unsigned int sz, enum psc_op opWARN (1 ,"invalid memory op %d\n" , op); } -static int vmgexit_psc (struct snp_psc_desc *desc) +static int vmgexit_psc (struct ghcb *ghcb,struct snp_psc_desc *desc) {int cur_entry, end_entry, ret =0 ;struct snp_psc_desc *data; -struct ghcb_state state;struct es_em_ctxt ctxt; -unsigned long flags; -struct ghcb *ghcb; -/* -* __sev_get_ghcb() needs to run with IRQs disabled because it is using -* a per-CPU GHCB. -*/ -local_irq_save (flags); - - ghcb =__sev_get_ghcb (&state); -if (!ghcb) { - ret =1 ; - goto out_unlock; - } +vc_ghcb_invalidate (ghcb);/* Copy the input desc into GHCB shared buffer */ data = (struct snp_psc_desc *)ghcb->shared_buffer;
@@ -818,20 +817,18 @@ static int vmgexit_psc(struct snp_psc_desc *desc) } out: -__sev_put_ghcb (&state); - -out_unlock: -local_irq_restore (flags); -return ret; }static void __set_pages_state (struct snp_psc_desc *data,unsigned long vaddr,unsigned long vaddr_end,int op) { +struct ghcb_state state;struct psc_hdr *hdr;struct psc_entry *e; +unsigned long flags;unsigned long pfn; +struct ghcb *ghcb;int i; hdr = &data->hdr;
@@ -861,8 +858,20 @@ static void __set_pages_state(struct snp_psc_desc *data, unsigned long vaddr, i++; } -if (vmgexit_psc (data)) +local_irq_save (flags); + +if (sev_cfg.ghcbs_initialized) + ghcb =__sev_get_ghcb (&state); +else + ghcb = boot_ghcb; + +if (!ghcb ||vmgexit_psc (ghcb, data))sev_es_terminate (SEV_TERM_SET_LINUX ,GHCB_TERM_PSC ); + +if (sev_cfg.ghcbs_initialized) +__sev_put_ghcb (&state); + +local_irq_restore (flags); }static void set_pages_state (unsigned long vaddr,unsigned int npages,int op)
@@ -870,6 +879,10 @@ static void set_pages_state(unsigned long vaddr, unsigned int npages, int op)unsigned long vaddr_end, next_vaddr;struct snp_psc_desc desc; +/* Use the MSR protocol when a GHCB is not available. */ +if (!boot_ghcb) +return early_set_pages_state (__pa (vaddr), npages, op); + vaddr = vaddr &PAGE_MASK ; vaddr_end = vaddr + (npages <<PAGE_SHIFT );
@@ -1248,6 +1261,8 @@ void setup_ghcb(void)if (cc_platform_has (CC_ATTR_GUEST_SEV_SNP ))snp_register_per_cpu_ghcb (); + sev_cfg.ghcbs_initialized = true; +return ; }
Signature · 1 line
2.37.2
PATCH v4 3/4 x86/sev: Use large PSC requests if applicable
In reply to Tom LendackyIn advance of providing support for unaccepted memory, request 2M Page State Change (PSC) requests when the address range allows for it. By using a 2M page size, more PSC operations can be handled in a single request to the hypervisor. The hypervisor will determine if it can accommodate the larger request by checking the mapping in the nested page table. If mapped as a large page, then the 2M page request can be performed, otherwise the 2M page request will be broken down into 512 4K page requests. This is still more efficient than having the guest perform multiple PSC requests in order to process the 512 4K pages.
In conjunction with the 2M PSC requests, attempt to perform the associated PVALIDATE instruction of the page using the 2M page size. If PVALIDATE fails with a size mismatch, then fallback to validating 512 4K pages. To do this, page validation is modified to work with the PSC structure and not just a virtual address range.
--- arch/x86/include/asm/sev.h | 4 ++ arch/x86/kernel/sev.c | 125 ++++++++++++++++++++++++------------- 2 files changed, 84 insertions(+), 45 deletions(-)
arch/x86/include/asm/sev.h +4 -0
@@ -79,11 +79,15 @@ extern void vc_no_ghcb(void);extern void vc_boot_ghcb (void );extern bool handle_vc_boot_ghcb (struct pt_regs *regs); +/* PVALIDATE return codes */ +#define PVALIDATE_FAIL_SIZEMISMATCH 6 +/* Software defined (when rFlags.CF = 1) */ #define PVALIDATE_FAIL_NOUPDATE 255/* RMP page size */ #define RMP_PG_SIZE_4K 0 +#define RMP_PG_SIZE_2M 1#define RMPADJUST_VMSA_PAGE_BIT BIT(16)
arch/x86/kernel/sev.c +80 -45
228 lines not on this page · raw message
Signature · 1 line
2.37.2