x86/boot: Cleanup
cover x86/boot: Cleanup
More cleanup, following on from the __ro_after_init work.
Andrew Cooper (3):
x86/boot: Drop pte_update_limit from physical relocation logic x86/boot: Drop move_memory() and use memcpy() directly x86/boot: Don't double-copy the stack during physical relocation
xen/arch/x86/setup.c | 105 ++++++++++++--------------------------------------- 1 file changed, 25 insertions(+), 80 deletions(-)
Signature · 1 line
2.11.0
PATCH 1/3 x86/boot: Drop pte_update_limit from physical relocation logic
This check has existed in one form or another since c/s 369bafdb1c1 "xen: Big changes to x86 start-of-day" in 2007. Even then, AFAICT, it wasn't necessary because there was a clean split between the statically created entries (L3 and higher) and the dynamically created ones (L2 and lower).
Without dissecting the myriad changes over the past 14 years, I'm pretty certain Xen only booted by accident when l2_xenmap[0] was handled specially and skipped the pte_update_limit check which would have left it corrupt.
Nevertheless, as of right now, all non-leaf PTEs (the first loop), and all 2M xenmap leaf mappings (the second loop) need relocating. They are no unused mappings in the range, no mappings which will be encountered multiple times, and it is unlikely that such mappings would be introduced.
---
CC: Jan Beulich <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Wei Liu <[email protected]>
--- xen/arch/x86/setup.c | 18 +++--------------- 1 file changed, 3 insertions(+), 15 deletions(-)
xen/arch/x86/setup.c +3 -15
@@ -1230,7 +1230,6 @@ void __init noreturn __start_xen(unsigned long mbi_p)l3_pgentry_t *pl3e;l2_pgentry_t *pl2e;int i, j, k; -unsigned long pte_update_limit;/* Select relocation address. */ xen_phys_start = end - reloc_size;
@@ -1238,14 +1237,6 @@ void __init noreturn __start_xen(unsigned long mbi_p)bootsym (trampoline_xen_phys_start) = xen_phys_start;/* -* No PTEs pointing above this address are candidates for relocation. -* Due to possibility of partial overlap of the end of source image -* and the beginning of region for destination image some PTEs may -* point to addresses in range [e, e + XEN_IMG_OFFSET). -*/ - pte_update_limit =PFN_DOWN (e); - -/* * Perform relocation to new physical address. * Before doing so we must sync static/global data with main memory * with a barrier(). After this we must *not* modify static/global
@@ -1267,8 +1258,7 @@ void __init noreturn __start_xen(unsigned long mbi_p) {/* Not present, 1GB mapping, or already relocated? */ if ( !(l3e_get_flags (*pl3e) & _PAGE_PRESENT) || - (l3e_get_flags (*pl3e) & _PAGE_PSE) || - (l3e_get_pfn (*pl3e) >= pte_update_limit) ) + (l3e_get_flags (*pl3e) & _PAGE_PSE) )continue ; *pl3e =l3e_from_intpte (l3e_get_intpte (*pl3e) + xen_phys_start);
@@ -1277,8 +1267,7 @@ void __init noreturn __start_xen(unsigned long mbi_p) {/* Not present, PSE, or already relocated? */ if ( !(l2e_get_flags (*pl2e) & _PAGE_PRESENT) || - (l2e_get_flags (*pl2e) & _PAGE_PSE) || - (l2e_get_pfn (*pl2e) >= pte_update_limit) ) + (l2e_get_flags (*pl2e) & _PAGE_PSE) )continue ; *pl2e =l2e_from_intpte (l2e_get_intpte (*pl2e) + xen_phys_start);
@@ -1291,8 +1280,7 @@ void __init noreturn __start_xen(unsigned long mbi_p)for ( i =0 ; i <L2_PAGETABLE_ENTRIES ; i++, pl2e++ ) {if ( !(l2e_get_flags (*pl2e) & _PAGE_PRESENT) || - !(l2e_get_flags (*pl2e) & _PAGE_PSE) || - (l2e_get_pfn (*pl2e) >= pte_update_limit) ) + !(l2e_get_flags (*pl2e) & _PAGE_PSE) )continue ; *pl2e =l2e_from_intpte (l2e_get_intpte (*pl2e) + xen_phys_start);
Signature · 1 line
2.11.0
Re: [PATCH 1/3] x86/boot: Drop pte_update_limit from physical relocation logic
Quoted 9 lines · Andrew Cooper
This check has existed in one form or another since c/s
369bafdb1c1"xen: Big changes to x86 start-of-day" in 2007. Even then, AFAICT, it wasn't necessary because there was a clean split between the statically created entries (L3 and higher) and the dynamically created ones (L2 and lower).Without dissecting the myriad changes over the past 14 years, I'm pretty certain Xen only booted by accident when l2_xenmap[0] was handled specially and skipped the
pte_update_limitcheck which would have left it corrupt.
Quoted 6 lines
Nevertheless, as of right now, all non-leaf PTEs (the first loop), and all 2M xenmap leaf mappings (the second loop) need relocating. They are no unused mappings in the range, no mappings which will be encountered multiple times, and it is unlikely that such mappings would be introduced.
Signed-off-by Andrew Cooper <[email protected]>
However, as to the description and ...
Quoted 11 lines
xen/arch/x86/setup.c +0
-1@@ -1230,7 +1230,6 @@ void __init noreturn __start_xen(unsigned long mbi_p)l3_pgentry_t *pl3e;l2_pgentry_t *pl2e;int i, j, k; -unsigned long pte_update_limit;/* Select relocation address. */ xen_phys_start = end - reloc_size;
xen/arch/x86/setup.c void __init noreturn __start_xen(unsigned long mbi_p) · line 1246
Quoted 9 lines
xen/arch/x86/setup.c +0
-6@@ -1238,14 +1237,6 @@ void __init noreturn __start_xen(unsigned long mbi_p)bootsym (trampoline_xen_phys_start) = xen_phys_start;/* -* No PTEs pointing above this address are candidates for relocation. -* Due to possibility of partial overlap of the end of source image -* and the beginning of region for destination image some PTEs may -* point to addresses in range [e, e + XEN_IMG_OFFSET). -*/ - pte_update_limit =PFN_DOWN (e);
... considering the comment here: Isn't it 0d31d1680868 ("x86/setup: do not relocate Xen over current Xen image placement") where need for this disappeared? Afaict the non-overlap of source and destination is the crucial factor here, yet your description doesn't mention this aspect at all. I'd therefore like to ask for an adjustment there.
Jan
Re: [PATCH 1/3] x86/boot: Drop pte_update_limit from physical relocation logic
Quoted 33 lines · Jan Beulich
Quoted 9 lines · Andrew Cooper
This check has existed in one form or another since c/s
369bafdb1c1"xen: Big changes to x86 start-of-day" in 2007. Even then, AFAICT, it wasn't necessary because there was a clean split between the statically created entries (L3 and higher) and the dynamically created ones (L2 and lower).Without dissecting the myriad changes over the past 14 years, I'm pretty certain Xen only booted by accident when l2_xenmap[0] was handled specially and skipped the
pte_update_limitcheck which would have left it corrupt.Quoted 6 lines
Nevertheless, as of right now, all non-leaf PTEs (the first loop), and all 2M xenmap leaf mappings (the second loop) need relocating. They are no unused mappings in the range, no mappings which will be encountered multiple times, and it is unlikely that such mappings would be introduced.
Signed-off-by Andrew Cooper <[email protected]>Reviewed-by Jan Beulich <[email protected]>However, as to the description and ...
Quoted 13 lines
xen/arch/x86/setup.c +0
-1@@ -1230,7 +1230,6 @@ void __init noreturn __start_xen(unsigned long mbi_p)l3_pgentry_t *pl3e;l2_pgentry_t *pl2e;int i, j, k; -unsigned long pte_update_limit;/* Select relocation address. */ xen_phys_start = end - reloc_size;@@ -1238,14 +1237,6 @@ void __init noreturn __start_xen(unsigned long mbi_p)bootsym (trampoline_xen_phys_start) = xen_phys_start;
Quoted 12 lines
xen/arch/x86/setup.c void __init noreturn __start_xen(unsigned long mbi_p) · line 1246
Quoted 7 lines
xen/arch/x86/setup.c +0
-6/* -* No PTEs pointing above this address are candidates for relocation. -* Due to possibility of partial overlap of the end of source image -* and the beginning of region for destination image some PTEs may -* point to addresses in range [e, e + XEN_IMG_OFFSET). -*/ - pte_update_limit =PFN_DOWN (e);... considering the comment here: Isn't it
0d31d1680868("x86/setup: do not relocate Xen over current Xen image placement") where need for this disappeared? Afaict the non-overlap of source and destination is the crucial factor here, yet your description doesn't mention this aspect at all. I'd therefore like to ask for an adjustment there.
I don't consider that commit relevant.
There is no circumstance ever where you can relocate Xen with actually-overlapping ranges, because one way or another, you'd end up copying non-pagetable data over the live pagetables.
That particular commit was part of trying to make Xen's entry code relocatable, so the MB2 path could load Xen at somewhere which wasn't 0, but to this day we still skip any physical relocation if Xen isn't started at 0.
To the comment specifically, it's actively wrong.
Back when XEN_IMG_OFFSET was 1M, and we had 16M worth of unconditional mappings, then we could get PTE overlap as described, in the corner case where we were moving Xen from 0 to anywhere between 4 and 16M physical (2M physical was in principle a problem, but not an eligible position to relocate to, given Xen's compile size).
And in that corner case, the logic would "corrupt" various PTEs by not relocating them. The PTE coving _start at 1M was special cased ahead of the 2nd loop (finally fixed last week) and the PTEs mapping beyond _end were unused which is why nothing actually went wrong.
~Andrew
Re: [PATCH 1/3] x86/boot: Drop pte_update_limit from physical relocation logic
Quoted 28 lines · Andrew Cooper
Quoted 14 lines · Jan Beulich
Quoted 13 lines · Andrew Cooper
xen/arch/x86/setup.c +0
-1@@ -1230,7 +1230,6 @@ void __init noreturn __start_xen(unsigned long mbi_p)l3_pgentry_t *pl3e;l2_pgentry_t *pl2e;int i, j, k; -unsigned long pte_update_limit;/* Select relocation address. */ xen_phys_start = end - reloc_size;@@ -1238,14 +1237,6 @@ void __init noreturn __start_xen(unsigned long mbi_p)bootsym (trampoline_xen_phys_start) = xen_phys_start;Quoted 12 lines
xen/arch/x86/setup.c void __init noreturn __start_xen(unsigned long mbi_p) · line 1246
Quoted 7 lines
xen/arch/x86/setup.c +0
-6/* -* No PTEs pointing above this address are candidates for relocation. -* Due to possibility of partial overlap of the end of source image -* and the beginning of region for destination image some PTEs may -* point to addresses in range [e, e + XEN_IMG_OFFSET). -*/ - pte_update_limit =PFN_DOWN (e);... considering the comment here: Isn't it
0d31d1680868("x86/setup: do not relocate Xen over current Xen image placement") where need for this disappeared? Afaict the non-overlap of source and destination is the crucial factor here, yet your description doesn't mention this aspect at all. I'd therefore like to ask for an adjustment there.
Quoted 5 lines
I don't consider that commit relevant.
There is no circumstance ever where you can relocate Xen with actually-overlapping ranges, because one way or another, you'd end up copying non-pagetable data over the live pagetables.
That was fragile, yes. I think I (vaguely!) recall a discussion I had with Keir about this, with him pointing out that the logic builds upon all necessary mappings being held in the TLB. If you strictly think that's not worthwhile to consider or mention, then so be it.
Quoted 14 lines
That particular commit was part of trying to make Xen's entry code relocatable, so the MB2 path could load Xen at somewhere which wasn't 0, but to this day we still skip any physical relocation if Xen isn't started at 0.
To the comment specifically, it's actively wrong.
Back when
XEN_IMG_OFFSETwas 1M, and we had 16M worth of unconditional mappings, then we could get PTE overlap as described, in the corner case where we were moving Xen from 0 to anywhere between 4 and 16M physical (2M physical was in principle a problem, but not an eligible position to relocate to, given Xen's compile size).
Quoted 4 lines
And in that corner case, the logic would "corrupt" various PTEs by not relocating them. The PTE coving
_startat 1M was special cased ahead of the 2nd loop (finally fixed last week) and the PTEs mapping beyond_endwere unused which is why nothing actually went wrong.
The latter fact being why I guess you've put "corrupt" in quotes.
Jan
Re: [PATCH 1/3] x86/boot: Drop pte_update_limit from physical relocation logic
Quoted 30 lines · Jan Beulich
Quoted 29 lines · Andrew Cooper
Quoted 14 lines · Jan Beulich
Quoted 13 lines · Andrew Cooper
xen/arch/x86/setup.c +0
-1@@ -1230,7 +1230,6 @@ void __init noreturn __start_xen(unsigned long mbi_p)l3_pgentry_t *pl3e;l2_pgentry_t *pl2e;int i, j, k; -unsigned long pte_update_limit;/* Select relocation address. */ xen_phys_start = end - reloc_size;@@ -1238,14 +1237,6 @@ void __init noreturn __start_xen(unsigned long mbi_p)bootsym (trampoline_xen_phys_start) = xen_phys_start;Quoted 12 lines
xen/arch/x86/setup.c void __init noreturn __start_xen(unsigned long mbi_p) · line 1246
Quoted 7 lines
xen/arch/x86/setup.c +0
-6/* -* No PTEs pointing above this address are candidates for relocation. -* Due to possibility of partial overlap of the end of source image -* and the beginning of region for destination image some PTEs may -* point to addresses in range [e, e + XEN_IMG_OFFSET). -*/ - pte_update_limit =PFN_DOWN (e);... considering the comment here: Isn't it
0d31d1680868("x86/setup: do not relocate Xen over current Xen image placement") where need for this disappeared? Afaict the non-overlap of source and destination is the crucial factor here, yet your description doesn't mention this aspect at all. I'd therefore like to ask for an adjustment there.I don't consider that commit relevant.
Quoted 7 lines
Quoted 3 lines
There is no circumstance ever where you can relocate Xen with actually-overlapping ranges, because one way or another, you'd end up copying non-pagetable data over the live pagetables.
That was fragile, yes. I think I (vaguely!) recall a discussion I had with Keir about this, with him pointing out that the logic builds upon all necessary mappings being held in the TLB. If you strictly think that's not worthwhile to consider or mention, then so be it.
Ok, I'll tweak the commit message. But having come back to this after more than a year away, I think it's worth pointing out that the details in 0d31d1680868 demonstrate that the "partial overlap" logic was indeed buggy.
In a VM, a vcpu migration at just the wrong moment will empty the TLB under your feet. So will a whole slew of micro-architectural conditions, or a poorly timed SMI.
Whatever people have tried to call it in the past, it was broken plain and simple.
~Andrew
PATCH 2/3 x86/boot: Drop move_memory() and use memcpy() directly
In reply to Andrew CooperThe way move_memory() sets up the virtual mappings means that there are always two non-overlapping regions. The virtual layout means that memmove()'s forward/backwards check doesn't do what the caller intends, as the check ought to be performed in physical space rather than virtual.
Luckily both callers already provide non-overlapping mappings, so this bug doesn't manifest, and we can move to memcpy() to avoid a backwards copy. Backwards rep movs's are typically far slower than forwards copies.
Furthermore, both callers already have suitable directmap mappings. There is no need to spend time managing early boot mappings, or chunking the copy through them.
For the main Xen relocation, we can read out of the virtual mapping that we're executing on, and write directly into the directmap. In fact, this removes one dependency on Xen being "at 0" (the XEN_IMG_OFFSET passed as src) for relocation to occur.
For the module relocation, just transcribe the move_memory() call into an equivalent memcpy().
---
CC: Jan Beulich <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Wei Liu <[email protected]>
--- xen/arch/x86/setup.c | 58 +++++----------------------------------------------- 1 file changed, 5 insertions(+), 53 deletions(-)
xen/arch/x86/setup.c +5 -53
@@ -413,53 +413,6 @@ void *__init bootstrap_map(const module_t *mod)return ret; } -static void *__init move_memory ( -uint64_t dst,uint64_t src,unsigned int size,bool keep) -{ -unsigned int blksz =BOOTSTRAP_MAP_LIMIT -BOOTSTRAP_MAP_BASE ; -unsigned int mask = (1L <<L2_PAGETABLE_SHIFT ) -1 ; - -if ( src + size >BOOTSTRAP_MAP_BASE ) - blksz >>=1 ; - -while ( size ) - { -module_t mod; -unsigned int soffs = src & mask; -unsigned int doffs = dst & mask; -unsigned int sz; -void *d, *s; - - mod.mod_start = (src - soffs) >>PAGE_SHIFT ; - mod.mod_end = soffs + size; -if ( mod.mod_end > blksz ) - mod.mod_end = blksz; - sz = mod.mod_end - soffs; - s =bootstrap_map (&mod); - - mod.mod_start = (dst - doffs) >>PAGE_SHIFT ; - mod.mod_end = doffs + size; -if ( mod.mod_end > blksz ) - mod.mod_end = blksz; -if ( sz > mod.mod_end - doffs ) - sz = mod.mod_end - doffs; - d =bootstrap_map (&mod); - -memmove (d + doffs, s + soffs, sz); - - dst += sz; - src += sz; - size -= sz; - -if ( keep ) -return size ?NULL : d + doffs; - -bootstrap_map (NULL ); - } - -return NULL ; -} -#undef BOOTSTRAP_MAP_LIMIT static uint64_t __init consider_modules(
@@ -1243,7 +1196,7 @@ void __init noreturn __start_xen(unsigned long mbi_p)* data until after we have switched to the relocated pagetables! */ barrier (); -move_memory (e,XEN_IMG_OFFSET , _end - _start,1 ); +memcpy (__va (__pa (_start)), _start, _end - _start);/* Walk idle_pg_table, relocating non-leaf entries. */ pl4e =__va (__pa (idle_pg_table));
@@ -1300,8 +1253,6 @@ void __init noreturn __start_xen(unsigned long mbi_p) "1" (__va(__pa(cpu0_stack))), "2" (STACK_SIZE / 8) : "memory" ); - bootstrap_map(NULL ); - printk("New Xen image base address: %#lx\n" , xen_phys_start); }
@@ -1325,9 +1276,10 @@ void __init noreturn __start_xen(unsigned long mbi_p) (headroom || ((end - size) >> PAGE_SHIFT) > mod[j].mod_start) ) { -move_memory (end - size + headroom, - (uint64_t )mod[j].mod_start <<PAGE_SHIFT , - mod[j].mod_end,0 ); +memcpy (__va (end - size + headroom), +__va ((uint64_t )mod[j].mod_start <<PAGE_SHIFT ), + mod[j].mod_end); + mod[j].mod_start = (end - size) >>PAGE_SHIFT ; mod[j].mod_end += headroom; mod[j].reserved =1 ;
Signature · 1 line
2.11.0
Re: [PATCH 2/3] x86/boot: Drop move_memory() and use memcpy() directly
Quoted 10 lines · Andrew Cooper
xen/arch/x86/setup.c +1
-1@@ -1243,7 +1196,7 @@ void __init noreturn __start_xen(unsigned long mbi_p)* data until after we have switched to the relocated pagetables! */ barrier (); -move_memory (e,XEN_IMG_OFFSET , _end - _start,1 ); +memcpy (__va (__pa (_start)), _start, _end - _start);/* Walk idle_pg_table, relocating non-leaf entries. */ pl4e =__va (__pa (idle_pg_table));
xen/arch/x86/setup.c void __init noreturn __start_xen(unsigned long mbi_p) · line 1257
Quoted 7 lines
xen/arch/x86/setup.c +0
-2@@ -1300,8 +1253,6 @@ void __init noreturn __start_xen(unsigned long mbi_p) "1" (__va(__pa(cpu0_stack))), "2" (STACK_SIZE / 8) : "memory" ); - bootstrap_map(NULL ); - printk("New Xen image base address: %#lx\n" , xen_phys_start); }
This bootstrap_map() must have been dead code already before, except for the "keep" argument above needlessly having got passed as 1? Afaict passing 1 was pointless without using the function's return value.
xen/arch/x86/setup.c void __init noreturn __start_xen(unsigned long mbi_p) · line 1281
Quoted 10 lines
xen/arch/x86/setup.c +3
-3@@ -1325,9 +1276,10 @@ void __init noreturn __start_xen(unsigned long mbi_p) (headroom || ((end - size) >> PAGE_SHIFT) > mod[j].mod_start) ) { -move_memory (end - size + headroom, - (uint64_t )mod[j].mod_start <<PAGE_SHIFT , - mod[j].mod_end,0 ); +memcpy (__va (end - size + headroom), +__va ((uint64_t )mod[j].mod_start <<PAGE_SHIFT ), + mod[j].mod_end);
I'm not convinced this can be memcpy() - consider_modules() specifically allows for the current module's source and destination areas to overlap. See also the comment ahead of its invocation a few lines up from here.
I'm also not convinced we have the source range (fully) direct-mapped at this point. Only full superpages have been mapped so far, and only those for the current or higher address E820 entries (plus of course the pre- built mappings of the space below 1Gb [PREBUILT_MAP_LIMIT]) located below 4Gb.
As to the 2nd argument - if this can indeed be converted in the first place, may I suggest to also switch to using pfn_to_paddr()?
Jan
Re: [PATCH 2/3] x86/boot: Drop move_memory() and use memcpy() directly
Quoted 14 lines · Jan Beulich
Quoted 13 lines · Andrew Cooper
xen/arch/x86/setup.c +1
-1@@ -1243,7 +1196,7 @@ void __init noreturn __start_xen(unsigned long mbi_p)* data until after we have switched to the relocated pagetables! */ barrier (); -move_memory (e,XEN_IMG_OFFSET , _end - _start,1 ); +memcpy (__va (__pa (_start)), _start, _end - _start);/* Walk idle_pg_table, relocating non-leaf entries. */ pl4e =__va (__pa (idle_pg_table));@@ -1300,8 +1253,6 @@ void __init noreturn __start_xen(unsigned long mbi_p) "1" (__va(__pa(cpu0_stack))), "2" (STACK_SIZE / 8) : "memory" );
Quoted 7 lines
xen/arch/x86/setup.c void __init noreturn __start_xen(unsigned long mbi_p) · line 1257
Quoted 4 lines
xen/arch/x86/setup.c +0
-2-bootstrap_map (NULL ); -printk ("New Xen image base address: %#lx\n" , xen_phys_start); }This
bootstrap_map()must have been dead code already before, except for the "keep" argument above needlessly having got passed as 1? Afaict passing 1 was pointless without using the function's return value.
bootstrap_map(NULL) is necessary to zap the constructed mappings, but it seems like the use of the return address was dropped by c/s 0b76ce20de "x86/setup: don't relocate the VGA hole" in 2013.
Quoted 14 lines
xen/arch/x86/setup.c void __init noreturn __start_xen(unsigned long mbi_p) · line 1281
Quoted 11 lines
xen/arch/x86/setup.c +3
-3@@ -1325,9 +1276,10 @@ void __init noreturn __start_xen(unsigned long mbi_p) (headroom || ((end - size) >> PAGE_SHIFT) > mod[j].mod_start) ) { -move_memory (end - size + headroom, - (uint64_t )mod[j].mod_start <<PAGE_SHIFT , - mod[j].mod_end,0 ); +memcpy (__va (end - size + headroom), +__va ((uint64_t )mod[j].mod_start <<PAGE_SHIFT ), + mod[j].mod_end);I'm not convinced this can be
memcpy()-consider_modules()specifically allows for the current module's source and destination areas to overlap. See also the comment ahead of its invocation a few lines up from here.
The comment which says:
/* Don't overlap with other modules (or Xen itself). */ end = consider_modules(s, e, size, mod, mbi->mods_count + relocated, j);
?
memmove() in move_memory() is broken, and in fact always results in a backwards copy, which means that one way or another, overlapping source and destination doesn't work.
If it was really broken before, then it can be fixed now by using memmove() here, because using 2 directmap mappings means the forward/backward check will now work as expected.
Quoted 5 lines
I'm also not convinced we have the source range (fully) direct-mapped at this point. Only full superpages have been mapped so far, and only those for the current or higher address E820 entries (plus of course the pre- built mappings of the space below 1Gb [
PREBUILT_MAP_LIMIT]) located below 4Gb.
PREBUILT_MAP_LIMIT is 2M, and that's only to cover the fact that we build l1_directmap[] with the VGA UC range at build time. I was hoping to remove it in due course.
As to the other mappings, that is awkward. Perhaps what we ought to do is split the loops. First fill in all 2M superpages into the directmap, then relocate Xen, at which point we've got plenty of frames to feed into the allocator, to let us do a second pass filling in non-2M regions.
We can depend on the modules living in RAM regions, but might want to explicitly confirm.
Quoted 2 lines
As to the 2nd argument - if this can indeed be converted in the first place, may I suggest to also switch to using
pfn_to_paddr()?
Honestly, that's taking a terrible situation and making it worse.
Calling pfn_to_paddr() on what is logically a paddr_t already ought to be a compilation error, and the logic which makes this change deliberately is some of the most nack-worthy logic I've ever come across.
It's very much not ok to have mod_start be a paddr or pfn, and for mod_end to either be an end or a sized, epending on where you are during boot.
~Andrew
Re: [PATCH 2/3] x86/boot: Drop move_memory() and use memcpy() directly
Quoted 42 lines · Andrew Cooper
Quoted 14 lines · Jan Beulich
Quoted 13 lines · Andrew Cooper
xen/arch/x86/setup.c +1
-1@@ -1243,7 +1196,7 @@ void __init noreturn __start_xen(unsigned long mbi_p)* data until after we have switched to the relocated pagetables! */ barrier (); -move_memory (e,XEN_IMG_OFFSET , _end - _start,1 ); +memcpy (__va (__pa (_start)), _start, _end - _start);/* Walk idle_pg_table, relocating non-leaf entries. */ pl4e =__va (__pa (idle_pg_table));@@ -1300,8 +1253,6 @@ void __init noreturn __start_xen(unsigned long mbi_p) "1" (__va(__pa(cpu0_stack))), "2" (STACK_SIZE / 8) : "memory" );Quoted 7 lines
xen/arch/x86/setup.c void __init noreturn __start_xen(unsigned long mbi_p) · line 1257
Quoted 4 lines
xen/arch/x86/setup.c +0
-2-bootstrap_map (NULL ); -printk ("New Xen image base address: %#lx\n" , xen_phys_start); }This
bootstrap_map()must have been dead code already before, except for the "keep" argument above needlessly having got passed as 1? Afaict passing 1 was pointless without using the function's return value.
bootstrap_map(NULL)is necessary to zap the constructed mappings, but it seems like the use of the return address was dropped by c/s0b76ce20de"x86/setup: don't relocate the VGA hole" in 2013.Quoted 14 lines
xen/arch/x86/setup.c void __init noreturn __start_xen(unsigned long mbi_p) · line 1281
Quoted 11 lines
xen/arch/x86/setup.c +3
-3@@ -1325,9 +1276,10 @@ void __init noreturn __start_xen(unsigned long mbi_p) (headroom || ((end - size) >> PAGE_SHIFT) > mod[j].mod_start) ) { -move_memory (end - size + headroom, - (uint64_t )mod[j].mod_start <<PAGE_SHIFT , - mod[j].mod_end,0 ); +memcpy (__va (end - size + headroom), +__va ((uint64_t )mod[j].mod_start <<PAGE_SHIFT ), + mod[j].mod_end);I'm not convinced this can be
memcpy()-consider_modules()specifically allows for the current module's source and destination areas to overlap. See also the comment ahead of its invocation a few lines up from here.
Quoted 7 lines
The comment which says:
/* Don't overlap with other modules (or Xen itself). */ end =
consider_modules(s, e, size, mod, mbi->mods_count + relocated, j);?
Yes, with the emphasis on "other".
Quoted 3 lines
memmove()inmove_memory()is broken, and in fact always results in a backwards copy, which means that one way or another, overlapping source and destination doesn't work.
We agree on this aspect.
Quoted 3 lines
If it was really broken before, then it can be fixed now by using
memmove()here, because using 2 directmap mappings means the forward/backward check will now work as expected.
Use of memmove() here is what I've been trying to suggest. I'm actually surprised we've had no reports of breakage, so I will admit I'm not fully certain that I'm not overlooking some aspect excluding the potential for any overlap here.
Quoted 6 lines
Quoted 5 lines
I'm also not convinced we have the source range (fully) direct-mapped at this point. Only full superpages have been mapped so far, and only those for the current or higher address E820 entries (plus of course the pre- built mappings of the space below 1Gb [
PREBUILT_MAP_LIMIT]) located below 4Gb.
Quoted 8 lines
PREBUILT_MAP_LIMITis 2M, and that's only to cover the fact that we build l1_directmap[] with the VGA UC range at build time. I was hoping to remove it in due course.As to the other mappings, that is awkward. Perhaps what we ought to do is split the loops. First fill in all 2M superpages into the directmap, then relocate Xen, at which point we've got plenty of frames to feed into the allocator, to let us do a second pass filling in non-2M regions.
Well, we already have such a split of the loops, just that the 2nd part occurs later right now. So what you suggest is either moving up that 2nd part or moving down the relocation of the modules. If you're convinced either of these can be done without breaking anything, this would indeed seem to be the way to go. I have to admit that I'm not convinced (yet).
Quoted 8 lines
We can depend on the modules living in RAM regions, but might want to explicitly confirm.
Quoted 2 lines
As to the 2nd argument - if this can indeed be converted in the first place, may I suggest to also switch to using
pfn_to_paddr()?Honestly, that's taking a terrible situation and making it worse.
Quoted 7 lines
Calling
pfn_to_paddr()on what is logically apaddr_talready ought to be a compilation error, and the logic which makes this change deliberately is some of the most nack-worthy logic I've ever come across.It's very much not ok to have
mod_startbe a paddr or pfn, and formod_endto either be an end or a sized, epending on where you are during boot.
Well, feel free to propose an improvement. Using the modules array for recording this data seemed better to me at the time than having a 2nd, almost redundant data object.
Jan
PATCH 3/3 x86/boot: Don't double-copy the stack during physical relocation
In reply to Andrew Coopercpu0_stack is contained within .data, which means the memcpy() already takes a snapshot at the start of the critical region.
Later, when we switch to the relocated Xen, we do end up losing updates to the local variables, but that's fine because the only variables we've modified go out of scope after the printk(). Use this properly to avoid copying the whole stack (32k) a second time.
---
CC: Jan Beulich <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Wei Liu <[email protected]>
Ever so slightly RFC, as it has only had light testing, but I'm confident in the reasoning.
--- xen/arch/x86/setup.c | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-)
xen/arch/x86/setup.c +17 -12
@@ -1183,6 +1183,7 @@ void __init noreturn __start_xen(unsigned long mbi_p)l3_pgentry_t *pl3e;l2_pgentry_t *pl2e;int i, j, k; +unsigned long tmp;/* Select relocation address. */ xen_phys_start = end - reloc_size;
@@ -1193,7 +1194,8 @@ void __init noreturn __start_xen(unsigned long mbi_p)* Perform relocation to new physical address. * Before doing so we must sync static/global data with main memory * with a barrier(). After this we must *not* modify static/global -* data until after we have switched to the relocated pagetables! +* data, or locals that need to survive, until after we have +* switched to the relocated pagetables! */ barrier ();memcpy (__va (__pa (_start)), _start, _end - _start);
@@ -1239,18 +1241,21 @@ void __init noreturn __start_xen(unsigned long mbi_p) *pl2e =l2e_from_intpte (l2e_get_intpte (*pl2e) + xen_phys_start); } -/* Re-sync the stack and then switch to relocated pagetables. */ +/* +* Switch to relocated pagetables. This also discards updates to +* any local variables since the memmove() call above, but that's +* fine because we don't use any of them again. +*/ asmvolatile ( -"rep movsq ; " /* re-sync the stack */ -"movq %%cr4,%%rsi ; " -"andb $0x7f,%%sil ; " -"movq %%rsi,%%cr4 ; " /* CR4.PGE == 0 */ -"movq %[pg],%%cr3 ; " /* CR3 == new pagetables */ -"orb $0x80,%%sil ; " -"movq %%rsi,%%cr4 " /* CR4.PGE == 1 */ - :"=&S" (i),"=&D" (i),"=&c" (i)/* All outputs discarded. */ - : [pg]"r" (__pa (idle_pg_table)),"0" (cpu0_stack), -"1" (__va (__pa (cpu0_stack))),"2" (STACK_SIZE /8 ) +"mov %%cr4, %[cr4]\n\t" +"andb $~%c[pge], %b[cr4]\n\t" +"mov %[cr4], %%cr4\n\t" /* CR4.PGE = 0 */ +"mov %[cr3], %%cr3\n\t" /* CR3 = new pagetables */ +"orb $%c[pge], %b[cr4]\n\t" +"mov %[cr4], %%cr4\n\t" /* CR4.PGE = 1 */ + : [cr4]"=&a" (tmp)/* Could be "r", but "a" makes better asm */ + : [cr3]"r" (__pa (idle_pg_table)), + [pge]"i" (X86_CR4_PGE ) :"memory" );printk ("New Xen image base address: %#lx\n" , xen_phys_start);
Signature · 1 line
2.11.0
Re: [PATCH 3/3] x86/boot: Don't double-copy the stack during physical relocation
Quoted 5 lines · Andrew Cooper
cpu0_stackis contained within .data, which means thememcpy()already takes a snapshot at the start of the critical region.Later, when we switch to the relocated Xen, we do end up losing updates to the local variables,
You word this as if that was the case already before your change, ...
Quoted 3 lines
but that's fine because the only variables we've modified go
out of scope after theprintk(). Use this properly to avoid copying the whole stack (32k) a second time.
... not the least because of "Use this properly ...". But aiui this is only a result of your change, in that you no longer "re-sync the stack" (as the comment says that you remove).
I can appreciate that copying 32k is too much. Yet I do think that we're putting ourselves at risk if we rely on local variables going out of scope which have been updated. Even more so with no comment next to their declarations making clear that it is imperative for them to not move to outer scopes. (Seeing e.g. another "i" and "j" in the outer scope, one might actually be inclined to do so. For "j" this would, afaict, even work, while "i" collides with the outer one no matter what.)
You mentioning the printk() - did you consider someone, perhaps just for debugging purposes, adding uses there of some of these local variables?
We could limit the copying to just the primary stack(s), for example. Otoh I'm not convinced trying to save 20k or even 32k of copied memory is worth it in this boot-time code. There are larger gains to be had elsewhere - see e.g. my "x86: memcpy() / memset() (non-)ERMS flavors plus fallout" and "IOMMU: superpage support when not sharing pagetables" which continue to be pending.
Quoted 33 lines
xen/arch/x86/setup.c +8
-4@@ -1183,6 +1183,7 @@ void __init noreturn __start_xen(unsigned long mbi_p)l3_pgentry_t *pl3e;l2_pgentry_t *pl2e;int i, j, k; +unsigned long tmp;/* Select relocation address. */ xen_phys_start = end - reloc_size;@@ -1193,7 +1194,8 @@ void __init noreturn __start_xen(unsigned long mbi_p)* Perform relocation to new physical address. * Before doing so we must sync static/global data with main memory * with a barrier(). After this we must *not* modify static/global -* data until after we have switched to the relocated pagetables! +* data, or locals that need to survive, until after we have +* switched to the relocated pagetables! */ barrier ();memcpy (__va (__pa (_start)), _start, _end - _start);@@ -1239,18 +1241,21 @@ void __init noreturn __start_xen(unsigned long mbi_p) *pl2e =l2e_from_intpte (l2e_get_intpte (*pl2e) + xen_phys_start); } -/* Re-sync the stack and then switch to relocated pagetables. */ +/* +* Switch to relocated pagetables. This also discards updates to +* any local variables since the memmove() call above, but that's +* fine because we don't use any of them again. +*/ asm volatile ( -"rep movsq ; " /* re-sync the stack */ - "movq %%cr4,%%rsi ; "
xen/arch/x86/setup.c void __init noreturn __start_xen(unsigned long mbi_p) · line 1251
Quoted 10 lines
xen/arch/x86/setup.c +2
-8-"andb $0x7f,%%sil ; " -"movq %%rsi,%%cr4 ; " /* CR4.PGE == 0 */ -"movq %[pg],%%cr3 ; " /* CR3 == new pagetables */ -"orb $0x80,%%sil ; " -"movq %%rsi,%%cr4 " /* CR4.PGE == 1 */ - : "=&S" (i), "=&D" (i), "=&c" (i) /* All outputs discarded. */ - : [pg] "r" (__pa(idle_pg_table)), "0" (cpu0_stack), -"1" (__va (__pa (cpu0_stack))),"2" (STACK_SIZE /8 ) +"mov %%cr4, %[cr4]\n\t" +"andb $~%c[pge], %b[cr4]\n\t"
Why not simply %[pge] (name subject to improvement) with the value below becoming ~X86_CR4_PGE (and the constraint becoming "K")?
xen/arch/x86/setup.c void __init noreturn __start_xen(unsigned long mbi_p) · line 1254
Quoted 3 lines
xen/arch/x86/setup.c +3
-0+"mov %[cr4], %%cr4\n\t" /* CR4.PGE = 0 */ +"mov %[cr3], %%cr3\n\t" /* CR3 = new pagetables */ +"orb $%c[pge], %b[cr4]\n\t"
Oh, I see - the constant gets reused here.
Jan
Quoted 8 lines
xen/arch/x86/setup.c +4
-0+"mov %[cr4], %%cr4\n\t" /* CR4.PGE = 1 */ + : [cr4] "=&a" (tmp) /* Could be "r", but "a" makes better asm */ + : [cr3] "r" (__pa(idle_pg_table)), + [pge] "i" (X86_CR4_PGE) : "memory" ); printk("New Xen image base address: %#lx\n" , xen_phys_start);
PATCH 1/1 x86/boot: Relocate Xen using memcpy() directly
In reply to Andrew CooperWe can relocate Xen by reading out of the virtual mapping that we're executing on, and write directly into the directmap. In fact, this removes one dependency on Xen being "at 0" (the XEN_IMG_OFFSET passed as src) for relocation to occur.
This removes all the temporary pagetable handling under the covers of move_memory(), and results in a forward copy rather than a chunked backwards copy (caused by move_memory() always constructing src and dst in a way to trigger memmove() to copy backwards).
With the penultimate caller of move_memory() dropped, clean up the API. Drop the keep boolean, folding in 0 from the final caller, and drop the return address which has been unused since c/s 0b76ce20de85 ("x86/setup: don't relocate the VGA hole.") in 2007.
---
CC: Jan Beulich <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Wei Liu <[email protected]>
v2-ish:
* Split out previous series. This was the "easy to shuffle" work that still gets a win. Everything else I'm going to rework differently, so will have to be deferred for now.
--- xen/arch/x86/setup.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)
xen/arch/x86/setup.c +4 -11
@@ -425,8 +425,8 @@ void *__init bootstrap_map(const module_t *mod)return ret; } -static void *__init move_memory ( -uint64_t dst,uint64_t src,unsigned int size,bool keep) +static void __init move_memory ( +uint64_t dst,uint64_t src,unsigned int size) {unsigned int blksz =BOOTSTRAP_MAP_LIMIT -BOOTSTRAP_MAP_BASE ;unsigned int mask = (1L <<L2_PAGETABLE_SHIFT ) -1 ;
@@ -463,13 +463,8 @@ static void *__init move_memory( src += sz; size -= sz; -if ( keep ) -return size ?NULL : d + doffs; -bootstrap_map (NULL ); } - -return NULL ; }#undef BOOTSTRAP_MAP_LIMIT
@@ -1277,7 +1272,7 @@ void __init noreturn __start_xen(unsigned long mbi_p)* data until after we have switched to the relocated pagetables! */ barrier (); -move_memory (e,XEN_IMG_OFFSET , _end - _start,1 ); +memcpy (__va (__pa (_start)), _start, _end - _start);/* Walk idle_pg_table, relocating non-leaf entries. */ pl4e =__va (__pa (idle_pg_table));
@@ -1334,8 +1329,6 @@ void __init noreturn __start_xen(unsigned long mbi_p) "1" (__va(__pa(cpu0_stack))), "2" (STACK_SIZE / 8) : "memory" ); - bootstrap_map(NULL ); - printk("New Xen image base address: %#lx\n" , xen_phys_start); }
@@ -1361,7 +1354,7 @@ void __init noreturn __start_xen(unsigned long mbi_p) {move_memory (end - size + headroom, (uint64_t )mod[j].mod_start <<PAGE_SHIFT , - mod[j].mod_end,0 ); + mod[j].mod_end); mod[j].mod_start = (end - size) >>PAGE_SHIFT ; mod[j].mod_end += headroom; mod[j].reserved =1 ;
Signature · 1 line
2.11.0
Re: [PATCH v2-ish] x86/boot: Relocate Xen using memcpy() directly
Quoted 10 lines · Andrew Cooper
We can relocate Xen by reading out of the virtual mapping that we're executing on, and write directly into the directmap. In fact, this removes one dependency on Xen being "at 0" (the
XEN_IMG_OFFSETpassed as src) for relocation to occur.This removes all the temporary pagetable handling under the covers of
move_memory(), and results in a forward copy rather than a chunked backwards copy (caused bymove_memory()always constructing src and dst in a way to triggermemmove()to copy backwards).
Quoted 6 lines
With the penultimate caller of
move_memory()dropped, clean up the API. Drop the keep boolean, folding in 0 from the final caller, and drop the return address which has been unused since c/s0b76ce20de85("x86/setup: don't relocate the VGA hole.") in 2007.Signed-off-by Andrew Cooper <[email protected]>
PATCH 1/1 x86/boot: Factor move_xen() out of __start_xen()
In reply to Andrew CooperPartly for clarity because there is a lot of subtle magic at work here. Expand the commentary of what's going on.
Also, because there is no need to double copy the stack (32kB) to retrieve local values spilled to the stack under the old alias, when all of the aforementioned local variables are going out of scope anyway.
There is also a logic change when walking l2_xenmap[]. The old logic would skip recursing into 4k mappings; this would corrupt Xen if it were used. There are no 4k mappings in l2_xenmap[] this early on boot, so assert the property instead of leaving a latent breakage.
---
CC: Jan Beulich <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Wei Liu <[email protected]>
We probably want to drop PGE from XEN_MINIMAL_CR4. It will simplify several boot paths which don't need to care about global pages, and PGE is conditional anyway now with the PCID support.
v2-ish:
* Split out previous series. Was previously "x86/boot: Don't double-copy the stack during physical relocation" reworked to remove the risk of losing local stack updated we might care about.
Note that putting a printk() in the critical region wouldn't have worked even with the old logic, and is not a reduction in functionality caused by this patch avoiding the second stack copy.
--- xen/arch/x86/setup.c | 178 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 108 insertions(+), 70 deletions(-)
xen/arch/x86/setup.c +108 -70
@@ -467,6 +467,113 @@ static void __init move_memory( } } +static void __init noinline move_xen(void) +{ +l4_pgentry_t *pl4e; +l3_pgentry_t *pl3e; +l2_pgentry_t *pl2e; +unsigned long tmp; +int i, j, k; + +/* +* The caller has selected xen_phys_start, ensuring that the old and new +* locations do not overlap. Make it so. +* +* Prevent the compiler from reordering anything across this point. Such +* things will end badly. +*/ +barrier (); + +/* +* Copy out of the current alias, into the directmap at the new location. +* This includes a snapshot of the current stack. +*/ +memcpy (__va (__pa (_start)), _start, _end - _start); + +/* +* We are now in a critical region. Any write (modifying a global, +* spilling a local to the stack, etc) via the current alias will get lost +* when we switch to the new pagetables. This even means no printk()'s +* debugging purposes. +* +* Walk the soon-to-be-used pagetables in the new location, relocating all +* intermediate (non-leaf) entries to point to their new-location +* equivalents. All writes are via the directmap alias. +*/ + pl4e =__va (__pa (idle_pg_table)); +for ( i =0 ; i <L4_PAGETABLE_ENTRIES ; i++, pl4e++ ) + { +if ( !(l4e_get_flags (*pl4e) & _PAGE_PRESENT) ) +continue ; + + *pl4e =l4e_from_intpte (l4e_get_intpte (*pl4e) + xen_phys_start); + pl3e =__va (l4e_get_paddr (*pl4e)); +for ( j =0 ; j <L3_PAGETABLE_ENTRIES ; j++, pl3e++ ) + { +if ( !(l3e_get_flags (*pl3e) & _PAGE_PRESENT) || + (l3e_get_flags (*pl3e) & _PAGE_PSE) ) +continue ; + + *pl3e =l3e_from_intpte (l3e_get_intpte (*pl3e) + xen_phys_start); + pl2e =__va (l3e_get_paddr (*pl3e)); +for ( k =0 ; k <L2_PAGETABLE_ENTRIES ; k++, pl2e++ ) + { +if ( !(l2e_get_flags (*pl2e) & _PAGE_PRESENT) || + (l2e_get_flags (*pl2e) & _PAGE_PSE) ) +continue ; + + *pl2e =l2e_from_intpte (l2e_get_intpte (*pl2e) + xen_phys_start); + } + } + } + +/* +* Walk the soon-to-be-used l2_xenmap[], relocating all the leaf mappings +* so text/data/bss etc refer to the new location in memory. +*/ + pl2e =__va (__pa (l2_xenmap)); +for ( i =0 ; i <L2_PAGETABLE_ENTRIES ; i++, pl2e++ ) + { +if ( !(l2e_get_flags (*pl2e) & _PAGE_PRESENT) ) +continue ; + +/* +* We don't have 4k mappings in l2_xenmap[] at this point in boot, nor +* is this anticipated to change. Simply assert the fact, rather than +* introducing dead logic to decend into l1 tables. +*/ +ASSERT (l2e_get_flags (*pl2e) & _PAGE_PSE); + + *pl2e =l2e_from_intpte (l2e_get_intpte (*pl2e) + xen_phys_start); + } + +/* +* Switch to relocated pagetables, shooting down global mappings as we go. +*/ + asmvolatile ( +"mov %%cr4, %[cr4]\n\t" +"andb $~%c[pge], %b[cr4]\n\t" +"mov %[cr4], %%cr4\n\t" /* CR4.PGE = 0 */ +"mov %[cr3], %%cr3\n\t" /* CR3 = new pagetables */ +"orb $%c[pge], %b[cr4]\n\t" +"mov %[cr4], %%cr4\n\t" /* CR4.PGE = 1 */ + : [cr4]"=&a" (tmp)/* Could be "r", but "a" makes better asm */ + : [cr3]"r" (__pa (idle_pg_table)), + [pge]"i" (X86_CR4_PGE ) + :"memory" ); + +/* +* End of the critical region. Updates to locals and globals now work as +* expected. +* +* Updates to local variables which have been spilled to the stack since +* the memcpy() have been lost. But we don't care, because they're all +* going out of scope imminently. +*/ + +printk ("New Xen image base address: %#lx\n" , xen_phys_start); +} +#undef BOOTSTRAP_MAP_LIMIT static uint64_t __init consider_modules(
@@ -1255,81 +1362,12 @@ void __init noreturn __start_xen(unsigned long mbi_p)*/ if ( (end > s) && (end - reloc_size +XEN_IMG_OFFSET >=__pa (_end)) ) { -l4_pgentry_t *pl4e; -l3_pgentry_t *pl3e; -l2_pgentry_t *pl2e; -int i, j, k; -/* Select relocation address. */ xen_phys_start = end - reloc_size; e = xen_phys_start +XEN_IMG_OFFSET ;bootsym (trampoline_xen_phys_start) = xen_phys_start; -/* -* Perform relocation to new physical address. -* Before doing so we must sync static/global data with main memory -* with a barrier(). After this we must *not* modify static/global -* data until after we have switched to the relocated pagetables! -*/ -barrier (); -memcpy (__va (__pa (_start)), _start, _end - _start); - -/* Walk idle_pg_table, relocating non-leaf entries. */ - pl4e =__va (__pa (idle_pg_table)); -for ( i =0 ; i <L4_PAGETABLE_ENTRIES ; i++, pl4e++ ) - { -if ( !(l4e_get_flags (*pl4e) & _PAGE_PRESENT) ) -continue ; - *pl4e =l4e_from_intpte (l4e_get_intpte (*pl4e) + - xen_phys_start); - pl3e =__va (l4e_get_paddr (*pl4e)); -for ( j =0 ; j <L3_PAGETABLE_ENTRIES ; j++, pl3e++ ) - { -/* Not present, 1GB mapping, or already relocated? */ -if ( !(l3e_get_flags (*pl3e) & _PAGE_PRESENT) || - (l3e_get_flags (*pl3e) & _PAGE_PSE) ) -continue ; - *pl3e =l3e_from_intpte (l3e_get_intpte (*pl3e) + - xen_phys_start); - pl2e =__va (l3e_get_paddr (*pl3e)); -for ( k =0 ; k <L2_PAGETABLE_ENTRIES ; k++, pl2e++ ) - { -/* Not present, PSE, or already relocated? */ -if ( !(l2e_get_flags (*pl2e) & _PAGE_PRESENT) || - (l2e_get_flags (*pl2e) & _PAGE_PSE) ) -continue ; - *pl2e =l2e_from_intpte (l2e_get_intpte (*pl2e) + - xen_phys_start); - } - } - } - -/* Walk l2_xenmap[], relocating 2M superpage leaves. */ - pl2e =__va (__pa (l2_xenmap)); -for ( i =0 ; i <L2_PAGETABLE_ENTRIES ; i++, pl2e++ ) - { -if ( !(l2e_get_flags (*pl2e) & _PAGE_PRESENT) || - !(l2e_get_flags (*pl2e) & _PAGE_PSE) ) -continue ; - - *pl2e =l2e_from_intpte (l2e_get_intpte (*pl2e) + xen_phys_start); - } - -/* Re-sync the stack and then switch to relocated pagetables. */ - asmvolatile ( -"rep movsq ; " /* re-sync the stack */ -"movq %%cr4,%%rsi ; " -"andb $0x7f,%%sil ; " -"movq %%rsi,%%cr4 ; " /* CR4.PGE == 0 */ -"movq %[pg],%%cr3 ; " /* CR3 == new pagetables */ -"orb $0x80,%%sil ; " -"movq %%rsi,%%cr4 " /* CR4.PGE == 1 */ - :"=&S" (i),"=&D" (i),"=&c" (i)/* All outputs discarded. */ - : [pg]"r" (__pa (idle_pg_table)),"0" (cpu0_stack), -"1" (__va (__pa (cpu0_stack))),"2" (STACK_SIZE /8 ) - :"memory" ); - -printk ("New Xen image base address: %#lx\n" , xen_phys_start); +move_xen (); }/* Is the region suitable for relocating the multiboot modules? */
Signature · 1 line
2.11.0
Re: [PATCH v2-ish] x86/boot: Factor move_xen() out of __start_xen()
Quoted 13 lines · Andrew Cooper
Partly for clarity because there is a lot of subtle magic at work here. Expand the commentary of what's going on.
Also, because there is no need to double copy the stack (32kB) to retrieve local values spilled to the stack under the old alias, when all of the aforementioned local variables are going out of scope anyway.
There is also a logic change when walking l2_xenmap[]. The old logic would skip recursing into 4k mappings; this would corrupt Xen if it were used. There are no 4k mappings in l2_xenmap[] this early on boot, so assert the property instead of leaving a latent breakage.
Signed-off-by Andrew Cooper <[email protected]>
Quoted 8 lines
---
CC: Jan Beulich <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Wei Liu <[email protected]>We probably want to drop PGE from
XEN_MINIMAL_CR4. It will simplify several boot paths which don't need to care about global pages, and PGE is conditional anyway now with the PCID support.
Perhaps, especially if - as you say - this allows for simplification.
xen/arch/x86/setup.c static void __init move_memory( · line 476
Quoted 13 lines
xen/arch/x86/setup.c +7
-0@@ -467,6 +467,113 @@ static void __init move_memory( } } +static void __init noinline move_xen(void) +{ +l4_pgentry_t *pl4e; +l3_pgentry_t *pl3e; +l2_pgentry_t *pl2e; +unsigned long tmp; +int i, j, k;
Can these become "unsigned int" please at this occasion? (Perhaps as another reason to make the change, mention that the old instances of i and j did shadow outer scope variables in the same function?)
xen/arch/x86/setup.c static void __init move_memory( · line 480
Quoted 3 lines
xen/arch/x86/setup.c +3
-0+/* +* The caller has selected xen_phys_start, ensuring that the old and new +* locations do not overlap. Make it so.
As a non-native speaker I'm struggling with what "Make it so" is supposed to tell me here.
Quoted 6 lines
xen/arch/x86/setup.c +6
-0+* Prevent the compiler from reordering anything across this point. Such +* things will end badly. +*/ +barrier (); + +/*
xen/arch/x86/setup.c static void __init move_memory( · line 497
Quoted 10 lines
xen/arch/x86/setup.c +10
-0+* Copy out of the current alias, into the directmap at the new location. +* This includes a snapshot of the current stack. +*/ +memcpy (__va (__pa (_start)), _start, _end - _start); + +/* +* We are now in a critical region. Any write (modifying a global, +* spilling a local to the stack, etc) via the current alias will get lost +* when we switch to the new pagetables. This even means no printk()'s +* debugging purposes.
Nit: Missing "for"?
Quoted 35 lines
xen/arch/x86/setup.c +35
-0+* Walk the soon-to-be-used pagetables in the new location, relocating all +* intermediate (non-leaf) entries to point to their new-location +* equivalents. All writes are via the directmap alias. +*/ + pl4e =__va (__pa (idle_pg_table)); +for ( i =0 ; i <L4_PAGETABLE_ENTRIES ; i++, pl4e++ ) + { +if ( !(l4e_get_flags (*pl4e) & _PAGE_PRESENT) ) +continue ; + + *pl4e =l4e_from_intpte (l4e_get_intpte (*pl4e) + xen_phys_start); + pl3e =__va (l4e_get_paddr (*pl4e)); +for ( j =0 ; j <L3_PAGETABLE_ENTRIES ; j++, pl3e++ ) + { +if ( !(l3e_get_flags (*pl3e) & _PAGE_PRESENT) || + (l3e_get_flags (*pl3e) & _PAGE_PSE) ) +continue ; + + *pl3e =l3e_from_intpte (l3e_get_intpte (*pl3e) + xen_phys_start); + pl2e =__va (l3e_get_paddr (*pl3e)); +for ( k =0 ; k <L2_PAGETABLE_ENTRIES ; k++, pl2e++ ) + { +if ( !(l2e_get_flags (*pl2e) & _PAGE_PRESENT) || + (l2e_get_flags (*pl2e) & _PAGE_PSE) ) +continue ; + + *pl2e =l2e_from_intpte (l2e_get_intpte (*pl2e) + xen_phys_start); + } + } + } + +/* +* Walk the soon-to-be-used l2_xenmap[], relocating all the leaf mappings +* so text/data/bss etc refer to the new location in memory. +*/
xen/arch/x86/setup.c static void __init move_memory( · line 543
Quoted 10 lines
xen/arch/x86/setup.c +10
-0+ pl2e =__va (__pa (l2_xenmap)); +for ( i =0 ; i <L2_PAGETABLE_ENTRIES ; i++, pl2e++ ) + { +if ( !(l2e_get_flags (*pl2e) & _PAGE_PRESENT) ) +continue ; + +/* +* We don't have 4k mappings in l2_xenmap[] at this point in boot, nor +* is this anticipated to change. Simply assert the fact, rather than +* introducing dead logic to decend into l1 tables.
Nit: "descend"?
xen/arch/x86/setup.c static void __init move_memory( · line 542
Quoted 2 lines
xen/arch/x86/setup.c +2
-0+*/ +ASSERT (l2e_get_flags (*pl2e) & _PAGE_PSE);
The warning about the use of printk() around here could make one think that using ASSERT() (or BUG()) is similarly bad. However, aiui using printk() isn't by itself a problem, just that the log message would be lost as far as the circular buffer goes. The message would be observable on the serial con sole though, at least with "sync_console". It is on this basis that using ASSERT() here makes sense. Perhaps the printk() related comment could be slightly adjusted to express this?
xen/arch/x86/setup.c static void __init move_memory( · line 558
Quoted 12 lines
xen/arch/x86/setup.c +12
-0+ *pl2e =l2e_from_intpte (l2e_get_intpte (*pl2e) + xen_phys_start); + } + +/* +* Switch to relocated pagetables, shooting down global mappings as we go. +*/ + asmvolatile ( +"mov %%cr4, %[cr4]\n\t" +"andb $~%c[pge], %b[cr4]\n\t" +"mov %[cr4], %%cr4\n\t" /* CR4.PGE = 0 */ +"mov %[cr3], %%cr3\n\t" /* CR3 = new pagetables */ + "orb $%c[pge], %b[cr4]\n\t"
I understand you need %c to apply the ~ in the operand of ANDB above. But could I talk you into keeping things as simple as possible here by using plain %[pge]?
xen/arch/x86/setup.c static void __init move_memory( · line 563
Quoted 5 lines
xen/arch/x86/setup.c +5
-0+"mov %[cr4], %%cr4\n\t" /* CR4.PGE = 1 */ + : [cr4] "=&a" (tmp) /* Could be "r", but "a" makes better asm */ + : [cr3] "r" (__pa(idle_pg_table)), + [pge] "i" (X86_CR4_PGE) + : "memory" );
The removed stack copying worries me, to be honest. Yes, for local variables of ours it doesn't matter because they are about to go out of scope. But what if the compiler instantiates any for its own use, e.g. ...
xen/arch/x86/setup.c static void __init move_memory( · line 574
Quoted 10 lines
xen/arch/x86/setup.c +10
-0+/* +* End of the critical region. Updates to locals and globals now work as +* expected. +* +* Updates to local variables which have been spilled to the stack since +* the memcpy() have been lost. But we don't care, because they're all +* going out of scope imminently. +*/ + +printk ("New Xen image base address: %#lx\n" , xen_phys_start);
... the result of the address calculation for the string literal here? Such auxiliary calculations can happen at any point in the function, and won't necessarily (hence the example chosen) become impossible for the compiler to do with the memory clobber in the asm(). And while the string literal's address is likely cheap enough to calculate right in the function invocation sequence (and an option would also be to retain the printk() in the caller), other instrumentation options could be undermined by this as well.
Jan
Re: [PATCH v2-ish] x86/boot: Factor move_xen() out of __start_xen()
Quoted 30 lines · Jan Beulich
Quoted 13 lines · Andrew Cooper
Partly for clarity because there is a lot of subtle magic at work here. Expand the commentary of what's going on.
Also, because there is no need to double copy the stack (32kB) to retrieve local values spilled to the stack under the old alias, when all of the aforementioned local variables are going out of scope anyway.
There is also a logic change when walking l2_xenmap[]. The old logic would skip recursing into 4k mappings; this would corrupt Xen if it were used. There are no 4k mappings in l2_xenmap[] this early on boot, so assert the property instead of leaving a latent breakage.
Signed-off-by Andrew Cooper <[email protected]>Quoted 8 lines
---
CC: Jan Beulich <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Wei Liu <[email protected]>We probably want to drop PGE from
XEN_MINIMAL_CR4. It will simplify several boot paths which don't need to care about global pages, and PGE is conditional anyway now with the PCID support.Perhaps, especially if - as you say - this allows for simplification.
Quoted 6 lines
xen/arch/x86/setup.c +0
-0@@ -467,6 +467,113 @@ static void __init move_memory( } }
Quoted 8 lines
xen/arch/x86/setup.c static void __init move_memory( · line 476
Quoted 7 lines
xen/arch/x86/setup.c +7
-0+static void __init noinline move_xen(void) +{ +l4_pgentry_t *pl4e; +l3_pgentry_t *pl3e; +l2_pgentry_t *pl2e; +unsigned long tmp; +int i, j, k;Can these become "unsigned int" please at this occasion?
Fixed.
Quoted 3 lines
(Perhaps as
another reason to make the change, mention that the old instances of i and j did shadow outer scope variables in the same function?)
Oh, so it does. I'm slightly surprised that we haven't seen a compiler objection from that.
Quoted 6 lines
xen/arch/x86/setup.c static void __init move_memory( · line 480
Quoted 4 lines
xen/arch/x86/setup.c +3
-0+/* +* The caller has selected xen_phys_start, ensuring that the old and new +* locations do not overlap. Make it so. As a non-native speaker I'm struggling with what "Make it so" is supposed to tell me here.
"Actually do it"
I'll drop it. It's not overly important to the understanding here.
Quoted 10 lines
Quoted 10 lines
xen/arch/x86/setup.c +9
-0+* Prevent the compiler from reordering anything across this point. Such +* things will end badly. +*/ +barrier (); + +/* +* Copy out of the current alias, into the directmap at the new location. +* This includes a snapshot of the current stack. +*/
Quoted 8 lines
xen/arch/x86/setup.c static void __init move_memory( · line 497
Quoted 7 lines
xen/arch/x86/setup.c +7
-0+memcpy (__va (__pa (_start)), _start, _end - _start); + +/* +* We are now in a critical region. Any write (modifying a global, +* spilling a local to the stack, etc) via the current alias will get lost +* when we switch to the new pagetables. This even means no printk()'s +* debugging purposes. Nit: Missing "for"?
Fixed.
Quoted 39 lines
Quoted 39 lines
xen/arch/x86/setup.c +38
-0+* Walk the soon-to-be-used pagetables in the new location, relocating all +* intermediate (non-leaf) entries to point to their new-location +* equivalents. All writes are via the directmap alias. +*/ + pl4e =__va (__pa (idle_pg_table)); +for ( i =0 ; i <L4_PAGETABLE_ENTRIES ; i++, pl4e++ ) + { +if ( !(l4e_get_flags (*pl4e) & _PAGE_PRESENT) ) +continue ; + + *pl4e =l4e_from_intpte (l4e_get_intpte (*pl4e) + xen_phys_start); + pl3e =__va (l4e_get_paddr (*pl4e)); +for ( j =0 ; j <L3_PAGETABLE_ENTRIES ; j++, pl3e++ ) + { +if ( !(l3e_get_flags (*pl3e) & _PAGE_PRESENT) || + (l3e_get_flags (*pl3e) & _PAGE_PSE) ) +continue ; + + *pl3e =l3e_from_intpte (l3e_get_intpte (*pl3e) + xen_phys_start); + pl2e =__va (l3e_get_paddr (*pl3e)); +for ( k =0 ; k <L2_PAGETABLE_ENTRIES ; k++, pl2e++ ) + { +if ( !(l2e_get_flags (*pl2e) & _PAGE_PRESENT) || + (l2e_get_flags (*pl2e) & _PAGE_PSE) ) +continue ; + + *pl2e =l2e_from_intpte (l2e_get_intpte (*pl2e) + xen_phys_start); + } + } + } + +/* +* Walk the soon-to-be-used l2_xenmap[], relocating all the leaf mappings +* so text/data/bss etc refer to the new location in memory. +*/ + pl2e =__va (__pa (l2_xenmap)); +for ( i =0 ; i <L2_PAGETABLE_ENTRIES ; i++, pl2e++ ) + {
Quoted 8 lines
xen/arch/x86/setup.c static void __init move_memory( · line 543
Quoted 7 lines
xen/arch/x86/setup.c +7
-0+if ( !(l2e_get_flags (*pl2e) & _PAGE_PRESENT) ) +continue ; + +/* +* We don't have 4k mappings in l2_xenmap[] at this point in boot, nor +* is this anticipated to change. Simply assert the fact, rather than +* introducing dead logic to decend into l1 tables. Nit: "descend"?
Fixed.
Quoted 10 lines
xen/arch/x86/setup.c static void __init move_memory( · line 542
Quoted 3 lines
xen/arch/x86/setup.c +2
-0+*/ +ASSERT (l2e_get_flags (*pl2e) & _PAGE_PSE);The warning about the use of
printk()around here could make one think that usingASSERT()(orBUG()) is similarly bad. However, aiui usingprintk()isn't by itself a problem, just that the log message would be lost as far as the circular buffer goes. The message would be observable on the serial con sole though, at least with "sync_console". It is on this basis that usingASSERT()here makes sense. Perhaps theprintk()related comment could be slightly adjusted to express this?
I did try to describe it, and it gets complicated, so I opted for the simple approach.
Before the pagetable switch, it will operate "properly" and emit text onto the screen, serial, etc.
After the pagetable switch, all Xen data will be lost, which includes the main console buffer, but also things like the local state for Xen's UART driver. I gave up trying to reason what would happen at that point.
The ASSERT() is very much a best-effort attempt. Anyone liable to trip the assert is already working on the boot pagetable and ought to be aware.
Quoted 13 lines
xen/arch/x86/setup.c static void __init move_memory( · line 558
Quoted 13 lines
xen/arch/x86/setup.c +12
-0+ *pl2e =l2e_from_intpte (l2e_get_intpte (*pl2e) + xen_phys_start); + } + +/* +* Switch to relocated pagetables, shooting down global mappings as we go. +*/ + asmvolatile ( +"mov %%cr4, %[cr4]\n\t" +"andb $~%c[pge], %b[cr4]\n\t" +"mov %[cr4], %%cr4\n\t" /* CR4.PGE = 0 */ +"mov %[cr3], %%cr3\n\t" /* CR3 = new pagetables */ + "orb $%c[pge], %b[cr4]\n\t"
Quoted 3 lines
I understand you need %c to apply the ~ in the operand of ANDB above. But could I talk you into keeping things as simple as possible here by using plain %[pge]?
The $ is still useful in the second case, for consistency if nothing else.
Both of these instances disappear if PGE gets cleaned up on boot.
Quoted 21 lines
xen/arch/x86/setup.c static void __init move_memory( · line 563
Quoted 6 lines
xen/arch/x86/setup.c +5
-0+"mov %[cr4], %%cr4\n\t" /* CR4.PGE = 1 */ + : [cr4] "=&a" (tmp) /* Could be "r", but "a" makes better asm */ + : [cr3] "r" (__pa(idle_pg_table)), + [pge] "i" (X86_CR4_PGE) + : "memory" );The removed stack copying worries me, to be honest. Yes, for local variables of ours it doesn't matter because they are about to go out of scope. But what if the compiler instantiates any for its own use, e.g. ...
xen/arch/x86/setup.c static void __init move_memory( · line 574
Quoted 10 lines
xen/arch/x86/setup.c +10
-0+/* +* End of the critical region. Updates to locals and globals now work as +* expected. +* +* Updates to local variables which have been spilled to the stack since +* the memcpy() have been lost. But we don't care, because they're all +* going out of scope imminently. +*/ + +printk ("New Xen image base address: %#lx\n" , xen_phys_start);
Quoted 8 lines
... the result of the address calculation for the string literal here? Such auxiliary calculations can happen at any point in the function, and won't necessarily (hence the example chosen) become impossible for the compiler to do with the memory clobber in the
asm(). And while the string literal's address is likely cheap enough to calculate right in the function invocation sequence (and an option would also be to retain theprintk()in the caller), other instrumentation options could be undermined by this as well.
Right now, the compiler is free to calculate the address of the string literal in a register, and move it the other side of the TLB flush. This will work just fine.
However, the compiler cannot now, or ever in the future, spill such a calculation to the stack.
Whether it's spelt "memory", or something else in the future, OSes really do genuinely need a way of telling the compiler "you literally cannot move anything the other side of this asm()", because otherwise malfunctions will occur.
It's not the locals which worry me - we really do lose things like coverage data in here. Short of writing it fully in asm (which would be irritating to maintain), I don't see any other option.
~Andrew
Re: [PATCH v2-ish] x86/boot: Factor move_xen() out of __start_xen()
Quoted 31 lines · Andrew Cooper
Quoted 21 lines · Jan Beulich
xen/arch/x86/setup.c static void __init move_memory( · line 563
Quoted 5 lines · Andrew Cooper
xen/arch/x86/setup.c +5
-0+"mov %[cr4], %%cr4\n\t" /* CR4.PGE = 1 */ + : [cr4] "=&a" (tmp) /* Could be "r", but "a" makes better asm */ + : [cr3] "r" (__pa(idle_pg_table)), + [pge] "i" (X86_CR4_PGE) + : "memory" );The removed stack copying worries me, to be honest. Yes, for local variables of ours it doesn't matter because they are about to go out of scope. But what if the compiler instantiates any for its own use, e.g. ...
xen/arch/x86/setup.c static void __init move_memory( · line 574
Quoted 10 lines
xen/arch/x86/setup.c +10
-0+/* +* End of the critical region. Updates to locals and globals now work as +* expected. +* +* Updates to local variables which have been spilled to the stack since +* the memcpy() have been lost. But we don't care, because they're all +* going out of scope imminently. +*/ + +printk ("New Xen image base address: %#lx\n" , xen_phys_start);Quoted 8 lines
... the result of the address calculation for the string literal here? Such auxiliary calculations can happen at any point in the function, and won't necessarily (hence the example chosen) become impossible for the compiler to do with the memory clobber in the
asm(). And while the string literal's address is likely cheap enough to calculate right in the function invocation sequence (and an option would also be to retain theprintk()in the caller), other instrumentation options could be undermined by this as well.
Quoted 6 lines
Right now, the compiler is free to calculate the address of the string literal in a register, and move it the other side of the TLB flush. This will work just fine.
However, the compiler cannot now, or ever in the future, spill such a calculation to the stack.
I'm afraid the compiler's view at things is different: Whatever it puts on the stack is viewed as virtual registers, unaffected by a memory clobber (of course there can be effects resulting from uses of named variables). Look at -O3 output of gcc12 (which is what I happened to play with; I don't think it's overly version dependent) for this (clearly contrived) piece of code:
int __attribute__((const)) calc(int);
int test(int i) {
int j = calc(i);
asm("nopl %0" : "+m" (j));
asm("nopq %%rsp" ::: "memory", "ax", "cx", "dx", "bx", "bp", "si", "di",
"r8", "r9", "r10", "r11", "r12", "r13", "r14", "r15");
j = calc(i);
asm("nopl %0" :: "m" (j));
return j;
}It instantiates a local on the stack for the result of calc(); it does not re-invoke calc() a 2nd time. Which means the memory clobber in the middle asm() does not affect that (and by implication: any) stack slot.
Using godbolt I can also see that clang15 agrees with gcc12 in this regard. I didn't bother trying other versions.
Quoted 4 lines
Whether it's spelt "memory", or something else in the future, OSes really do genuinely need a way of telling the compiler "you literally cannot move anything the other side of this
asm()", because otherwise malfunctions will occur.
Something like this may indeed be desirable in situations like this one, but I don't think such a construct exists.
Jan
Re: [PATCH v2-ish] x86/boot: Factor move_xen() out of __start_xen()
Quoted 58 lines · Jan Beulich
Quoted 30 lines · Andrew Cooper
Quoted 21 lines · Jan Beulich
xen/arch/x86/setup.c static void __init move_memory( · line 563
Quoted 5 lines · Andrew Cooper
xen/arch/x86/setup.c +5
-0+"mov %[cr4], %%cr4\n\t" /* CR4.PGE = 1 */ + : [cr4] "=&a" (tmp) /* Could be "r", but "a" makes better asm */ + : [cr3] "r" (__pa(idle_pg_table)), + [pge] "i" (X86_CR4_PGE) + : "memory" );The removed stack copying worries me, to be honest. Yes, for local variables of ours it doesn't matter because they are about to go out of scope. But what if the compiler instantiates any for its own use, e.g. ...
xen/arch/x86/setup.c static void __init move_memory( · line 574
Quoted 10 lines
xen/arch/x86/setup.c +10
-0+/* +* End of the critical region. Updates to locals and globals now work as +* expected. +* +* Updates to local variables which have been spilled to the stack since +* the memcpy() have been lost. But we don't care, because they're all +* going out of scope imminently. +*/ + +printk ("New Xen image base address: %#lx\n" , xen_phys_start);Quoted 8 lines
... the result of the address calculation for the string literal here? Such auxiliary calculations can happen at any point in the function, and won't necessarily (hence the example chosen) become impossible for the compiler to do with the memory clobber in the
asm(). And while the string literal's address is likely cheap enough to calculate right in the function invocation sequence (and an option would also be to retain theprintk()in the caller), other instrumentation options could be undermined by this as well.Quoted 6 lines
Right now, the compiler is free to calculate the address of the string literal in a register, and move it the other side of the TLB flush. This will work just fine.
However, the compiler cannot now, or ever in the future, spill such a calculation to the stack.
I'm afraid the compiler's view at things is different: Whatever it puts on the stack is viewed as virtual registers, unaffected by a memory clobber (of course there can be effects resulting from uses of named variables). Look at -O3 output of gcc12 (which is what I happened to play with; I don't think it's overly version dependent) for this (clearly contrived) piece of code:
int __attribute__((const)) calc(int);
int test(int i) { int j = calc(i); asm("nopl %0" : "+m" (j)); asm("nopq %%rsp" ::: "memory", "ax", "cx", "dx", "bx", "bp", "si", "di", "r8", "r9", "r10", "r11", "r12", "r13", "r14", "r15"); j = calc(i); asm("nopl %0" :: "m" (j)); return j; }
Quoted 6 lines
It instantiates a local on the stack for the result of
calc(); it does not re-invokecalc()a 2nd time. Which means the memory clobber in the middleasm()does not affect that (and by implication: any) stack slot.Using godbolt I can also see that clang15 agrees with gcc12 in this regard. I didn't bother trying other versions.
Well this is problematic, because it contradicts what we depend on asm("":::"memory") doing...
https://godbolt.org/z/xeGMc3sM9
But I don't fully agree with the conclusions drawn by this example.
It only instantiates a local on the stack because you force a memory operand to satisfy the "m" constraints, not to satisfy the "memory" constraint.
By declaring calc as const, you are permitting the compiler to make an explicit transformation to delete one of the calls, irrespective of anything else in the function.
It is weird that 'j' ends up taking two stack slots when would be absolutely fine for it to only have 1, and indeed this is what happens when you remove the first and third asm()'s. It is these which force 'j' to be on the stack, not the memory clobber in the middle.
Observe that after commenting those two out, Clang transforms things to spill 'i' onto the stack, rather than 'j', and then tail-call calc() on the way out. This is actually deleting the first calc() call, rather than the second.
~Andrew
Re: [PATCH v2-ish] x86/boot: Factor move_xen() out of __start_xen()
Quoted 63 lines · Andrew Cooper
Quoted 62 lines · Jan Beulich
Quoted 30 lines · Andrew Cooper
Quoted 21 lines · Jan Beulich
xen/arch/x86/setup.c static void __init move_memory( · line 563
Quoted 5 lines · Andrew Cooper
xen/arch/x86/setup.c +5
-0+"mov %[cr4], %%cr4\n\t" /* CR4.PGE = 1 */ + : [cr4] "=&a" (tmp) /* Could be "r", but "a" makes better asm */ + : [cr3] "r" (__pa(idle_pg_table)), + [pge] "i" (X86_CR4_PGE) + : "memory" );The removed stack copying worries me, to be honest. Yes, for local variables of ours it doesn't matter because they are about to go out of scope. But what if the compiler instantiates any for its own use, e.g. ...
xen/arch/x86/setup.c static void __init move_memory( · line 574
Quoted 10 lines
xen/arch/x86/setup.c +10
-0+/* +* End of the critical region. Updates to locals and globals now work as +* expected. +* +* Updates to local variables which have been spilled to the stack since +* the memcpy() have been lost. But we don't care, because they're all +* going out of scope imminently. +*/ + +printk ("New Xen image base address: %#lx\n" , xen_phys_start);Quoted 8 lines
... the result of the address calculation for the string literal here? Such auxiliary calculations can happen at any point in the function, and won't necessarily (hence the example chosen) become impossible for the compiler to do with the memory clobber in the
asm(). And while the string literal's address is likely cheap enough to calculate right in the function invocation sequence (and an option would also be to retain theprintk()in the caller), other instrumentation options could be undermined by this as well.Quoted 6 lines
Right now, the compiler is free to calculate the address of the string literal in a register, and move it the other side of the TLB flush. This will work just fine.
However, the compiler cannot now, or ever in the future, spill such a calculation to the stack.
I'm afraid the compiler's view at things is different: Whatever it puts on the stack is viewed as virtual registers, unaffected by a memory clobber (of course there can be effects resulting from uses of named variables). Look at -O3 output of gcc12 (which is what I happened to play with; I don't think it's overly version dependent) for this (clearly contrived) piece of code:
int __attribute__((const)) calc(int);
int test(int i) { int j = calc(i); asm("nopl %0" : "+m" (j)); asm("nopq %%rsp" ::: "memory", "ax", "cx", "dx", "bx", "bp", "si", "di", "r8", "r9", "r10", "r11", "r12", "r13", "r14", "r15"); j = calc(i); asm("nopl %0" :: "m" (j)); return j; }It instantiates a local on the stack for the result of
calc(); it does not re-invokecalc()a 2nd time. Which means the memory clobber in the middleasm()does not affect that (and by implication: any) stack slot.
Quoted 5 lines
Quoted 2 lines
Using godbolt I can also see that clang15 agrees with gcc12 in this regard. I didn't bother trying other versions.
Well this is problematic, because it contradicts what we depend on
asm("":::"memory")doing...
Does it? I'm unaware of instances where we use "memory" to deal with local variables.
Quoted 7 lines
https://godbolt.org/z/xeGMc3sM9
But I don't fully agree with the conclusions drawn by this example.
It only instantiates a local on the stack because you force a memory operand to satisfy the "m" constraints, not to satisfy the "memory" constraint.
Sure, all I wanted to show is that the compiler may make such transformations. As said, the example is clearly contrived, but I also didn't want to spend too much time on finding a yet better one.
Quoted 9 lines
By declaring calc as const, you are permitting the compiler to make an explicit transformation to delete one of the calls, irrespective of anything else in the function.
It is weird that 'j' ends up taking two stack slots when would be absolutely fine for it to only have 1, and indeed this is what happens when you remove the first and third asm()'s. It is these which force 'j' to be on the stack, not the memory clobber in the middle.
Quoted 4 lines
Observe that after commenting those two out, Clang transforms things to spill 'i' onto the stack, rather than 'j', and then tail-call
calc()on the way out. This is actually deleting the firstcalc()call, rather than the second.
Which would still demonstrate what I wanted to demonstrate - we can't assume that "memory" guards against the use of stack locals by the compiler.
Jan