EFI runtime and kexec
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 37 lines · Borislav Petkov
Hi guys,
so I was talking with mfleming on IRC and he said I should talk to you about it. I actually pestered hpa about it already, sorry :).
So I've been looking into making EFI runtime services available in the kexec'ed kernel. What I've found out so far is that
efi_enter_virtual_mode()in the first kernel iterates over the EFI memmap and ioremaps all those EFI runtime services mappings. On my DELL workstation it looks like the dump below.Now, once this is done and
SetVirtualAddressMap()is called, for the duration of this boot, those virtual addresses cannot change as the UEFI spec states: "A virtual address map may only be applied one time. Once the runtime system is in virtual mode, calls to this function returnEFI_UNSUPPORTED."Now, looking at those mappings, they're spread all over the VA space and their size is ~159.887Mb (which is 159MB too many for a goddam BIOS crap but whatever, everyone is jumping on this train so I'm gonna have to follow, unwillingly.</EndOfRant>)
AFAICT, in a kexec kernel I'd have to recreate the exact-same mappings, i.e.
phys_addr-> va for all those regions. And since I'm not an mm guy, I'd rather ask the experts before I dive into a catch-22 thing.So even if I manage to do the mappings in the kexec kernel correctly for all those regions which
efi_ioremap()serves only with direct mappings throughinit_memory_mapping(), I probably won't be that lucky with regions of typeEFI_MEMORY_MAPPED_IO(type0xbbelow) for which we really do ioremap and those virtual addresses I can't control, AFAICT.So what do you guys think, would it be possible
* to make all EFI runtime services use predefined mappings which are globally valid and I can read them out in kexec or
Quoted 7 lines
* make those mappings virtually contiguous so that kexec kernel only gets a
va_startand a size and after that it knows what to do or* an even better idea.
In general, any suggestion is appreciated.
Adding a few more people.
This has been a big topic, and yes, we have a problem.
We seem to have a few options:
1. We could always map 1:1, with the EFI mappings being in the "user" part of the virtual address space. This MAY be what Windows does already. Some Apple platforms are known to fail in this configuration, but perhaps we can blacklist those platforms or do something special.
2. We could always map them into a fixed address that can be relied upon to be consistent. The most logical such area is the second quadrant of the address space (again, in the "user" portion.) It would be beneficial if we could define it so that whenever Linux needs to go to more than 48 virtual address bits at some point in the future this can be compatible between 48-bit and N-bit kernels, but if that is the only thing that breaks, then oh well.
3. We could just always map at the kernel virtual address. The 64-bit address space is large enough that we could make every ioremap() land at its identity-mapped address instead of in a unique part of the virtual address space.
4. We could export a table of mappings to the kexec'd kernel. In that case, we have to re-establish those mappings very early in the kernel boot so that nothing else steps on them.
What is quite interesting in your case is that you have a mishmash of the identity-mapped and the non-identity-mapped mappings.
-hpa
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 70 lines · H. Peter Anvin
Quoted 37 lines · Borislav Petkov
Hi guys,
so I was talking with mfleming on IRC and he said I should talk to you about it. I actually pestered hpa about it already, sorry :).
So I've been looking into making EFI runtime services available in the kexec'ed kernel. What I've found out so far is that
efi_enter_virtual_mode()in the first kernel iterates over the EFI memmap and ioremaps all those EFI runtime services mappings. On my DELL workstation it looks like the dump below.Now, once this is done and
SetVirtualAddressMap()is called, for the duration of this boot, those virtual addresses cannot change as the UEFI spec states: "A virtual address map may only be applied one time. Once the runtime system is in virtual mode, calls to this function returnEFI_UNSUPPORTED."Now, looking at those mappings, they're spread all over the VA space and their size is ~159.887Mb (which is 159MB too many for a goddam BIOS crap but whatever, everyone is jumping on this train so I'm gonna have to follow, unwillingly.</EndOfRant>)
AFAICT, in a kexec kernel I'd have to recreate the exact-same mappings, i.e.
phys_addr-> va for all those regions. And since I'm not an mm guy, I'd rather ask the experts before I dive into a catch-22 thing.So even if I manage to do the mappings in the kexec kernel correctly for all those regions which
efi_ioremap()serves only with direct mappings throughinit_memory_mapping(), I probably won't be that lucky with regions of typeEFI_MEMORY_MAPPED_IO(type0xbbelow) for which we really do ioremap and those virtual addresses I can't control, AFAICT.So what do you guys think, would it be possible
* to make all EFI runtime services use predefined mappings which are globally valid and I can read them out in kexec or
Quoted 7 lines
* make those mappings virtually contiguous so that kexec kernel only gets a
va_startand a size and after that it knows what to do or* an even better idea.
In general, any suggestion is appreciated.
Adding a few more people.
This has been a big topic, and yes, we have a problem.
We seem to have a few options:
1. We could always map 1:1, with the EFI mappings being in the "user" part of the virtual address space. This MAY be what Windows does already. Some Apple platforms are known to fail in this configuration, but perhaps we can blacklist those platforms or do something special.
2. We could always map them into a fixed address that can be relied upon to be consistent. The most logical such area is the second quadrant of the address space (again, in the "user" portion.) It would be beneficial if we could define it so that whenever Linux needs to go to more than 48 virtual address bits at some point in the future this can be compatible between 48-bit and N-bit kernels, but if that is the only thing that breaks, then oh well.
3. We could just always map at the kernel virtual address. The 64-bit address space is large enough that we could make every
ioremap()land at its identity-mapped address instead of in a unique part of the virtual address space.
Quoted 6 lines
4. We could export a table of mappings to the kexec'd kernel. In that case, we have to re-establish those mappings very early in the kernel boot so that nothing else steps on them.
What is quite interesting in your case is that you have a mishmash of the identity-mapped and the non-identity-mapped mappings.
Yeah, the mishmash comes from regions of type EFI_MEMORY_MAPPED_IO which are really ioremapped instead of returning the kernel virtual address.
Btw, I always tend to like the simplest approaches so option 3. is kinda winking at me right now. I don't know whether for those EFI_MEMORY_MAPPED_IO type regions though, we can simply return the identity-mapped address.
If we can, the advantage would be great because then the kexec kernel would simply parse the efi memmap and use __va() on the physical addresses there and no need for special option passing to it.
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 35 lines · Borislav Petkov
Quoted 25 lines
Adding a few more people.
This has been a big topic, and yes, we have a problem.
We seem to have a few options:
1. We could always map 1:1, with the EFI mappings being in the "user" part of the virtual address space. This MAY be what Windows does already. Some Apple platforms are known to fail in this configuration, but perhaps we can blacklist those platforms or do something special.
2. We could always map them into a fixed address that can be relied upon to be consistent. The most logical such area is the second quadrant of the address space (again, in the "user" portion.) It would be beneficial if we could define it so that whenever Linux needs to go to more than 48 virtual address bits at some point in the future this can be compatible between 48-bit and N-bit kernels, but if that is the only thing that breaks, then oh well.
3. We could just always map at the kernel virtual address. The 64-bit address space is large enough that we could make every
ioremap()land at its identity-mapped address instead of in a unique part of the virtual address space.Quoted 6 lines
4. We could export a table of mappings to the kexec'd kernel. In that case, we have to re-establish those mappings very early in the kernel boot so that nothing else steps on them.
What is quite interesting in your case is that you have a mishmash of the identity-mapped and the non-identity-mapped mappings.
Yeah, the mishmash comes from regions of type
EFI_MEMORY_MAPPED_IOwhich are really ioremapped instead of returning the kernel virtual address.
Quoted 9 lines
Btw, I always tend to like the simplest approaches so option 3. is kinda winking at me right now. I don't know whether for those
EFI_MEMORY_MAPPED_IOtype regions though, we can simply return the identity-mapped address.If we can, the advantage would be great because then the kexec kernel would simply parse the efi memmap and use
__va()on the physical addresses there and no need for special option passing to it.
We can, and in fact we could do this for *all* ioremap()s in the 64-bit kernel. This doesn't help the 32-bit kernel in any way, however.
One thing I *really* don't like about it is that it exposes the kernel virtual address map as an ABI.
-hpa
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 35 lines · H. Peter Anvin
Quoted 35 lines
Quoted 20 lines
We seem to have a few options:
1. We could always map 1:1, with the EFI mappings being in the "user" part of the virtual address space. This MAY be what Windows does already. Some Apple platforms are known to fail in this configuration, but perhaps we can blacklist those platforms or do something special.
2. We could always map them into a fixed address that can be relied upon to be consistent. The most logical such area is the second quadrant of the address space (again, in the "user" portion.) It would be beneficial if we could define it so that whenever Linux needs to go to more than 48 virtual address bits at some point in the future this can be compatible between 48-bit and N-bit kernels, but if that is the only thing that breaks, then oh well.
3. We could just always map at the kernel virtual address. The 64-bit address space is large enough that we could make every
ioremap()land at its identity-mapped address instead of in a unique part of the virtual address space.Quoted 6 lines
4. We could export a table of mappings to the kexec'd kernel. In that case, we have to re-establish those mappings very early in the kernel boot so that nothing else steps on them.
What is quite interesting in your case is that you have a mishmash of the identity-mapped and the non-identity-mapped mappings.
Yeah, the mishmash comes from regions of type
EFI_MEMORY_MAPPED_IOwhich are really ioremapped instead of returning the kernel virtual address.Btw, I always tend to like the simplest approaches so option 3. is kinda winking at me right now. I don't know whether for those
EFI_MEMORY_MAPPED_IOtype regions though, we can simply return the identity-mapped address.
Quoted 7 lines
Quoted 4 lines
If we can, the advantage would be great because then the kexec kernel would simply parse the efi memmap and use
__va()on the physical addresses there and no need for special option passing to it.We can, and in fact we could do this for *all* ioremap()s in the 64-bit kernel. This doesn't help the 32-bit kernel in any way, however.
Right, ok.
Quoted 2 lines
One thing I *really* don't like about it is that it exposes the kernel virtual address map as an ABI.
Hmm, yeah, that's nasty. This also means option #2 can go too because of the fixed addresses. Option #1 is also kinda polluting user address space so maybe the most elegant one would be #4, AFAICT.
We just need a nice mechanism to tell those mappings to the kexec-d kernel and when it starts, to establish them right away.
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 3 lines · Borislav Petkov
Hmm, yeah, that's nasty. This also means option #2 can go too because of the fixed addresses. Option #1 is also kinda polluting user address space
User address space is there to be polluted. Create a "kernel thread" for invoking EFI, except that this kernel thread actually has userspace page tables. Set up those page tables however the hell you like, and then just make sure you always invoke EFI runtime services from that thread.
Quoted 4 lines
so maybe the most elegant one would be #4, AFAICT.
We just need a nice mechanism to tell those mappings to the kexec-d kernel and when it starts, to establish them right away.
Well, there's a *shitload* of things we already need to hand off to the kexec'd kernel that we don't. We don't pass the EFI system table, which means that even the ACPI tables aren't found by the child kernel. And all of the setup_data is missing. And there's probably more.
I was looking at this, but there was *other* breakage with kexec which was being sorted out at the time, and I kind of got distracted into doing CSM support in SeaBIOS.
The other option, for the long term, is to fix the damn firmware to allow SetVirtualAddressMap to be called more than once. It was stupid for it to be a one-time call anyway. I have a test build of OVMF here which does just that, but haven't got round to testing it yet. I wish I'd been given the *patch* not just a binary though...
Signature · 1 line
dwmw2
A message that is not in the archive (sent off-list or not yet received): 1362180625.29011.4.camel-Fexsq3y4057IgHVZqg5X0TlWvGAXklZc@public.gmane.org
Quoted 9 lines · David Woodhouse
Quoted 3 lines · Borislav Petkov
Hmm, yeah, that's nasty. This also means option #2 can go too because of the fixed addresses. Option #1 is also kinda polluting user address space
User address space is there to be polluted. Create a "kernel thread" for invoking EFI, except that this kernel thread actually has userspace page tables. Set up those page tables however the hell you like, and then just make sure you always invoke EFI runtime services from that thread.
Oh, I meant to mention: Matthew once told me about a bizarre issue on MacBook firmware, iirc. Some stuff doesn't work if the virtual address is above 2GiB. Or below 2GiB. I don't really remember what Matthew told me at all, to be honest. Only that Here Be Dragons and we don't quite have free rein about where we place stuff. But I suspect we ought to be able to find *somewhere* in the user address space that works, even for 32-bit kernels.
Signature · 1 line
dwmw2
A message that is not in the archive (sent off-list or not yet received): 1362180853.29011.6.camel-Fexsq3y4057IgHVZqg5X0TlWvGAXklZc@public.gmane.org
Quoted 7 lines · David Woodhouse
Oh, I meant to mention: Matthew once told me about a bizarre issue on MacBook firmware, iirc. Some stuff doesn't work if the virtual address is above 2GiB. Or below 2GiB. I don't really remember what Matthew told me at all, to be honest. Only that Here Be Dragons and we don't quite have free rein about where we place stuff. But I suspect we ought to be able to find *somewhere* in the user address space that works, even for 32-bit kernels.
Apple's enough of a special case that I've no objection to special-casing them.
Signature · 1 line
Matthew Garrett | [email protected]
Just commenting on this one for now, the rest tomorrow cuz I'm half asleep.
Quoted 3 lines · David Woodhouse
The other option, for the long term, is to fix the damn firmware to allow SetVirtualAddressMap to be called more than once. It was stupid for it to be a one-time call anyway.
Now this would be the cleanest solution. If we can do that, we can then simply call efi_enter_virtual_mode() in the kexec'd kernel without the need to pass any options to it. Actually, the kexec'd kernel can probably run the same efi code as the papa kernel.
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 4 lines · Borislav Petkov
Just commenting on this one for now, the rest tomorrow cuz I'm half asleep.
On Fri, Mar 01, 2013 at 11:30:25PM +0000, David Woodhouse wrote:
Quoted 9 lines
Quoted 3 lines
The other option, for the long term, is to fix the damn firmware to allow SetVirtualAddressMap to be called more than once. It was stupid for it to be a one-time call anyway.
Now this would be the cleanest solution. If we can do that, we can then simply call
efi_enter_virtual_mode()in the kexec'd kernel without the need to pass any options to it. Actually, the kexec'd kernel can probably run the same efi code as the papa kernel.
Well, if we can genuinely run with a zero relocation we don't even need to worry about that.
-hpa
Quoted 10 lines · David Woodhouse
Quoted 3 lines · Borislav Petkov
Hmm, yeah, that's nasty. This also means option #2 can go too because of the fixed addresses. Option #1 is also kinda polluting user address space
User address space is there to be polluted. Create a "kernel thread" for invoking EFI, except that this kernel thread actually has userspace page tables. Set up those page tables however the hell you like, and then just make sure you always invoke EFI runtime services from that thread.
Just FYI: as far as I know we already have such a "thread".
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines · H. Peter Anvin
Quoted 5 lines · David Woodhouse
Quoted 3 lines · Borislav Petkov
Hmm, yeah, that's nasty. This also means option #2 can go too because of the fixed addresses. Option #1 is also kinda polluting user address space
Quoted 7 lines
Quoted 5 lines
User address space is there to be polluted. Create a "kernel thread" for invoking EFI, except that this kernel thread actually has userspace page tables. Set up those page tables however the hell you like, and then just make sure you always invoke EFI runtime services from that thread.
Just FYI: as far as I know we already have such a "thread".
In that case it seems like this option is fairly much a no-brainer.
Do we make any calls to runtime services during early boot (before we'd have such a thread set up for use)?
Signature · 1 line
dwmw2
A message that is not in the archive (sent off-list or not yet received): 1362189072.32131.6.camel-Fexsq3y4057IgHVZqg5X0TlWvGAXklZc@public.gmane.org
Yes, but way before... not a problem. It isn't a thread in the scheduler sense.
Quoted 25 lines · David Woodhouse
Quoted 3 lines · H. Peter Anvin
Quoted 2 lines · David Woodhouse
Quoted 1 line · Borislav Petkov
Hmm, yeah, that's nasty. This also means option #2 can go too
because
Quoted 1 line
Quoted 1 line
Quoted 1 line
of the fixed addresses. Option #1 is also kinda polluting user
address
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
space
User address space is there to be polluted. Create a "kernel
thread" for
Quoted 1 line
Quoted 1 line
invoking EFI, except that this kernel thread actually has userspace
page
Quoted 1 line
Quoted 1 line
tables. Set up those page tables however the hell you like, and
then
Quoted 1 line
Quoted 1 line
just make sure you always invoke EFI runtime services from that
thread.
Quoted 3 lines
Quoted 1 line
Just FYI: as far as I know we already have such a "thread".
In that case it seems like this option is fairly much a no-brainer.
Do we make any calls to runtime services during early boot (before we'd have such a thread set up for use)?
Signature · 1 line
Sent from my mobile phone. Please excuse brevity and lack of formatting.
A message that is not in the archive (sent off-list or not yet received): 94a3c086-b0b2-4820-8f4c-66d1496b89dc-2ueSQiBKiTY7tOexoI0I+QC/[email protected]
Quoted 2 lines · H. Peter Anvin
Yes, but way before... not a problem. It isn't a thread in the scheduler sense.
Are you so enigmatically referring to the swapper?
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--Quoted 5 lines · Borislav Petkov
Hmm, yeah, that's nasty. This also means option #2 can go too because of the fixed addresses. Option #1 is also kinda polluting user address space so maybe the most elegant one would be #4, AFAICT.
No, it doesn't pollute the user address space, because the EFI BIOS is effectively a process. We *have* to switch cr3 anyway around EFI calls, because BIOSes in the field are known to use physical addresses instead of virtual.
If Windows really does map EFI BIOS 1:1 then that explains why, and is probably going to be the stable thing going forward.
-hpa