xen/Arm: Probe the entry point address of an uImage correctly
PATCH 1/1 xen/Arm: Probe the entry point address of an uImage correctly
Currently, kernel_uimage_probe() does not set info->zimage.start. As a result, it contains the default value (ie 0). This causes, kernel_zimage_place() to treat the binary (contained within uImage) as position independent executable. Thus, it loads it at an incorrect address.
The correct approach would be to read "uimage.ep" and set info->zimage.start. This will ensure that the binary is loaded at the correct address.
---
I uncovered this issue while loading Zephyr as a dom0less domU with Xen on R52 FVP. Zephyr builds with static device tree. Thus, the load address is always fixed.
xen/arch/arm/kernel.c | 2 ++ 1 file changed, 2 insertions(+)
xen/arch/arm/kernel.c +2 -0
@@ -222,6 +222,8 @@ static int __init kernel_uimage_probe(struct kernel_info *info,if ( len > size -sizeof (uimage) )return -EINVAL ; + info->zimage.start =be32_to_cpu (uimage.ep); + info->zimage.kernel_addr = addr +sizeof (uimage); info->zimage.len = len;
Signature · 1 line
2.17.1
Hi,
Title extra NIT: I have seen it multiple time and so far refrain to say it. Please use 'arm' rather than 'Arm'. This is for consistency in the way we name the subsystem in the title.
Quoted 8 lines · Ayan Kumar Halder
Currently,
kernel_uimage_probe()does not setinfo->zimage.start. As a result, it contains the default value (ie 0). This causes,kernel_zimage_place()to treat the binary (contained within uImage) as position independent executable. Thus, it loads it at an incorrect address.The correct approach would be to read "uimage.ep" and set
info->zimage.start. This will ensure that the binary is loaded at the correct address.
In non-statically allocated setup, a user doesn't know where the memory for dom0/domU will be allocated.
So I think this was correct to ignore the address. In fact, I am worry that...
Quoted 16 lines
Signed-off-by Ayan Kumar Halder <[email protected]>---
I uncovered this issue while loading Zephyr as a dom0less domU with Xen on R52 FVP. Zephyr builds with static device tree. Thus, the load address is always fixed.
xen/arch/arm/kernel.c | 2 ++ 1 file changed, 2 insertions(+)xen/arch/arm/kernel.c +0
-0
xen/arch/arm/kernel.c static int __init kernel_uimage_probe(struct kernel_info *info, · line 225
Quoted 4 lines
xen/arch/arm/kernel.c +1
-0@@ -222,6 +222,8 @@ static int __init kernel_uimage_probe(struct kernel_info *info,if ( len > size -sizeof (uimage) )return -EINVAL ; + info->zimage.start =be32_to_cpu (uimage.ep);
... this will now ended up to break anyone that may have set an address but didn't care where it should be loaded.
I also understand your use case but now, we have contradictory approaches. I am not entirely sure how we can solve it. We may have to break those users (Cc some folks that may use it). But we should figure out what is the alternative for them.
If we decide to break those users, then this should be documented in the commit message and in docs/misc/arm/booting.txt (which interestingly didn't mention uImage).
Cheers,
Signature · 1 line
Julien Grall
Quoted 1 line · Julien Grall
Hi,
Hi Julien,
Quoted 2 lines
Title extra NIT: I have seen it multiple time and so far refrain to say it. Please use 'arm' rather than 'Arm'. This is for consistency in the way we name the subsystem in the title.
I will take care of it henceforth.
Quoted 26 lines · Ayan Kumar Halder
Quoted 8 lines
Currently,
kernel_uimage_probe()does not setinfo->zimage.start. As a result, it contains the default value (ie 0). This causes,kernel_zimage_place()to treat the binary (contained within uImage) as position independent executable. Thus, it loads it at an incorrect address.The correct approach would be to read "uimage.ep" and set
info->zimage.start. This will ensure that the binary is loaded at the correct address.In non-statically allocated setup, a user doesn't know where the memory for dom0/domU will be allocated.
So I think this was correct to ignore the address. In fact, I am worry that...
Quoted 11 lines
Signed-off-by Ayan Kumar Halder <[email protected]>---
I uncovered this issue while loading Zephyr as a dom0less domU with Xen on R52 FVP. Zephyr builds with static device tree. Thus, the load address is always fixed.
xen/arch/arm/kernel.c| 2 ++1 file changed, 2 insertions(+)
Quoted 10 lines
xen/arch/arm/kernel.c static int __init kernel_uimage_probe(struct · line 225
Quoted 9 lines
xen/arch/arm/kernel.c +0
-0@@ -222,6 +222,8 @@ static int __init kernel_uimage_probe(struct kernel_info *info, if ( len > size - sizeof(uimage) ) return -EINVAL; + info->zimage.start = be32_to_cpu(uimage.ep);... this will now ended up to break anyone that may have set an address but didn't care where it should be loaded.
Can we use a config option (STATIC_MEMORY or any option you may suggest) to distinguish between the two ?
Or using some magic number (eg 0xdeadbeef) for uimage.ep which will denote position independent ? This needs to be documented in docs/misc/arm/booting.txt.
Or ...
Quoted 2 lines
I also understand your use case but now, we have contradictory approaches. I am not entirely sure how we can solve it. We may have to break those users (Cc some folks that may use it). But we should figure out what is the alternative for them.
I am open to any alternative suggestion.
Quoted 2 lines
If we decide to break those users, then this should be documented in the commit message and in
docs/misc/arm/booting.txt(which interestingly didn't mention uImage).
I agree.
- Ayan
Quoted 3 lines
Cheers,
Hi,
Quoted 48 lines · Julien Grall
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Hi,
Title extra NIT: I have seen it multiple time and so far refrain to say it. Please use 'arm' rather than 'Arm'. This is for consistency in the way we name the subsystem in the title.
Quoted 8 lines · Ayan Kumar Halder
Currently,
kernel_uimage_probe()does not setinfo->zimage.start. As a result, it contains the default value (ie 0). This causes,kernel_zimage_place()to treat the binary (contained within uImage) as position independent executable. Thus, it loads it at an incorrect address.The correct approach would be to read "uimage.ep" and set
info->zimage.start. This will ensure that the binary is loaded at the correct address.In non-statically allocated setup, a user doesn't know where the memory for dom0/domU will be allocated.
So I think this was correct to ignore the address. In fact, I am worry that...
Quoted 16 lines
Signed-off-by Ayan Kumar Halder <[email protected]>---
I uncovered this issue while loading Zephyr as a dom0less domU with Xen on R52 FVP. Zephyr builds with static device tree. Thus, the load address is always fixed.
xen/arch/arm/kernel.c | 2 ++ 1 file changed, 2 insertions(+)xen/arch/arm/kernel.c +0
-0xen/arch/arm/kernel.c static int __init kernel_uimage_probe(struct kernel_info *info, · line 225
Quoted 4 lines
xen/arch/arm/kernel.c +1
-0@@ -222,6 +222,8 @@ static int __init kernel_uimage_probe(struct kernel_info *info,if ( len > size -sizeof (uimage) )return -EINVAL ; + info->zimage.start =be32_to_cpu (uimage.ep);... this will now ended up to break anyone that may have set an address but didn't care where it should be loaded.
Quoted 9 lines
I also understand your use case but now, we have contradictory approaches. I am not entirely sure how we can solve it. We may have to break those users (Cc some folks that may use it). But we should figure out what is the alternative for them.
If we decide to break those users, then this should be documented in the commit message and in
docs/misc/arm/booting.txt(which interestingly didn't mention uImage).
So the first issue with Zephyr is that it does not support zImage protocol for arm32. Volodymyr added support only for Image header for arm64 Zephyr.
I guess this is why Ayan, willing to boot it on Xen (arm32), decided to add u-boot header.
Now, there is also a question about supporting arm64 uImage kernels. In Xen kernel_zimage_place, we do:
#ifdef CONFIG_ARM_64
if ( info->type == DOMAIN_64BIT )
return info->mem.bank[0].start + info->zimage.text_offset;#endif
So if we modify the uImage behavior for arm32, we will break consistency with arm64 (we would take uImage entry point address into account for arm32 but not for arm64). At the moment at least they are in sync.
Quoted 4 lines
Cheers,
--
Julien Grall
~Michal
Quoted 22 lines · Julien Grall
Quoted 8 lines · Ayan Kumar Halder
Currently,
kernel_uimage_probe()does not setinfo->zimage.start. As a result, it contains the default value (ie 0). This causes,kernel_zimage_place()to treat the binary (contained within uImage) as position independent executable. Thus, it loads it at an incorrect address.The correct approach would be to read "uimage.ep" and set
info->zimage.start. This will ensure that the binary is loaded at the correct address.In non-statically allocated setup, a user doesn't know where the memory for dom0/domU will be allocated.
So I think this was correct to ignore the address. In fact, I am worry that...
Quoted 6 lines
xen/arch/arm/kernel.c +0
-0@@ -222,6 +222,8 @@ static int __init kernel_uimage_probe(struct kernel_info *info,if ( len > size -sizeof (uimage) )return -EINVAL ;
Quoted 8 lines
xen/arch/arm/kernel.c static int __init kernel_uimage_probe(struct kernel_info *info, · line 225
Quoted 1 line
xen/arch/arm/kernel.c +1
-0+ info->zimage.start =be32_to_cpu (uimage.ep);... this will now ended up to break anyone that may have set an address but didn't care where it should be loaded.
I also understand your use case but now, we have contradictory approaches. I am not entirely sure how we can solve it. We may have to break those users (Cc some folks that may use it). But we should figure out what is the alternative for them.
I don't know anything about the uimage format, but is the ep field required to be non-zero? If not, it being non-zero would retain prior behavior.
Jan
Quoted 1 line · Michal Orzel
Hi,
Hi,
Quoted 63 lines · Julien Grall
Quoted 46 lines
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Hi,
Title extra NIT: I have seen it multiple time and so far refrain to say it. Please use 'arm' rather than 'Arm'. This is for consistency in the way we name the subsystem in the title.
Quoted 8 lines · Ayan Kumar Halder
Currently,
kernel_uimage_probe()does not setinfo->zimage.start. As a result, it contains the default value (ie 0). This causes,kernel_zimage_place()to treat the binary (contained within uImage) as position independent executable. Thus, it loads it at an incorrect address.The correct approach would be to read "uimage.ep" and set
info->zimage.start. This will ensure that the binary is loaded at the correct address.In non-statically allocated setup, a user doesn't know where the memory for dom0/domU will be allocated.
So I think this was correct to ignore the address. In fact, I am worry that...
Quoted 15 lines
Signed-off-by Ayan Kumar Halder <[email protected]>---
I uncovered this issue while loading Zephyr as a dom0less domU with Xen on R52 FVP. Zephyr builds with static device tree. Thus, the load address is always fixed.
xen/arch/arm/kernel.c | 2 ++ 1 file changed, 2 insertions(+)xen/arch/arm/kernel.c +0
-0xen/arch/arm/kernel.c static int __init kernel_uimage_probe(struct kernel_info *info, · line 225
Quoted 4 lines
xen/arch/arm/kernel.c +1
-0@@ -222,6 +222,8 @@ static int __init kernel_uimage_probe(struct kernel_info *info,if ( len > size -sizeof (uimage) )return -EINVAL ; + info->zimage.start =be32_to_cpu (uimage.ep);... this will now ended up to break anyone that may have set an address but didn't care where it should be loaded.
Quoted 9 lines
I also understand your use case but now, we have contradictory approaches. I am not entirely sure how we can solve it. We may have to break those users (Cc some folks that may use it). But we should figure out what is the alternative for them.
If we decide to break those users, then this should be documented in the commit message and in
docs/misc/arm/booting.txt(which interestingly didn't mention uImage).So the first issue with Zephyr is that it does not support zImage protocol for arm32. Volodymyr added support only for Image header for arm64 Zephyr.
I guess this is why Ayan, willing to boot it on Xen (arm32), decided to add u-boot header.Now, there is also a question about supporting arm64 uImage kernels. In Xen
kernel_zimage_place, we do:
Quoted 8 lines
#ifdef CONFIG_ARM_64 if ( info->type == DOMAIN_64BIT ) return info->mem.bank[0].start + info->zimage.text_offset;#endif
So if we modify the uImage behavior for arm32, we will break consistency with arm64 (we would take uImage entry point address into account for arm32 but not for arm64). At the moment at least they are in sync.
IIUC, at present arm64 and arm32 are not in sync.
For Arm32, we do not consider "info->zimage.text_offset".
- Ayan
Quoted 7 lines
Quoted 6 lines
Cheers,
--
Julien Grall~Michal
Hi,
Quoted 54 lines · Michal Orzel
Quoted 53 lines · Julien Grall
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Hi,
Title extra NIT: I have seen it multiple time and so far refrain to say it. Please use 'arm' rather than 'Arm'. This is for consistency in the way we name the subsystem in the title.
Quoted 8 lines · Ayan Kumar Halder
Currently,
kernel_uimage_probe()does not setinfo->zimage.start. As a result, it contains the default value (ie 0). This causes,kernel_zimage_place()to treat the binary (contained within uImage) as position independent executable. Thus, it loads it at an incorrect address.The correct approach would be to read "uimage.ep" and set
info->zimage.start. This will ensure that the binary is loaded at the correct address.In non-statically allocated setup, a user doesn't know where the memory for dom0/domU will be allocated.
So I think this was correct to ignore the address. In fact, I am worry that...
Quoted 16 lines
Signed-off-by Ayan Kumar Halder <[email protected]>---
I uncovered this issue while loading Zephyr as a dom0less domU with Xen on R52 FVP. Zephyr builds with static device tree. Thus, the load address is always fixed.
xen/arch/arm/kernel.c | 2 ++ 1 file changed, 2 insertions(+)xen/arch/arm/kernel.c +0
-0xen/arch/arm/kernel.c static int __init kernel_uimage_probe(struct kernel_info *info, · line 225
Quoted 4 lines
xen/arch/arm/kernel.c +1
-0@@ -222,6 +222,8 @@ static int __init kernel_uimage_probe(struct kernel_info *info,if ( len > size -sizeof (uimage) )return -EINVAL ; + info->zimage.start =be32_to_cpu (uimage.ep);... this will now ended up to break anyone that may have set an address but didn't care where it should be loaded.
I also understand your use case but now, we have contradictory approaches. I am not entirely sure how we can solve it. We may have to break those users (Cc some folks that may use it). But we should figure out what is the alternative for them.
Quoted 7 lines
Quoted 4 lines
If we decide to break those users, then this should be documented in the commit message and in
docs/misc/arm/booting.txt(which interestingly didn't mention uImage).So the first issue with Zephyr is that it does not support zImage protocol for arm32. Volodymyr added support only for Image header for arm64 Zephyr.
I guess this is why Ayan, willing to boot it on Xen (arm32), decided to add u-boot header.
If that's the only reason, then I would rather prefer if we go with zImage for a few reasons:
- The zImage protocol has at least some documentation (not perfect) of
the expected state of the memory/registers when jumping to the image.
- AFAICT libxl is not (yet) supporting uImage. So this means zephyr
cannot be loaded on older Xen releases (not great).
Note this doesn't mean we should not fix Xen for uImage.
Quoted 10 lines
Now, there is also a question about supporting arm64 uImage kernels. In Xen
kernel_zimage_place, we do:#ifdef CONFIG_ARM_64 if ( info->type == DOMAIN_64BIT ) return info->mem.bank[0].start + info->zimage.text_offset;#endif
So if we modify the uImage behavior for arm32, we will break consistency with arm64 (we would take uImage entry point address into account for arm32 but not for arm64). At the moment at least they are in sync.
That's a good point. It would be best if the behavior is consistent.
Cheers,
Signature · 1 line
Julien Grall
Quoted 1 line · Julien Grall
Hi,
Hi,
Quoted 61 lines · Michal Orzel
Quoted 59 lines · Julien Grall
Quoted 58 lines
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Hi,
Title extra NIT: I have seen it multiple time and so far refrain to say it. Please use 'arm' rather than 'Arm'. This is for consistency in the way we name the subsystem in the title.
Quoted 8 lines · Ayan Kumar Halder
Currently,
kernel_uimage_probe()does not setinfo->zimage.start. As a result, it contains the default value (ie 0). This causes,kernel_zimage_place()to treat the binary (contained within uImage) as position independent executable. Thus, it loads it at an incorrect address.The correct approach would be to read "uimage.ep" and set
info->zimage.start. This will ensure that the binary is loaded at the correct address.In non-statically allocated setup, a user doesn't know where the memory for dom0/domU will be allocated.
So I think this was correct to ignore the address. In fact, I am worry that...
Quoted 16 lines
Signed-off-by Ayan Kumar Halder <[email protected]>---
I uncovered this issue while loading Zephyr as a dom0less domU with Xen on R52 FVP. Zephyr builds with static device tree. Thus, the load address is always fixed.
xen/arch/arm/kernel.c| 2 ++1 file changed, 2 insertions(+)xen/arch/arm/kernel.c +0
-0xen/arch/arm/kernel.c static int __init kernel_uimage_probe(struct · line 226
Quoted 5 lines
xen/arch/arm/kernel.c +1
-0@@ -222,6 +222,8 @@ static int __init kernel_uimage_probe(struct kernel_info *info, if ( len > size - sizeof(uimage) ) return -EINVAL; + info->zimage.start = be32_to_cpu(uimage.ep);... this will now ended up to break anyone that may have set an address but didn't care where it should be loaded.
I also understand your use case but now, we have contradictory approaches. I am not entirely sure how we can solve it. We may have to break those users (Cc some folks that may use it). But we should figure out what is the alternative for them.
If we decide to break those users, then this should be documented in the commit message and in
docs/misc/arm/booting.txt(which interestingly didn't mention uImage).
Quoted 7 lines
Quoted 3 lines
So the first issue with Zephyr is that it does not support zImage protocol for arm32. Volodymyr added support only for Image header for arm64 Zephyr.
I guess this is why Ayan, willing to boot it on Xen (arm32), decided to add u-boot header.If that's the only reason, then I would rather prefer if we go with zImage for a few reasons:
- The zImage protocol has at least some documentation (not perfect) of the expected state of the memory/registers when jumping to the image.
- AFAICT libxl is not (yet) supporting uImage. So this means zephyr cannot be loaded on older Xen releases (not great).
I am exploring for a similar option as Volodymyr ie support zimage protocol for arm32.
But for that I need some public documentation that explains the zimage header format for arm32.
Refer xen/arch/arm/kernel.c
#define ZIMAGE32_MAGIC_OFFSET 0x24
#define ZIMAGE32_START_OFFSET 0x28
#define ZIMAGE32_END_OFFSET 0x2c
#define ZIMAGE32_HEADER_LEN 0x30
#define ZIMAGE32_MAGIC 0x016f2818
Is this documented anywhere ?
I had a look at https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/arm/booting.rst , but there is nothing that explains the header format.
Quoted 10 lines
Note this doesn't mean we should not fix Xen for uImage.
Quoted 7 lines
Now, there is also a question about supporting arm64 uImage kernels. In Xen
kernel_zimage_place, we do:
#ifdefCONFIG_ARM_64
if (info->type==DOMAIN_64BIT)
return info->mem.bank[0].start +info->zimage.text_offset;
#endif
Quoted 5 lines
Quoted 3 lines
So if we modify the uImage behavior for arm32, we will break consistency with arm64 (we would take uImage entry point address into account for arm32 but not for arm64). At the moment at least they are in sync.
That's a good point. It would be best if the behavior is consistent.
Currently, kernel_zimage_place() is called for both uImage and zImage.
Will it be sane if we write a different function for uImage ?
Something like this ...
static paddr_t __init kernel_uimage_place(struct kernel_info *info)
{
/* Read and return uImage header's load address */
return be32_to_cpu(uimage.load);
}
This will be consistent across arm32 and arm64
- Ayan
Quoted 3 lines
Cheers,
Quoted 94 lines · Ayan Kumar Halder
Quoted 1 line · Julien Grall
Hi,
Hi,
Quoted 71 lines · Michal Orzel
Quoted 63 lines · Julien Grall
Quoted 53 lines
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Hi,
Title extra NIT: I have seen it multiple time and so far refrain to say it. Please use 'arm' rather than 'Arm'. This is for consistency in the way we name the subsystem in the title.
Quoted 9 lines · Ayan Kumar Halder
Currently,
kernel_uimage_probe()does not setinfo->zimage.start. As a result, it contains the default value (ie 0). This causes,kernel_zimage_place()to treat the binary (contained within uImage) as position independent executable. Thus, it loads it at an incorrect address.The correct approach would be to read "uimage.ep" and set
info->zimage.start. This will ensure that the binary is loaded at the correct address.In non-statically allocated setup, a user doesn't know where the memory for dom0/domU will be allocated.
So I think this was correct to ignore the address. In fact, I am worry that...
Quoted 18 lines
Signed-off-by Ayan Kumar Halder <[email protected]>---
I uncovered this issue while loading Zephyr as a dom0less domU with Xen on
R52 FVP. Zephyr builds with static device tree. Thus, the load address is
always fixed.
xen/arch/arm/kernel.c| 2 ++1 file changed, 2 insertions(+)xen/arch/arm/kernel.c +0
-0xen/arch/arm/kernel.c static int __init kernel_uimage_probe(struct · line 226
Quoted 5 lines
xen/arch/arm/kernel.c +1
-0@@ -222,6 +222,8 @@ static int __init kernel_uimage_probe(struct kernel_info *info, if ( len > size - sizeof(uimage) ) return -EINVAL; + info->zimage.start = be32_to_cpu(uimage.ep);... this will now ended up to break anyone that may have set an address but didn't care where it should be loaded.
Quoted 9 lines
I also understand your use case but now, we have contradictory approaches. I am not entirely sure how we can solve it. We may have to break those users (Cc some folks that may use it). But we should figure out what is the alternative for them.
If we decide to break those users, then this should be documented in the commit message and in
docs/misc/arm/booting.txt(which interestingly didn't mention uImage).Quoted 5 lines
So the first issue with Zephyr is that it does not support zImage protocol for arm32.
Volodymyr added support only for Image header for arm64 Zephyr. I guess this is why Ayan, willing to boot it on Xen (arm32), decided to add u-boot header.Quoted 6 lines
If that's the only reason, then I would rather prefer if we go with zImage for a few reasons:
- The zImage protocol has at least some documentation (not perfect) of the expected state of the memory/registers when jumping to the image.
- AFAICT libxl is not (yet) supporting uImage. So this means zephyr cannot be loaded on older Xen releases (not great).I am exploring for a similar option as Volodymyr ie support zimage protocol for arm32.
But for that I need some public documentation that explains the zimage header format for arm32.
Refer
xen/arch/arm/kernel.c#define
ZIMAGE32_MAGIC_OFFSET0x24
#defineZIMAGE32_START_OFFSET0x28
#define ZIMAGE32_END_OFFSET0x2c
#define ZIMAGE32_HEADER_LEN0x30
Quoted 7 lines
#define
ZIMAGE32_MAGIC0x016f2818Is this documented anywhere ?
I had a look at
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/arm/booting.rst
, but there is nothing that explains the header format.
I think this is the closest to documentation of it:
http://www.simtec.co.uk/products/SWLINUX/files/booting_article.html
Quoted 23 lines · Ayan Kumar Halder
Quoted 7 lines · Julien Grall
Quoted 5 lines · Michal Orzel
So the first issue with Zephyr is that it does not support zImage protocol for arm32.
Volodymyr added support only for Image header for arm64 Zephyr. I guess this is why Ayan, willing to boot it on Xen (arm32), decided to add u-boot header.Quoted 6 lines
If that's the only reason, then I would rather prefer if we go with zImage for a few reasons:
- The zImage protocol has at least some documentation (not perfect) of the expected state of the memory/registers when jumping to the image.
- AFAICT libxl is not (yet) supporting uImage. So this means zephyr cannot be loaded on older Xen releases (not great).I am exploring for a similar option as Volodymyr ie support zimage protocol for arm32.
But for that I need some public documentation that explains the zimage header format for arm32.
Refer
xen/arch/arm/kernel.c
Quoted 8 lines
#define
ZIMAGE32_MAGIC_OFFSET0x24
#defineZIMAGE32_START_OFFSET0x28
#define ZIMAGE32_END_OFFSET0x2c
#define ZIMAGE32_HEADER_LEN0x30#define
ZIMAGE32_MAGIC0x016f2818Is this documented anywhere ?
zImage (32) is entirely undocumented. What exists is from reverse engineering. I found this while doing some XTF work, and went right back to the source - the Linux maintainers.
There are things which Xen does wrong with zImage handling. But I haven't had time to transcribe my notes into something more coherent and submit fixes.
~Andrew
Quoted 133 lines · Ayan Kumar Halder
Quoted 1 line · Julien Grall
Hi,
Hi,
Quoted 72 lines · Michal Orzel
Quoted 64 lines · Julien Grall
Quoted 58 lines
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Hi,
Title extra NIT: I have seen it multiple time and so far refrain to say it. Please use 'arm' rather than 'Arm'. This is for consistency in the way we name the subsystem in the title.
Quoted 9 lines · Ayan Kumar Halder
Currently,
kernel_uimage_probe()does not setinfo->zimage.start. As a result, it contains the default value (ie 0). This causes,kernel_zimage_place()to treat the binary (contained within uImage) as position independent executable. Thus, it loads it at an incorrect address.The correct approach would be to read "uimage.ep" and set
info->zimage.start. This will ensure that the binary is loaded at the correct address.In non-statically allocated setup, a user doesn't know where the memory for dom0/domU will be allocated.
So I think this was correct to ignore the address. In fact, I am worry that...
Quoted 18 lines
Signed-off-by Ayan Kumar Halder <[email protected]>---
I uncovered this issue while loading Zephyr as a dom0less domU with Xen on
R52 FVP. Zephyr builds with static device tree. Thus, the load address is
always fixed.
xen/arch/arm/kernel.c| 2 ++1 file changed, 2 insertions(+)xen/arch/arm/kernel.c +0
-0xen/arch/arm/kernel.c static int __init kernel_uimage_probe(struct · line 226
Quoted 5 lines
xen/arch/arm/kernel.c +1
-0@@ -222,6 +222,8 @@ static int __init kernel_uimage_probe(struct kernel_info *info, if ( len > size - sizeof(uimage) ) return -EINVAL; + info->zimage.start = be32_to_cpu(uimage.ep);... this will now ended up to break anyone that may have set an address but didn't care where it should be loaded.
I also understand your use case but now, we have contradictory approaches. I am not entirely sure how we can solve it. We may have to break those users (Cc some folks that may use it). But we should figure out what is the alternative for them.
Quoted 5 lines
If we decide to break those users, then this should be documented in the
commit message and indocs/misc/arm/booting.txt(which interestingly didn't mention uImage).Quoted 5 lines
So the first issue with Zephyr is that it does not support zImage protocol for arm32.
Volodymyr added support only for Image header for arm64 Zephyr. I guess this is why Ayan, willing to boot it on Xen (arm32), decided to add u-boot header.Quoted 6 lines
If that's the only reason, then I would rather prefer if we go with zImage for a few reasons:
- The zImage protocol has at least some documentation (not perfect) of the expected state of the memory/registers when jumping to the image.
- AFAICT libxl is not (yet) supporting uImage. So this means zephyr cannot be loaded on older Xen releases (not great).I am exploring for a similar option as Volodymyr ie support zimage protocol for arm32.
But for that I need some public documentation that explains the zimage header format for arm32.
Refer
xen/arch/arm/kernel.c#define
ZIMAGE32_MAGIC_OFFSET0x24
#defineZIMAGE32_START_OFFSET0x28
#define ZIMAGE32_END_OFFSET0x2c
#define ZIMAGE32_HEADER_LEN0x30#define
ZIMAGE32_MAGIC0x016f2818Is this documented anywhere ?
I had a look at
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/arm/booting.rst
, but there is nothing that explains the header format.Quoted 11 lines
Note this doesn't mean we should not fix Xen for uImage.
Quoted 8 lines
Now, there is also a question about supporting arm64 uImage kernels. In Xen
kernel_zimage_place,
we do:
#ifdefCONFIG_ARM_64
if (info->type==DOMAIN_64BIT)
return info->mem.bank[0].start +info->zimage.text_offset; #endifQuoted 7 lines
Quoted 5 lines
So if we modify the uImage behavior for arm32, we will break consistency with arm64
(we would take uImage entry point address into account for arm32 but not for arm64).
At the moment at least they are in sync.That's a good point. It would be best if the behavior is consistent.
Currently,
kernel_zimage_place()is called for both uImage and zImage.Will it be sane if we write a different function for uImage ?
Something like this ...
static
paddr_t__initkernel_uimage_place(struct kernel_info *info){
Quoted 8 lines
/* Read and return uImage header's load address */
return
be32_to_cpu(uimage.load);}
This will be consistent across arm32 and arm64
All of these does not make a lot of sense because we are allocating memory for a domain before probing the kernel image. This means that the load/entry address for a kernel must be within the bank allocated for a domain. So the kernel already needs to know that it is running e.g. as a Xen domU, and add corresponding u-boot header to load us at e.g. GUEST_RAM0_BASE. Otherwise Xen will fail trying to copy the kernel into domain's memory. Whereas for domU it is easy to guess the memory bank, for dom0 it is not.
zImage and Image are image protocols, uImage is not. It is just a legacy u-boot header (no requirements \wrt booting,memory,registers, etc.). It can be added on top of anything (even vmlinux or a text file). So I guess this is why Xen states that it supports zImage/Image and does not mention uImage. A header is one thing, the boot requirements are another. Supporting uImage is ok but if we specify that it must be a u-boot header added on top of zImage/Image.
Quoted 5 lines
- Ayan
Quoted 3 lines
Cheers,
~Michal
Hi Michal,
Quoted 131 lines · Michal Orzel
Quoted 130 lines · Ayan Kumar Halder
Quoted 1 line · Julien Grall
Hi,
Hi,
Quoted 68 lines · Michal Orzel
Quoted 62 lines · Julien Grall
Quoted 56 lines
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Hi,
Title extra NIT: I have seen it multiple time and so far refrain to say it. Please use 'arm' rather than 'Arm'. This is for consistency in the way we name the subsystem in the title.
Quoted 9 lines · Ayan Kumar Halder
Currently,
kernel_uimage_probe()does not setinfo->zimage.start. As a result, it contains the default value (ie 0). This causes,kernel_zimage_place()to treat the binary (contained within uImage) as position independent executable. Thus, it loads it at an incorrect address.The correct approach would be to read "uimage.ep" and set
info->zimage.start. This will ensure that the binary is loaded at the correct address.In non-statically allocated setup, a user doesn't know where the memory for dom0/domU will be allocated.
So I think this was correct to ignore the address. In fact, I am worry that...
Quoted 17 lines
Signed-off-by Ayan Kumar Halder <[email protected]>---
I uncovered this issue while loading Zephyr as a dom0less domU with Xen on
R52 FVP. Zephyr builds with static device tree. Thus, the load address is
always fixed.xen/arch/arm/kernel.c | 2 ++ 1 file changed, 2 insertions(+)xen/arch/arm/kernel.c +0
-0xen/arch/arm/kernel.c static int __init kernel_uimage_probe(struct · line 226
Quoted 5 lines
xen/arch/arm/kernel.c +1
-0@@ -222,6 +222,8 @@ static int __init kernel_uimage_probe(struct kernel_info *info, if ( len > size - sizeof(uimage) ) return -EINVAL; + info->zimage.start = be32_to_cpu(uimage.ep);... this will now ended up to break anyone that may have set an address but didn't care where it should be loaded.
I also understand your use case but now, we have contradictory approaches. I am not entirely sure how we can solve it. We may have to break those users (Cc some folks that may use it). But we should figure out what is the alternative for them.
Quoted 5 lines
If we decide to break those users, then this should be documented in the
commit message and indocs/misc/arm/booting.txt(which interestingly didn't mention uImage).Quoted 5 lines
So the first issue with Zephyr is that it does not support zImage protocol for arm32.
Volodymyr added support only for Image header for arm64 Zephyr. I guess this is why Ayan, willing to boot it on Xen (arm32), decided to add u-boot header.Quoted 6 lines
If that's the only reason, then I would rather prefer if we go with zImage for a few reasons:
- The zImage protocol has at least some documentation (not perfect)
of the expected state of the memory/registers when jumping to the image.
- AFAICT libxl is not (yet) supporting uImage. So this means zephyr
cannot be loaded on older Xen releases (not great).I am exploring for a similar option as Volodymyr ie support zimage protocol for arm32.
But for that I need some public documentation that explains the zimage header format for arm32.
Refer
xen/arch/arm/kernel.c#define
ZIMAGE32_MAGIC_OFFSET0x24
#defineZIMAGE32_START_OFFSET0x28
#define ZIMAGE32_END_OFFSET0x2c
#define ZIMAGE32_HEADER_LEN0x30#define
ZIMAGE32_MAGIC0x016f2818Is this documented anywhere ?
I had a look at
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/arm/booting.rst
, but there is nothing that explains the header format.Quoted 10 lines
Note this doesn't mean we should not fix Xen for uImage.
Quoted 8 lines
Now, there is also a question about supporting arm64 uImage kernels. In Xen
kernel_zimage_place,
we do:
#ifdefCONFIG_ARM_64
if (info->type==DOMAIN_64BIT)
return info->mem.bank[0].start +info->zimage.text_offset;
#endifQuoted 6 lines
Quoted 5 lines
So if we modify the uImage behavior for arm32, we will break consistency with arm64
(we would take uImage entry point address into account for arm32 but not for arm64).
At the moment at least they are in sync.That's a good point. It would be best if the behavior is consistent.
Currently,
kernel_zimage_place()is called for both uImage and zImage.Will it be sane if we write a different function for uImage ?
Something like this ...
static
paddr_t__initkernel_uimage_place(struct kernel_info *info){
/* Read and return uImage header's load address */
return be32_to_cpu(uimage.load);}
Quoted 8 lines
Quoted 2 lines
This will be consistent across arm32 and arm64
All of these does not make a lot of sense because we are allocating memory for a domain before probing the kernel image. This means that the load/entry address for a kernel must be within the bank allocated for a domain. So the kernel already needs to know that it is running e.g. as a Xen domU, and add corresponding u-boot header to load us at e.g.
GUEST_RAM0_BASE. Otherwise Xen will fail trying to copy the kernel into domain's memory. Whereas for domU it is easy to guess the memory bank, for dom0 it is not.
I am trying to see if I understand you correctly.
Currently, the sequence is like this. Refer construct_domU()
1. kernel_probe()
2. allocate_memory()
So, we allocate memory after we probe the kernel image.
In kernel_probe(), we do read the headers in kernel_zimage32_probe() and set "info->zimage.kernel_addr" accordingly.
Later in allocate_memory(), we determine the memory for the domain.
IIUC, it is assumed that "info->zimage.kernel_addr" is within the memory range for the domain. Else, Xen may crash.
If this is correct understanding, then we should also be able to probe uImage header and set the kernel_addr (similar to kernel_zimage32_probe()).
Quoted 6 lines
zImage and Image are image protocols, uImage is not. It is just a legacy u-boot header (no requirements \wrt booting,memory,registers, etc.). It can be added on top of anything (even vmlinux or a text file). So I guess this is why Xen states that it supports zImage/Image and does not mention uImage. A header is one thing, the boot requirements are another. Supporting uImage is ok but if we specify that it must be a u-boot header added on top of zImage/Image.
Let me first confine to Arm32 only.
zephyr.bin has to be loaded at a fixed address with which it has been built.
So, we could either use zImage header (where 'start_address' can be used to specify the load address).
Or uImage (where -a is used to specify the load address).
Adding uImage header on top of zImage does not make sense.
Now for Arm64, we do need to parse the zImage header
#ifdef CONFIG_ARM_64
if ( info->type == DOMAIN_64BIT )
{
return info->mem.bank[0].start + info->zimage.text_offset; }
#endif
Again, adding uImage header on top of zImage header does not make sense as well.
Also, I believe zImage boot requirements are specific for linux kernel.
zephyr or any other RTOS may not follow the same boot requirements.
- Ayan
Quoted 7 lines
Quoted 5 lines
- Ayan
Quoted 2 lines
Cheers,
~Michal
Hi Ayan,
Quoted 25 lines · Ayan Kumar Halder
Quoted 5 lines
zImage and Image are image protocols, uImage is not. It is just a legacy u-boot header (no requirements \wrt booting,memory,registers, etc.). It can be added on top of anything (even vmlinux or a text file). So I guess this is why Xen states that it supports zImage/Image and does not mention uImage. A header is one thing, the boot requirements are another. Supporting uImage is ok but if we specify that it must be a u-boot header added on top of zImage/Image.
Let me first confine to Arm32 only.
zephyr.bin has to be loaded at a fixed address with which it has been built.
So, we could either use zImage header (where '
start_address' can be used to specify the load address).Or uImage (where -a is used to specify the load address).
Adding uImage header on top of zImage does not make sense.
Now for Arm64, we do need to parse the zImage header
#ifdef
CONFIG_ARM_64
if (info->type==DOMAIN_64BIT)
{
return info->mem.bank[0].start +info->zimage.text_offset; }
#endif
Quoted 3 lines
Again, adding uImage header on top of zImage header does not make sense as well.
Also, I believe zImage boot requirements are specific for linux kernel.
Correct. But then this is what Xen tries to adhere to when preparing the guest. So...
Quoted 1 line
zephyr or any other RTOS may not follow the same boot requirements.
... if Zephyr or any other RTOS have different requirements, then we may need to modify Xen.
Can you describe the expectation of Zephyr for the following components:
- State of the memory/cache:
* Should the image be cleaned to PoC or more?
* What about the area not part of the binary (e.g. BSS)?
* What about the rest of the memory
- State of the co-processor registers:
* Can we call the kernel with I-cache enabled?
* ...
- State of the general purpose registers:
* For instance, Linux expects a pointer to the device-tree in r0
Cheers,
Signature · 1 line
Julien Grall
Quoted 1 line · Julien Grall
Hi Ayan,
Hi Julien,
I checked with the Zephyr mantainers. Their response is provided [1].
Quoted 35 lines · Ayan Kumar Halder
Quoted 25 lines
Quoted 5 lines
zImage and Image are image protocols, uImage is not. It is just a legacy u-boot header (no requirements \wrt booting,memory,registers, etc.). It can be added on top of anything (even vmlinux or a text file). So I guess this is why Xen states that it supports zImage/Image and does not mention uImage. A header is one thing, the boot requirements are another. Supporting uImage is ok but if we specify that it must be a u-boot header added on top of zImage/Image.
Let me first confine to Arm32 only.
zephyr.bin has to be loaded at a fixed address with which it has been built.
So, we could either use zImage header (where '
start_address' can be used to specify the load address).Or uImage (where -a is used to specify the load address).
Adding uImage header on top of zImage does not make sense.
Now for Arm64, we do need to parse the zImage header
#ifdef
CONFIG_ARM_64
if (info->type==DOMAIN_64BIT)
{
return info->mem.bank[0].start +info->zimage.text_offset; }
#endifQuoted 3 lines
Again, adding uImage header on top of zImage header does not make sense as well.
Also, I believe zImage boot requirements are specific for linux kernel.
Correct. But then this is what Xen tries to adhere to when preparing the guest. So...
Quoted 1 line
zephyr or any other RTOS may not follow the same boot requirements.
Quoted 7 lines
... if Zephyr or any other RTOS have different requirements, then we may need to modify Xen.
Can you describe the expectation of Zephyr for the following components:
- State of the memory/cache:
* Should the image be cleaned to PoC or more?
* What about the area not part of the binary (e.g. BSS)?
* What about the rest of the memory
Zephyr is expected to run as a baremetal binary. When loading from Xen or uboot, the interrupts should be disabled before jumping to Zephyr.
I/D caches need to be disabled as well.
The image should be cleaned to PoC. The BSS is cleared as part of the Zephyr boot process with z_bss_zero() and data is copied with z_data_copy(), see [2] for more details.
Quoted 2 lines
- State of the co-processor registers:
* Can we call the kernel with I-cache enabled?
I cache needs to be disabled before calling kernel.
Quoted 3 lines
* ...
- State of the general purpose registers:
* For instance, Linux expects a pointer to the device-tree in r0
Zephyr does not make any assumption about the state of the GPR at boot. Also, Zephyr is built with a device tree.
- Ayan
Quoted 3 lines
Cheers,
[1] https://lists.zephyrproject.org/g/devel/message/8805
[2] https://github.com/zephyrproject-rtos/zephyr/blob/main/arch/arm64/core/prep_c.c#L54
Quoted 1 line · Ayan Kumar Halder
Hi Zephyr Arm Team,
Hello,
Quoted 2 lines
A) Zephyr is expected to run as a baremetal binary. When loading from Xen or uboot, the interrupts should be disabled.
Correct. We disable the interrupts at boot for good measure before but in theory you want the interrupts to be disabled before jumping to Zephyr.
Quoted 1 line
There is nothing mentioned about the state of caches, but I assume I/D caches will have to be disabled. Otherwise, there is a chance of reading stale instructions from I cache.
Correct. When zephyr is chain-loaded by u-boot we usually disable I/D caches from u-boot itself before jumping.
Quoted 1 line
The state of the system should be similar to how it is out of reset state (SVC or EL1).
Yes, the only caveat is that in Zephyr you have to set CONFIG_ARMV8_A_NS whether you are booting from EL3 or EL1.
Quoted 1 line
When Zephyr is loaded in memory, I am expecting the image to be cleaned to PoC. However, I am not very sure on this.
The BSS is cleared as part of the boot process with z_bss_zero() and data is copied with z_data_copy(), see [1] for more details.
Quoted 1 line
A) I assume I cache needs to be disabled.
Yup.
Quoted 1 line
A) The general purpose registers can be in an in-determinate state.
We do not make any assumption about the state of the GPR at boot. If there is an hidden dependency, that is a bug to fix.
Quoted 1 line
A) Zephyr is built with a device tree. Thus, it does not expect pre-determined values in r0, r1, etc.
Correct.
Ciao!
[1]
https://github.com/zephyrproject-rtos/zephyr/blob/main/arch/arm64/core/prep_c.c#L54
Signature · 1 line
Carlo Caione
Quoted 5 lines · Ayan Kumar Halder
Quoted 1 line · Julien Grall
Hi Ayan,
Hi Julien,
Hi,
Quoted 1 line
I checked with the Zephyr mantainers. Their response is provided [1].
Thanks for checking.
Quoted 38 lines
Quoted 38 lines · Ayan Kumar Halder
Quoted 25 lines
Quoted 5 lines
zImage and Image are image protocols, uImage is not. It is just a legacy u-boot header (no requirements \wrt booting,memory,registers, etc.). It can be added on top of anything (even vmlinux or a text file). So I guess this is why Xen states that it supports zImage/Image and does not mention uImage. A header is one thing, the boot requirements are another. Supporting uImage is ok but if we specify that it must be a u-boot header added on top of zImage/Image.
Let me first confine to Arm32 only.
zephyr.bin has to be loaded at a fixed address with which it has been built.
So, we could either use zImage header (where '
start_address' can be used to specify the load address).Or uImage (where -a is used to specify the load address).
Adding uImage header on top of zImage does not make sense.
Now for Arm64, we do need to parse the zImage header
#ifdef
CONFIG_ARM_64
if (info->type==DOMAIN_64BIT)
{
return info->mem.bank[0].start +info->zimage.text_offset; }
#endifQuoted 3 lines
Again, adding uImage header on top of zImage header does not make sense as well.
Also, I believe zImage boot requirements are specific for linux kernel.
Correct. But then this is what Xen tries to adhere to when preparing the guest. So...
Quoted 1 line
zephyr or any other RTOS may not follow the same boot requirements.
... if Zephyr or any other RTOS have different requirements, then we may need to modify Xen.
Quoted 8 lines
Quoted 5 lines
Can you describe the expectation of Zephyr for the following components:
- State of the memory/cache:
* Should the image be cleaned to PoC or more?
* What about the area not part of the binary (e.g. BSS)?
* What about the rest of the memoryZephyr is expected to run as a baremetal binary. When loading from Xen or uboot, the interrupts should be disabled before jumping to Zephyr.
I/D caches need to be disabled as well.
For both 32-bit and 64-bit Linux, the instruction can may be on or off.
At the moment, Xen is choosing to disable it. But if that's a requirement for Zephyr, then I think we should document it.
That said, from the answer on the other thread, it was not clear whether this is a strong requirement or just because U-boot is doing it.
Quoted 6 lines
The image should be cleaned to PoC. The BSS is cleared as part of the Zephyr boot process with
z_bss_zero()and data is copied withz_data_copy(), see [2] for more details.Quoted 2 lines
- State of the co-processor registers:
* Can we call the kernel with I-cache enabled?I cache needs to be disabled before calling kernel.
See above for the I-cache. However, this question was not only about the I-cache. It was just an example to what I am looking for in this category.
It might be easier if you read linux/Documentation/arm/booting.rst and let me know whether there is any expectation that don't match.
Quoted 5 lines
Quoted 3 lines
* ...
- State of the general purpose registers:
* For instance, Linux expects a pointer to the device-tree in r0Zephyr does not make any assumption about the state of the GPR at boot. Also, Zephyr is built with a device tree.
Cheers,
Signature · 1 line
Julien Grall