Skip to contentSkip to conversation

soc: qcom: mdt_loader: Drop PT_LOAD check on hash segment

PATCH In review Shawn Guo 4 participants 17 messages Aug 24, 2021 – Aug 28, 2021

Subsystems (2)
lore
Share & tools
lore

Shawn Guo

author
#0 lore raw

PATCH 1/1 soc: qcom: mdt_loader: Drop PT_LOAD check on hash segment

From: Shawn Guo <[email protected]>

It's been observed on Sony Xperia M4 Aqua phone, that wcnss firmware has the type of the second segment holding hashes just be PT_LOAD. So drop the check on phdrs[1].p_type to get it go on that phone.

---
 drivers/soc/qcom/mdt_loader.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/soc/qcom/mdt_loader.c +1 -1
index eba7f76f9d61..6034cd8992b0 100644
@@ -98,7 +98,7 @@ void *qcom_mdt_read_metadata(const struct firmware *fw, size_t *data_len)
 	if (ehdr->e_phnum < 2)
 		return ERR_PTR(-EINVAL);
 
-	if (phdrs[0].p_type == PT_LOAD || phdrs[1].p_type == PT_LOAD)
+	if (phdrs[0].p_type == PT_LOAD)
 		return ERR_PTR(-EINVAL);
 
 	if ((phdrs[1].p_flags & QCOM_MDT_TYPE_MASK) != QCOM_MDT_TYPE_HASH)
Signature · 1 line
2.17.1
#1 lore raw

Hi Shawn,

Quoted 17 lines · Shawn Guo

From: Shawn Guo <[email protected]>

It's been observed on Sony Xperia M4 Aqua phone, that wcnss firmware has the type of the second segment holding hashes just be PT_LOAD. So drop the check on phdrs[1].p_type to get it go on that phone.

---
  drivers/soc/qcom/mdt_loader.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/soc/qcom/mdt_loader.c +0 -0
index eba7f76f9d61..6034cd8992b0 100644

drivers/soc/qcom/mdt_loader.c void *qcom_mdt_read_metadata(const struct firmware *fw, size_t *data_len) · line 104

Quoted 8 lines
drivers/soc/qcom/mdt_loader.c +1 -1
@@ -98,7 +98,7 @@ void *qcom_mdt_read_metadata(const struct firmware *fw, size_t *data_len)
  	if (ehdr->e_phnum < 2)
  		return ERR_PTR(-EINVAL);
 
-	if (phdrs[0].p_type == PT_LOAD || phdrs[1].p_type == PT_LOAD)
+	if (phdrs[0].p_type == PT_LOAD)
  		return ERR_PTR(-EINVAL);
 
  	if ((phdrs[1].p_flags & QCOM_MDT_TYPE_MASK) != QCOM_MDT_TYPE_HASH)

Konrad (on the CC-list) originally came up with a similar patch to make his Sony phone boot (Xperia XZ, MSM8996). We however concluded that this solution is wrong, for the simple fact that the code below expects a PT_NULL header with data in the right place. Skipping this check most likely means that the code below will read out of bounds since the mdt file isn't large enough; this data is supposed to be read from an external file as indicated by the meaning of PT_LOAD. We built a solution for this, and fortunately CAF independently submitted a similar solution to the lists a while ago which is more thorough:

https://lore.kernel.org/linux-arm-msm/[email protected]/

It's been a while since I last compared mdt files with and without PT_LOAD, but this is the conclusion I remember coming to:

The code that packs the hash from program header 1 tightly after the ELF header in program header 0 doesn't actually need to run, since our mdt binaries already contain the signature in that place; the bytes aren't used for anything else. Simply sending the contents of the entire file as-is (similar to our downstream kernels) worked just fine (of course files beyond the size of the mdt file still have to be appended from .bXX files, and I'm not sure why this isn't checking for PT_LOAD program-header type there).

- Marijn

#3 lore raw
In reply to Marijn Suijten

Hi Marijn,

Thanks much for the information!

Quoted 38 lines · Marijn Suijten

Hi Shawn,

Quoted 16 lines · Shawn Guo

From: Shawn Guo <[email protected]>

It's been observed on Sony Xperia M4 Aqua phone, that wcnss firmware has the type of the second segment holding hashes just be PT_LOAD. So drop the check on phdrs[1].p_type to get it go on that phone.

---
  drivers/soc/qcom/mdt_loader.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/soc/qcom/mdt_loader.c +0 -0
index eba7f76f9d61..6034cd8992b0 100644

drivers/soc/qcom/mdt_loader.c void *qcom_mdt_read_metadata(const struct firmware *fw, size_t *data_len) · line 103

Quoted 7 lines
drivers/soc/qcom/mdt_loader.c +1 -1
@@ -98,7 +98,7 @@ void *qcom_mdt_read_metadata(const struct firmware *fw, size_t *data_len)
  	if (ehdr->e_phnum < 2)
  		return ERR_PTR(-EINVAL);
-	if (phdrs[0].p_type == PT_LOAD || phdrs[1].p_type == PT_LOAD)
+	if (phdrs[0].p_type == PT_LOAD)
  		return ERR_PTR(-EINVAL);
  	if ((phdrs[1].p_flags & QCOM_MDT_TYPE_MASK) != QCOM_MDT_TYPE_HASH)
 

Konrad (on the CC-list) originally came up with a similar patch to make his Sony phone boot (Xperia XZ, MSM8996). We however concluded that this solution is wrong, for the simple fact that the code below expects a PT_NULL header with data in the right place. Skipping this check most likely means that the code below will read out of bounds since the mdt file isn't large enough; this data is supposed to be read from an external file as indicated by the meaning of PT_LOAD. We built a solution for this, and fortunately CAF independently submitted a similar solution to the lists a while ago which is more thorough:

Quoted 1 line

https://lore.kernel.org/linux-arm-msm/[email protected]/

While a thorough solution is good, I do not think my patch makes anything worse or breaks anything, while fixing my issue on Sony Xperia M4 Aqua. All the current assumptions hold true for my WCNSS firmware, only except that phdrs[1] gets a PT_LOAD type.

There is a blog[1] illustrating the situation quite nicely. Again, the only thing my WCNSS firmware differs from the illustration is that hash segment gets a PT_LOAD type.

Skipping the check will not cause problem for firmwares we have seen, because hash segment is duplicated in .mdt file, and we are using that duplication instead of reading from .b01 file.

Shawn

[1] http://bits-please.blogspot.com/2016/04/exploring-qualcomms-secure-execution.html

#4 lore raw

Hi Shawn,

Quoted 43 lines · Shawn Guo

Hi Marijn,

Thanks much for the information!

Quoted 38 lines · Marijn Suijten

Hi Shawn,

Quoted 16 lines · Shawn Guo

From: Shawn Guo <[email protected]>

It's been observed on Sony Xperia M4 Aqua phone, that wcnss firmware has the type of the second segment holding hashes just be PT_LOAD. So drop the check on phdrs[1].p_type to get it go on that phone.

---
   drivers/soc/qcom/mdt_loader.c | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
drivers/soc/qcom/mdt_loader.c +0 -0
index eba7f76f9d61..6034cd8992b0 100644

drivers/soc/qcom/mdt_loader.c void *qcom_mdt_read_metadata(const struct firmware *fw, size_t *data_len) · line 103

Quoted 7 lines
drivers/soc/qcom/mdt_loader.c +1 -1
@@ -98,7 +98,7 @@ void *qcom_mdt_read_metadata(const struct firmware *fw, size_t *data_len)
   	if (ehdr->e_phnum < 2)
   		return ERR_PTR(-EINVAL);
-	if (phdrs[0].p_type == PT_LOAD || phdrs[1].p_type == PT_LOAD)
+	if (phdrs[0].p_type == PT_LOAD)
   		return ERR_PTR(-EINVAL);
   	if ((phdrs[1].p_flags & QCOM_MDT_TYPE_MASK) != QCOM_MDT_TYPE_HASH)
 

Konrad (on the CC-list) originally came up with a similar patch to make his Sony phone boot (Xperia XZ, MSM8996). We however concluded that this solution is wrong, for the simple fact that the code below expects a PT_NULL header with data in the right place. Skipping this check most likely means that the code below will read out of bounds since the mdt file isn't large enough; this data is supposed to be read from an external file as indicated by the meaning of PT_LOAD. We built a solution for this, and fortunately CAF independently submitted a similar solution to the lists a while ago which is more thorough:

Quoted 6 lines
Quoted 1 line

https://lore.kernel.org/linux-arm-msm/[email protected]/

While a thorough solution is good, I do not think my patch makes anything worse or breaks anything, while fixing my issue on Sony Xperia M4 Aqua. All the current assumptions hold true for my WCNSS firmware, only except that phdrs[1] gets a PT_LOAD type.

It's not up to me to choose between a thorough or quick solution, but this patch seems to open up the possibility for an out-of-bounds read. The assertion was placed in this function for a reason after all.

Quoted 3 lines

There is a blog[1] illustrating the situation quite nicely. Again, the only thing my WCNSS firmware differs from the illustration is that hash segment gets a PT_LOAD type.

Yes, that blog post nicely explains the format but it doesn't cover that the hash is encoded in the second program header nor that it can be copied to be packed tightly against the ELF header? Maybe that only applies to newer formats?

Quoted 3 lines

Skipping the check will not cause problem for firmwares we have seen, because hash segment is duplicated in .mdt file, and we are using that duplication instead of reading from .b01 file.

Can you share some more details about the firmware file you're using: size and the program headers as shown by readelf -l? If possible, can you also validate whether QCOM_MDT_TYPE_HASH is set in phdrs[1].p_flags & QCOM_MDT_TYPE_MASK (using something like GNU poke)?

All the files I'm able to test adhere to /* Is the header and hash already packed */: ehdr_size + hash_size == fw->size (meaning the file solely consists of a tightly packed ELF header and hash signature) but I won't be surprised if there are firmware files out in the wild with different headers or the hash missing, otherwise the else clause wouldn't exist. This else clause causes a read starting at fw->data + phdrs[1].p_offset of phdrs[1].p_filesz bytes which is almost certainly out of bounds if the program header type is PT_LOAD instead of PT_NONE, otherwise it wouldn't need to be loaded from a .bXX file in the first place.

For this quick solution to be valid I propose to reject PT_LOAD in the else clause, and otherwise validate that phdrs[1].p_offset + hash_size <= fw->size.
The aforementioned patch series in qcom_mdt_bins_are_split (and certain firmware files from Sony phones) show that it is also possible to read PT_NULL headers from external files, as long as the header references a range that is out of bounds of the mdt file.

Quoted 4 lines

Shawn

[1] http://bits-please.blogspot.com/2016/04/exploring-qualcomms-secure-execution.html

- Marijn

#5 lore raw
Quoted 48 lines · Marijn Suijten

Hi Shawn,

Quoted 45 lines · Shawn Guo

Hi Marijn,

Thanks much for the information!

Quoted 39 lines · Marijn Suijten

Hi Shawn,

Quoted 16 lines · Shawn Guo

From: Shawn Guo <[email protected]>

It's been observed on Sony Xperia M4 Aqua phone, that wcnss firmware has the type of the second segment holding hashes just be PT_LOAD. So drop the check on phdrs[1].p_type to get it go on that phone.

---
   drivers/soc/qcom/mdt_loader.c | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
drivers/soc/qcom/mdt_loader.c +0 -0
index eba7f76f9d61..6034cd8992b0 100644

drivers/soc/qcom/mdt_loader.c void *qcom_mdt_read_metadata(const struct firmware *fw, size_t *data_len) · line 103

Quoted 7 lines
drivers/soc/qcom/mdt_loader.c +1 -1
@@ -98,7 +98,7 @@ void *qcom_mdt_read_metadata(const struct firmware *fw, size_t *data_len)
   	if (ehdr->e_phnum < 2)
   		return ERR_PTR(-EINVAL);
-	if (phdrs[0].p_type == PT_LOAD || phdrs[1].p_type == PT_LOAD)
+	if (phdrs[0].p_type == PT_LOAD)
   		return ERR_PTR(-EINVAL);
   	if ((phdrs[1].p_flags & QCOM_MDT_TYPE_MASK) != QCOM_MDT_TYPE_HASH)
 

Konrad (on the CC-list) originally came up with a similar patch to make his Sony phone boot (Xperia XZ, MSM8996). We however concluded that this solution is wrong, for the simple fact that the code below expects a PT_NULL header with data in the right place. Skipping this check most likely means that the code below will read out of bounds since the mdt file isn't large enough; this data is supposed to be read from an external file as indicated by the meaning of PT_LOAD. We built a solution for this, and fortunately CAF independently submitted a similar solution to the lists a while ago which is more thorough:

https://lore.kernel.org/linux-arm-msm/[email protected]/

Quoted 7 lines
Quoted 4 lines

While a thorough solution is good, I do not think my patch makes anything worse or breaks anything, while fixing my issue on Sony Xperia M4 Aqua. All the current assumptions hold true for my WCNSS firmware, only except that phdrs[1] gets a PT_LOAD type.

It's not up to me to choose between a thorough or quick solution,

To be clear, Siddharth's series doesn't resolve my problem, as the assumption that hash segment type cannot be PT_LOAD is still there. I have to add the following change on top to fix my problem.

@@ -126,8 +126,7 @@ void *qcom_mdt_read_metadata(struct device *dev, const struct firmware *fw, cons

                return ERR_PTR(-EINVAL);
 
        for (hash_index = 1; hash_index < ehdr->e_phnum; hash_index++) {
-               if (phdrs[hash_index].p_type != PT_LOAD &&
-                  (phdrs[hash_index].p_flags & QCOM_MDT_TYPE_MASK) == QCOM_MDT_TYPE_HASH)
+               if ((phdrs[hash_index].p_flags & QCOM_MDT_TYPE_MASK) == QCOM_MDT_TYPE_HASH)
                        break;
        }
        if (hash_index >= ehdr->e_phnum)

That said, my patch is merely to break the assumption that hash segment type cannot be PT_LOAD, and it's really orthogonal to Siddharth's series.

Quoted 3 lines

but this
patch seems to open up the possibility for an out-of-bounds read. The assertion was placed in this function for a reason after all.

I would much appreciate it if someone helps to elaborate the reason.

Quoted 10 lines
Quoted 4 lines

There is a blog[1] illustrating the situation quite nicely. Again, the only thing my WCNSS firmware differs from the illustration is that hash segment gets a PT_LOAD type.

Yes, that blog post nicely explains the format but it doesn't cover that the hash is encoded in the second program header nor that it can be copied to be packed tightly against the ELF header? Maybe that only applies to newer formats?

Hmm, it does cover the things. It's been illustrated that .mdt is simply a concatenating of .b00 and .b01. The .b00 includes ELF header and program headers, while .b01 is just hash segment.

$ cat wcnss.b00 wcnss.b01 > wcnss.b00_b01
$ cmp wcnss.b00_b01 wcnss.mdt
$

That said, .mdt = ELF header + program headers + hash

Quoted 8 lines
Quoted 4 lines

Skipping the check will not cause problem for firmwares we have seen, because hash segment is duplicated in .mdt file, and we are using that duplication instead of reading from .b01 file.

Can you share some more details about the firmware file you're using: size and the program headers as shown by readelf -l?

$ readelf -l wcnss.mdt

Elf file type is EXEC (Executable file) Entry point 0x8b6018d4
There are 12 program headers, starting at offset 52

Program Headers:

  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  NULL           0x000000 0x00000000 0x00000000 0x001b4 0x00000     0
  LOAD           0x001000 0x8bbe0000 0x8bbe0000 0x00c58 0x02000     0x1000
  LOAD           0x003000 0x8b600000 0x8b600000 0x03308 0x03438 RWE 0x100000
  LOAD           0x00afcc 0x8b604000 0x8b604000 0x00000 0x08000 RW  0x4000
  LOAD           0x00afcc 0x8b60c000 0x8b60c000 0x0f000 0x0f000 RW  0x4
  LOAD           0x019fcc 0x8b61b000 0x8b61b000 0x00000 0x0e000 RW  0x4
  LOAD           0x019fcc 0x8b629000 0x8b629000 0x32eb04 0x4c2b10 RWE 0x80
  LOAD           0x348ad0 0x8baebb80 0x8baebb80 0x00000 0x37cf8 RW  0x8
  LOAD           0x348ad0 0x8baebb80 0x8baebb80 0x00000 0x21c44 RW  0x4
  LOAD           0x348ad0 0x8bb30000 0x8bb30000 0x00034 0x001a8 RW  0x8
  LOAD           0x349fcc 0x8bb31000 0x8bb31000 0xa0000 0xa0000 RW  0x1000
  LOAD           0x3e9fcc 0x8bbd1000 0x8bbd1000 0x0ac3c 0x0ee00 RWE 0x1000
Quoted 3 lines

If possible, can you also
validate whether QCOM_MDT_TYPE_HASH is set in phdrs[1].p_flags & QCOM_MDT_TYPE_MASK (using something like GNU poke)?

It is set, otherwise the QCOM_MDT_TYPE_HASH check in qcom_mdt_read_metadata() will just fail. To be clear, everything works fine for me, as long as I drop the check of (phdrs[1].p_type == PT_LOAD).

Quoted 4 lines

All the files I'm able to test adhere to /* Is the header and hash already packed */: ehdr_size + hash_size == fw->size (meaning the file solely consists of a tightly packed ELF header and hash signature)

Yeah, my wcnss firmware is same here. Actually, all split firmwares I have seen are this case. But bear it in mind there are non-split firmware like a single .mbn file. My understanding is that condition (ehdr_size + hash_size == fw->size) is being used to check whether it's a split firmware or non-split one.

Quoted 7 lines

but I won't be
surprised if there are firmware files out in the wild with different headers or the hash missing, otherwise the else clause wouldn't exist. This else clause causes a read starting at fw->data + phdrs[1].p_offset of phdrs[1].p_filesz bytes which is almost certainly out of bounds if the program header type is PT_LOAD instead of PT_NONE, otherwise it wouldn't need to be loaded from a .bXX file in the first place.

So the else clause is there to handle non-split firmware, which has everything within fw->size.

Quoted 4 lines

For this quick solution to be valid I propose to reject PT_LOAD in the else clause, and otherwise validate that phdrs[1].p_offset + hash_size <= fw->size.

I'm not sure what you propose here are required for non-split firmware.

Quoted 4 lines

The aforementioned patch series in qcom_mdt_bins_are_split (and certain firmware files from Sony phones) show that it is also possible to read PT_NULL headers from external files, as long as the header references a range that is out of bounds of the mdt file.

It's been shown that hashes can be read from .mdt or hash segment, and independently the hash segment type could be PT_NULL or PT_LOAD. With Siddharth's patch, instead of using the hash duplication in .mdt, hash will be read from hash segment. But again, to resolve my problem, the assertion that hash segment type cannot be PT_LOAD has to be dropped. And that's the only thing my patch is doing.

Shawn

#6 lore raw

Hi Shawn,

Quoted 8 lines · Shawn Guo
Quoted 2 lines

[..]
It's not up to me to choose between a thorough or quick solution,

To be clear, Siddharth's series doesn't resolve my problem, as the assumption that hash segment type cannot be PT_LOAD is still there. I have to add the following change on top to fix my problem.

Review void *qcom_mdt_read_metadata(struct device *dev, const struct firmware *fw, cons · line 132

Quoted 9 lines
(quoted diff) +1 -2
@@ -126,8 +126,7 @@ void *qcom_mdt_read_metadata(struct device *dev, const struct firmware *fw, cons
                 return ERR_PTR(-EINVAL);
 
         for (hash_index = 1; hash_index < ehdr->e_phnum; hash_index++) {
-               if (phdrs[hash_index].p_type != PT_LOAD &&
-                  (phdrs[hash_index].p_flags & QCOM_MDT_TYPE_MASK) == QCOM_MDT_TYPE_HASH)
+               if ((phdrs[hash_index].p_flags & QCOM_MDT_TYPE_MASK) == QCOM_MDT_TYPE_HASH)
                         break;
         }
         if (hash_index >= ehdr->e_phnum)

Patch 3/3 allows the hash segment to be read from an external file and should indeed get rid of this comparison, as external file loading is possible with PT_NULL and required with PT_LOAD. I'd go as far as moving the check into the if, next to qcom_mdt_bins_are_split:

if (phdrs[hash_index].p_type == PT_LOAD || qcom_mdt_bins_are_split(fw))

Unfortunately it seems this patchset lost optimization for the packed ehdr_size + hash_size == fw->size case, where the hash segment is already available in the loaded mbt.

Quoted 3 lines

That said, my patch is merely to break the assumption that hash segment type cannot be PT_LOAD, and it's really orthogonal to Siddharth's series.

It is fine - correct even - to break that assumption, but it should go with extra validation that we are dealing with packed mdt instead.

Quoted 5 lines
Quoted 3 lines

but this
patch seems to open up the possibility for an out-of-bounds read. The assertion was placed in this function for a reason after all.

I would much appreciate it if someone helps to elaborate the reason.

PT_LOAD specifies that the segment is to be loaded externally. The fact that our .mdt file is a tight pack of b00 + b01 is mere convenience, but is it also a given for the future? Can we rely on this assumption to never change?

Quoted 14 lines
Quoted 9 lines
Quoted 3 lines

There is a blog[1] illustrating the situation quite nicely. Again, the only thing my WCNSS firmware differs from the illustration is that hash segment gets a PT_LOAD type.

Yes, that blog post nicely explains the format but it doesn't cover that the hash is encoded in the second program header nor that it can be copied to be packed tightly against the ELF header? Maybe that only applies to newer formats?

Hmm, it does cover the things. It's been illustrated that .mdt is simply a concatenating of .b00 and .b01. The .b00 includes ELF header and program headers, while .b01 is just hash segment.

Quoted 5 lines
$ cat wcnss.b00 wcnss.b01 > wcnss.b00_b01
$ cmp wcnss.b00_b01 wcnss.mdt
$

That said, .mdt = ELF header + program headers + hash

Ack, there's one picture halfway demonstrating the ehdr_size + hash_size == fw->size case.

Quoted 29 lines
Quoted 7 lines
Quoted 3 lines

Skipping the check will not cause problem for firmwares we have seen, because hash segment is duplicated in .mdt file, and we are using that duplication instead of reading from .b01 file.

Can you share some more details about the firmware file you're using: size and the program headers as shown by readelf -l?

$ readelf -l wcnss.mdt

Elf file type is EXEC (Executable file) Entry point 0x8b6018d4
There are 12 program headers, starting at offset 52

Program Headers:

   Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
   NULL           0x000000 0x00000000 0x00000000 0x001b4 0x00000     0
   LOAD           0x001000 0x8bbe0000 0x8bbe0000 0x00c58 0x02000     0x1000
   LOAD           0x003000 0x8b600000 0x8b600000 0x03308 0x03438 RWE 0x100000
   LOAD           0x00afcc 0x8b604000 0x8b604000 0x00000 0x08000 RW  0x4000
   LOAD           0x00afcc 0x8b60c000 0x8b60c000 0x0f000 0x0f000 RW  0x4
   LOAD           0x019fcc 0x8b61b000 0x8b61b000 0x00000 0x0e000 RW  0x4
   LOAD           0x019fcc 0x8b629000 0x8b629000 0x32eb04 0x4c2b10 RWE 0x80
   LOAD           0x348ad0 0x8baebb80 0x8baebb80 0x00000 0x37cf8 RW  0x8
   LOAD           0x348ad0 0x8baebb80 0x8baebb80 0x00000 0x21c44 RW  0x4
   LOAD           0x348ad0 0x8bb30000 0x8bb30000 0x00034 0x001a8 RW  0x8
   LOAD           0x349fcc 0x8bb31000 0x8bb31000 0xa0000 0xa0000 RW  0x1000
   LOAD           0x3e9fcc 0x8bbd1000 0x8bbd1000 0x0ac3c 0x0ee00 RWE 0x1000
Quoted 7 lines
Quoted 3 lines

If possible, can you also
validate whether QCOM_MDT_TYPE_HASH is set in phdrs[1].p_flags & QCOM_MDT_TYPE_MASK (using something like GNU poke)?

It is set, otherwise the QCOM_MDT_TYPE_HASH check in qcom_mdt_read_metadata() will just fail. To be clear, everything works fine for me, as long as I drop the check of (phdrs[1].p_type == PT_LOAD).

Thanks, this all checks out and is similar to what I'm seeing here. It all comes down to the mdt packing b00 and b01 tightly together already.

Quoted 9 lines
Quoted 3 lines

All the files I'm able to test adhere to /* Is the header and hash already packed */: ehdr_size + hash_size == fw->size (meaning the file solely consists of a tightly packed ELF header and hash signature)

Yeah, my wcnss firmware is same here. Actually, all split firmwares I have seen are this case. But bear it in mind there are non-split firmware like a single .mbn file. My understanding is that condition (ehdr_size + hash_size == fw->size) is being used to check whether it's a split firmware or non-split one.

Is it unreasonable to assume that more configurations besides split and non-split firmware might occur in the future (or are already out in the wild)? These program headers allow for a variety of configurations which we should - in my opinion - parse and handle in a generic manner.
ehdr_size + hash_size == fw->size is convenient to have, but not
something we should rely on.

Quoted 10 lines
Quoted 7 lines

but I won't be
surprised if there are firmware files out in the wild with different headers or the hash missing, otherwise the else clause wouldn't exist. This else clause causes a read starting at fw->data + phdrs[1].p_offset of phdrs[1].p_filesz bytes which is almost certainly out of bounds if the program header type is PT_LOAD instead of PT_NONE, otherwise it wouldn't need to be loaded from a .bXX file in the first place.

So the else clause is there to handle non-split firmware, which has everything within fw->size.

Is it too much to ask for extra validation here, instead of always assuming either "split" or "non-split" firmware? As mentioned before these program headers allow for a variety of configurations, which is confirmed by Siddharth's first patch looking for QCOM_MDT_TYPE_HASH in all headers instead of assuming it to reside in the second.

Quoted 6 lines
Quoted 4 lines

For this quick solution to be valid I propose to reject PT_LOAD in the else clause, and otherwise validate that phdrs[1].p_offset + hash_size <= fw->size.

I'm not sure what you propose here are required for non-split firmware.

Would it help if I sent a patch based on yours, with this extra validation and comments + commit message explaining the cases?

Alternatively we can try re-spinning the patches from Siddharth while taking review comments, the possible .mdt == .b00 + . b01 packing, and my suggestion to handle the headers generically into account.

Quoted 11 lines
Quoted 4 lines

The aforementioned patch series in qcom_mdt_bins_are_split (and certain firmware files from Sony phones) show that it is also possible to read PT_NULL headers from external files, as long as the header references a range that is out of bounds of the mdt file.

It's been shown that hashes can be read from .mdt or hash segment, and independently the hash segment type could be PT_NULL or PT_LOAD. With Siddharth's patch, instead of using the hash duplication in .mdt, hash will be read from hash segment. But again, to resolve my problem, the assertion that hash segment type cannot be PT_LOAD has to be dropped. And that's the only thing my patch is doing.

Correct. If I were to respin Siddharth's patchset, I'd write qcom_mdt_read_metadata as follows:

1. Find the header that contains QCOM_MDT_TYPE_HASH (allowing PT_NONE and PT_LOAD);
2. If ehdr_size + hash_size == fw->size, this is split firmware and the mdt file can be used as-is for metadata;
3. If the type is PT_LOAD, _or_ if the type is PT_NULL but p_offset + p_filesz does not fit inside the .mdt, this is (a variant of) split-firmware, and the hash needs to be loaded from an external file and appended to the ELF header.
4. If none of the above, this is a non-split firmware file. Concatenate the ELF and hash headers by reading them directly from the firmware.

- Marijn

#7 lore raw
Quoted 44 lines · Marijn Suijten

Hi Shawn,

Quoted 8 lines · Shawn Guo
Quoted 2 lines

[..]
It's not up to me to choose between a thorough or quick solution,

To be clear, Siddharth's series doesn't resolve my problem, as the assumption that hash segment type cannot be PT_LOAD is still there. I have to add the following change on top to fix my problem.

Review void *qcom_mdt_read_metadata(struct device *dev, const struct firmware *fw, cons · line 131

Quoted 8 lines
(quoted diff) +1 -2
@@ -126,8 +126,7 @@ void *qcom_mdt_read_metadata(struct device *dev, const struct firmware *fw, cons
                 return ERR_PTR(-EINVAL);
         for (hash_index = 1; hash_index < ehdr->e_phnum; hash_index++) {
-               if (phdrs[hash_index].p_type != PT_LOAD &&
-                  (phdrs[hash_index].p_flags & QCOM_MDT_TYPE_MASK) == QCOM_MDT_TYPE_HASH)
+               if ((phdrs[hash_index].p_flags & QCOM_MDT_TYPE_MASK) == QCOM_MDT_TYPE_HASH)
                         break;
         }
         if (hash_index >= ehdr->e_phnum)

Patch 3/3 allows the hash segment to be read from an external file and should indeed get rid of this comparison, as external file loading is possible with PT_NULL and required with PT_LOAD. I'd go as far as moving the check into the if, next to qcom_mdt_bins_are_split:

if (phdrs[hash_index].p_type == PT_LOAD || qcom_mdt_bins_are_split(fw))

Unfortunately it seems this patchset lost optimization for the packed ehdr_size + hash_size == fw->size case, where the hash segment is already available in the loaded mbt.

Quoted 3 lines

That said, my patch is merely to break the assumption that hash segment type cannot be PT_LOAD, and it's really orthogonal to Siddharth's series.

It is fine - correct even - to break that assumption, but it should go with extra validation that we are dealing with packed mdt instead.

Quoted 4 lines
Quoted 3 lines

but this
patch seems to open up the possibility for an out-of-bounds read. The assertion was placed in this function for a reason after all.

Quoted 7 lines
Quoted 1 line

I would much appreciate it if someone helps to elaborate the reason.

PT_LOAD specifies that the segment is to be loaded externally. The fact that our .mdt file is a tight pack of b00 + b01 is mere convenience, but is it also a given for the future? Can we rely on this assumption to never change?

My patch is trying to fix an existing issue, not anything for the future.

Quoted 110 lines
Quoted 15 lines
Quoted 10 lines
Quoted 4 lines

There is a blog[1] illustrating the situation quite nicely. Again, the only thing my WCNSS firmware differs from the illustration is that hash segment gets a PT_LOAD type.

Yes, that blog post nicely explains the format but it doesn't cover that the hash is encoded in the second program header nor that it can be copied to be packed tightly against the ELF header? Maybe that only applies to newer formats?

Hmm, it does cover the things. It's been illustrated that .mdt is simply a concatenating of .b00 and .b01. The .b00 includes ELF header and program headers, while .b01 is just hash segment.

Quoted 5 lines
$ cat wcnss.b00 wcnss.b01 > wcnss.b00_b01
$ cmp wcnss.b00_b01 wcnss.mdt
$

That said, .mdt = ELF header + program headers + hash

Ack, there's one picture halfway demonstrating the ehdr_size + hash_size == fw->size case.

Quoted 29 lines
Quoted 7 lines
Quoted 3 lines

Skipping the check will not cause problem for firmwares we have seen, because hash segment is duplicated in .mdt file, and we are using that duplication instead of reading from .b01 file.

Can you share some more details about the firmware file you're using: size and the program headers as shown by readelf -l?

$ readelf -l wcnss.mdt

Elf file type is EXEC (Executable file) Entry point 0x8b6018d4
There are 12 program headers, starting at offset 52

Program Headers:

   Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
   NULL           0x000000 0x00000000 0x00000000 0x001b4 0x00000     0
   LOAD           0x001000 0x8bbe0000 0x8bbe0000 0x00c58 0x02000     0x1000
   LOAD           0x003000 0x8b600000 0x8b600000 0x03308 0x03438 RWE 0x100000
   LOAD           0x00afcc 0x8b604000 0x8b604000 0x00000 0x08000 RW  0x4000
   LOAD           0x00afcc 0x8b60c000 0x8b60c000 0x0f000 0x0f000 RW  0x4
   LOAD           0x019fcc 0x8b61b000 0x8b61b000 0x00000 0x0e000 RW  0x4
   LOAD           0x019fcc 0x8b629000 0x8b629000 0x32eb04 0x4c2b10 RWE 0x80
   LOAD           0x348ad0 0x8baebb80 0x8baebb80 0x00000 0x37cf8 RW  0x8
   LOAD           0x348ad0 0x8baebb80 0x8baebb80 0x00000 0x21c44 RW  0x4
   LOAD           0x348ad0 0x8bb30000 0x8bb30000 0x00034 0x001a8 RW  0x8
   LOAD           0x349fcc 0x8bb31000 0x8bb31000 0xa0000 0xa0000 RW  0x1000
   LOAD           0x3e9fcc 0x8bbd1000 0x8bbd1000 0x0ac3c 0x0ee00 RWE 0x1000
Quoted 7 lines
Quoted 3 lines

If possible, can you also
validate whether QCOM_MDT_TYPE_HASH is set in phdrs[1].p_flags & QCOM_MDT_TYPE_MASK (using something like GNU poke)?

It is set, otherwise the QCOM_MDT_TYPE_HASH check in qcom_mdt_read_metadata() will just fail. To be clear, everything works fine for me, as long as I drop the check of (phdrs[1].p_type == PT_LOAD).

Thanks, this all checks out and is similar to what I'm seeing here. It all comes down to the mdt packing b00 and b01 tightly together already.

Quoted 9 lines
Quoted 3 lines

All the files I'm able to test adhere to /* Is the header and hash already packed */: ehdr_size + hash_size == fw->size (meaning the file solely consists of a tightly packed ELF header and hash signature)

Yeah, my wcnss firmware is same here. Actually, all split firmwares I have seen are this case. But bear it in mind there are non-split firmware like a single .mbn file. My understanding is that condition (ehdr_size + hash_size == fw->size) is being used to check whether it's a split firmware or non-split one.

Is it unreasonable to assume that more configurations besides split and non-split firmware might occur in the future (or are already out in the wild)? These program headers allow for a variety of configurations which we should - in my opinion - parse and handle in a generic manner. ehdr_size + hash_size == fw->size is convenient to have, but not something we should rely on.

Quoted 10 lines
Quoted 7 lines

but I won't be
surprised if there are firmware files out in the wild with different headers or the hash missing, otherwise the else clause wouldn't exist. This else clause causes a read starting at fw->data + phdrs[1].p_offset of phdrs[1].p_filesz bytes which is almost certainly out of bounds if the program header type is PT_LOAD instead of PT_NONE, otherwise it wouldn't need to be loaded from a .bXX file in the first place.

So the else clause is there to handle non-split firmware, which has everything within fw->size.

Is it too much to ask for extra validation here, instead of always assuming either "split" or "non-split" firmware? As mentioned before these program headers allow for a variety of configurations, which is confirmed by Siddharth's first patch looking for QCOM_MDT_TYPE_HASH in all headers instead of assuming it to reside in the second.

Quoted 6 lines
Quoted 4 lines

For this quick solution to be valid I propose to reject PT_LOAD in the else clause, and otherwise validate that phdrs[1].p_offset + hash_size <= fw->size.

I'm not sure what you propose here are required for non-split firmware.

Quoted 6 lines

Would it help if I sent a patch based on yours, with this extra validation and comments + commit message explaining the cases?

Alternatively we can try re-spinning the patches from Siddharth while taking review comments, the possible .mdt == .b00 + . b01 packing, and my suggestion to handle the headers generically into account.

I would suggest this path. It's been 8 months, so Siddharth probably lost the interest here. It's reasonable that someone picks up the work.

Quoted 16 lines
Quoted 6 lines
Quoted 5 lines

The aforementioned patch series in qcom_mdt_bins_are_split (and certain firmware files from Sony phones) show that it is also possible to read PT_NULL headers from external files, as long as the header references a range that is out of bounds of the mdt file.

Quoted 6 lines

It's been shown that hashes can be read from .mdt or hash segment, and independently the hash segment type could be PT_NULL or PT_LOAD. With Siddharth's patch, instead of using the hash duplication in .mdt, hash will be read from hash segment. But again, to resolve my problem, the assertion that hash segment type cannot be PT_LOAD has to be dropped. And that's the only thing my patch is doing.

Correct. If I were to respin Siddharth's patchset, I'd write qcom_mdt_read_metadata as follows:

Quoted 10 lines

1. Find the header that contains QCOM_MDT_TYPE_HASH (allowing PT_NONE and PT_LOAD);
2. If ehdr_size + hash_size == fw->size, this is split firmware and the mdt file can be used as-is for metadata;
3. If the type is PT_LOAD, _or_ if the type is PT_NULL but p_offset + p_filesz does not fit inside the .mdt, this is (a variant of) split-firmware, and the hash needs to be loaded from an external file and appended to the ELF header.
4. If none of the above, this is a non-split firmware file. Concatenate the ELF and hash headers by reading them directly from the firmware.

Looks good to me. I will be happy to review patches if you pick up the work.

Shawn

#8 lore raw

Hi Shawn,

Quoted 8 lines · Shawn Guo
Quoted 5 lines

[..]
PT_LOAD specifies that the segment is to be loaded externally. The fact that our .mdt file is a tight pack of b00 + b01 is mere convenience, but is it also a given for the future? Can we rely on this assumption to never change?

My patch is trying to fix an existing issue, not anything for the future.

We both agree that the PT_LOAD assertion here is too strict, but removing it altogether makes the function too lenient and allows for possible bugs. To solve your issue in the simple case I have already suggested to add an extra bounds check.

Quoted 35 lines
Quoted 4 lines

[..]
Alternatively we can try re-spinning the patches from Siddharth while taking review comments, the possible .mdt == .b00 + . b01 packing, and my suggestion to handle the headers generically into account.

I would suggest this path. It's been 8 months, so Siddharth probably lost the interest here. It's reasonable that someone picks up the work.

Quoted 20 lines
Quoted 6 lines
Quoted 5 lines

The aforementioned patch series in qcom_mdt_bins_are_split (and certain firmware files from Sony phones) show that it is also possible to read PT_NULL headers from external files, as long as the header references a range that is out of bounds of the mdt file.

Quoted 6 lines

It's been shown that hashes can be read from .mdt or hash segment, and independently the hash segment type could be PT_NULL or PT_LOAD. With Siddharth's patch, instead of using the hash duplication in .mdt, hash will be read from hash segment. But again, to resolve my problem, the assertion that hash segment type cannot be PT_LOAD has to be dropped. And that's the only thing my patch is doing.

Correct. If I were to respin Siddharth's patchset, I'd write qcom_mdt_read_metadata as follows:

1. Find the header that contains QCOM_MDT_TYPE_HASH (allowing PT_NONE

    and PT_LOAD);
2. If `ehdr_size + hash_size == fw->size`, this is split firmware and
    the mdt file can be used as-is for metadata;
Quoted 6 lines

3. If the type is PT_LOAD, _or_ if the type is PT_NULL but p_offset + p_filesz does not fit inside the .mdt, this is (a variant of) split-firmware, and the hash needs to be loaded from an external file and appended to the ELF header.
4. If none of the above, this is a non-split firmware file. Concatenate the ELF and hash headers by reading them directly from the firmware.

Quoted 2 lines

Looks good to me. I will be happy to review patches if you pick up the work.

I'll try to get this going over the weekend or next week. I don't think I can salvage anything from Siddharth's patchset and will likely start from scratch as I'm not confident "just detecting split firmware" is enough, but favour generic handling of the program headers as described above instead (for both the hash and firmware loading itself).

Will keep you posted!

- Marijn

#9 lore raw
Quoted 9 lines · Marijn Suijten

Hi Shawn,

Quoted 6 lines · Shawn Guo
Quoted 5 lines

[..]
PT_LOAD specifies that the segment is to be loaded externally. The fact that our .mdt file is a tight pack of b00 + b01 is mere convenience, but is it also a given for the future? Can we rely on this assumption to never change?

Quoted 8 lines
Quoted 2 lines

My patch is trying to fix an existing issue, not anything for the future.

We both agree that the PT_LOAD assertion here is too strict, but removing it altogether makes the function too lenient and allows for possible bugs. To solve your issue in the simple case I have already suggested to add an extra bounds check.

So you proposed to reject PT_LOAD in the else clause, which right now handles .mbn case, are you sure hash segment in .mbn is not going to be PT_LOAD?

Shawn

#10 lore raw

Hi Shawn,

Quoted 4 lines · Shawn Guo

[..]

So you proposed to reject PT_LOAD in the else clause, which right now handles .mbn case

Yes, I propose to reject PT_LOAD in the else-case, and additionally reject cases where p_offset+p_filesz > sw->size since PT_NULL can also cause external file loads (meaning split-firmware). This is what Siddharth's patchset - or my respin of it - is going to implement.

Quoted 2 lines

are you sure hash segment in .mbn is not going to be PT_LOAD?

PT_LOAD unambiguously indicates a program header that ought to be loaded from an external file. Any mbn file (non-split firmware) without split files that set PT_LOAD are misusing this program header type field. I have no way to validate whether such mbns are in circulation.

Of note, I have never referenced the definition of the program header types yet. Please see [1]:

PT_LOAD (1)
Indicates that this program header describes a segment to be loaded from the file.

Let me know if you're planning to send a v2 of this patch with aforementioned improvements, then I'll adjust my plans to respin Siddharth's patchset accordingly.

- Marijn

[1]: https://ftp.gnu.org/old-gnu/Manuals/ld-2.9.1/html_node/ld_23.html

#11 lore raw
Quoted 14 lines · Marijn Suijten

Hi Shawn,

Quoted 4 lines · Shawn Guo

[..]

So you proposed to reject PT_LOAD in the else clause, which right now handles .mbn case

Yes, I propose to reject PT_LOAD in the else-case, and additionally reject cases where p_offset+p_filesz > sw->size since PT_NULL can also cause external file loads (meaning split-firmware). This is what Siddharth's patchset - or my respin of it - is going to implement.

Quoted 8 lines
Quoted 2 lines

are you sure hash segment in .mbn is not going to be PT_LOAD?

PT_LOAD unambiguously indicates a program header that ought to be loaded from an external file. Any mbn file (non-split firmware) without split files that set PT_LOAD are misusing this program header type field. I have no way to validate whether such mbns are in circulation.

Following your take on PT_LOAD, I assume that no PT_LOAD segment should be found in .mbn file, correct? Here are two .mbn I found in circulation. Both have PT_LOAD type for a few segments.

$ readelf -l venus.mbn 

Elf file type is EXEC (Executable file) Entry point 0xf500000
There are 5 program headers, starting at offset 52

Program Headers:

  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  NULL           0x000000 0x00000000 0x00000000 0x000d4 0x00000     0
  LOAD           0x001000 0x0fa00000 0x0fa00000 0x00b88 0x02000     0x1000
  LOAD           0x003000 0x00000000 0x0f500000 0xeecd0 0xeecd0 R E 0x100000
  LOAD           0x0f1cd0 0x000ef000 0x0f5ef000 0x015c0 0x405000 RW  0x4000
  LOAD           0x0f3290 0x004ff000 0x0f9ff000 0x00020 0x00020 RW  0x4
$ readelf -l mba.mbn 

Elf file type is EXEC (Executable file) Entry point 0x4417000
There are 5 program headers, starting at offset 52

Program Headers:

  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  NULL           0x000000 0x00000000 0x00000000 0x000d4 0x00000     0
  NULL           0x001000 0x8ea4a000 0x8ea4a000 0x019c8 0x02000     0x1000
  LOAD           0x003000 0x04417000 0x8ea00000 0x350d0 0x3bbc8 RWE 0x1000
  LOAD           0x039000 0x04460000 0x8ea49000 0x00380 0x00380 RW  0x1000
  GNU_STACK      0x002000 0x00000000 0x00000000 0x00000 0x00000 RWE 0x4

Or you think these are all misusing of PT_LOAD? Sorry, I hardly believe your understanding on PT_LOAD matches the reality. Instead, I'm inclined to agree with Bjorn's comment.

Quote:

"I would expect that PT_LOAD denotes that the segment should be loaded into the final firmware region and that the hash segment would be PT_NULL regardless of being part of the .mdt, single .mbn or a separate .bNN segment."

The only part that doesn't hold is "the hash segment would be PT_NULL". But the point is that PT_LOAD doesn't mean the segment should be loaded externally (from .bNN file).

Quoted 11 lines

Of note, I have never referenced the definition of the program header types yet. Please see [1]:

PT_LOAD (1)
Indicates that this program header describes a segment to be loaded from the file.

Let me know if you're planning to send a v2 of this patch with aforementioned improvements, then I'll adjust my plans to respin Siddharth's patchset accordingly.

I will send v2. However there will be no code change but just commit log update based on all these discussion. Thanks!

Shawn

Quoted 1 line

[1]: https://ftp.gnu.org/old-gnu/Manuals/ld-2.9.1/html_node/ld_23.html

#12 lore raw

Hi Shawn,

Quoted 5 lines · Shawn Guo

[..]

Or you think these are all misusing of PT_LOAD? Sorry, I hardly believe your understanding on PT_LOAD matches the reality. Instead, I'm inclined to agree with Bjorn's comment.

Agreed, PT_LOAD does not imply _external_ loading at all, merely whether it has to be copied to the final memory region. That's my misunderstanding, even after quoting the documentation.

This external loading is merely a qcom implementation detail AFAIK, and solely detected by the source boundaries (p_offset - p_offset+p_filesz) residing outside the .mdt file.

Quoted 10 lines

Quote:

"I would expect that PT_LOAD denotes that the segment should be loaded into the final firmware region and that the hash segment would be PT_NULL regardless of being part of the .mdt, single .mbn or a separate .bNN segment."

The only part that doesn't hold is "the hash segment would be PT_NULL". But the point is that PT_LOAD doesn't mean the segment should be loaded externally (from .bNN file).

Yes, this is strange. Why would the .mdt request the hash segment to be loaded to firmware memory when it's filtered out anyway? Are we supposed to filter it out?

Quoted 4 lines

[..]

I will send v2. However there will be no code change but just commit log update based on all these discussion. Thanks!

Sounds good - glad we could have this discussion to get to the bottom of this and conclude that removing this check is the right approach without any side-effects, including a detailed justification in the commit-message.

- Marijn

#13 lore raw
In reply to Shawn Guo
Quoted 2 lines · Shawn Guo
Quoted 1 line · Marijn Suijten

On 8/27/21 8:24 AM, Shawn Guo wrote:

[..]

Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines

For this quick solution to be valid I propose to reject PT_LOAD in the else clause, and otherwise validate that phdrs[1].p_offset + hash_size <= fw->size.

I'm not sure what you propose here are required for non-split firmware.

Would it help if I sent a patch based on yours, with this extra validation and comments + commit message explaining the cases?

Quoted 7 lines
Quoted 3 lines

Alternatively we can try re-spinning the patches from Siddharth while taking review comments, the possible .mdt == .b00 + . b01 packing, and my suggestion to handle the headers generically into account.

I would suggest this path. It's been 8 months, so Siddharth probably lost the interest here. It's reasonable that someone picks up the work.

Prior to 498b98e93900 ("soc: qcom: mdt_loader: Support loading non-split images") we'd just blindly pass the entire .mdt into the SCM call.

So reading through your discussion and looking at the problem is that I assumed (based on the firmware files I looked at) that the hash segment is explicitly mentioned in the ELF header - i.e. segment 0 and 1 are not PT_LOAD and define the part that should be passed to init_image, and that this may or may not be packed.

And the problem we have here is that we have the packed case, but the hash segment isn't described explicitly in the ELF header.

Quoted 16 lines
Quoted 16 lines
Quoted 6 lines
Quoted 5 lines

The aforementioned patch series in qcom_mdt_bins_are_split (and certain firmware files from Sony phones) show that it is also possible to read PT_NULL headers from external files, as long as the header references a range that is out of bounds of the mdt file.

Quoted 6 lines

It's been shown that hashes can be read from .mdt or hash segment, and independently the hash segment type could be PT_NULL or PT_LOAD. With Siddharth's patch, instead of using the hash duplication in .mdt, hash will be read from hash segment. But again, to resolve my problem, the assertion that hash segment type cannot be PT_LOAD has to be dropped. And that's the only thing my patch is doing.

Correct. If I were to respin Siddharth's patchset, I'd write qcom_mdt_read_metadata as follows:

Quoted 8 lines
Quoted 8 lines

1. Find the header that contains QCOM_MDT_TYPE_HASH (allowing PT_NONE and PT_LOAD);
2. If ehdr_size + hash_size == fw->size, this is split firmware and the mdt file can be used as-is for metadata;
3. If the type is PT_LOAD, _or_ if the type is PT_NULL but p_offset + p_filesz does not fit inside the .mdt, this is (a variant of) split-firmware, and the hash needs to be loaded from an external file and appended to the ELF header.

I would expect that PT_LOAD denotes that the segment should be loaded into the final firmware region and that the hash segment would be PT_NULL regardless of being part of the .mdt, single .mbn or a separate .bNN segment.

Quoted 3 lines
Quoted 2 lines

4. If none of the above, this is a non-split firmware file. Concatenate the ELF and hash headers by reading them directly from the firmware.

I'm happy with this plan and I think Sid will be as well.

I hope that we can introduce a change that fixes Shawn's reported problem first, so that can be backported to stable, and then add Sid's need for loading the additional .bNN after that (same series is fine).

Quoted 3 lines

Looks good to me. I will be happy to review patches if you pick up the work.

Thanks to the both of you for looking into this!

Regards,
Bjorn

#14 lore raw

Hi Bjorn,

Quoted 27 lines · Bjorn Andersson
Quoted 2 lines · Shawn Guo
Quoted 1 line · Marijn Suijten

On 8/27/21 8:24 AM, Shawn Guo wrote:

[..]

Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines

For this quick solution to be valid I propose to reject PT_LOAD in the else clause, and otherwise validate that phdrs[1].p_offset + hash_size <= fw->size.

I'm not sure what you propose here are required for non-split firmware.

Would it help if I sent a patch based on yours, with this extra validation and comments + commit message explaining the cases?

Quoted 7 lines
Quoted 3 lines

Alternatively we can try re-spinning the patches from Siddharth while taking review comments, the possible .mdt == .b00 + . b01 packing, and my suggestion to handle the headers generically into account.

I would suggest this path. It's been 8 months, so Siddharth probably lost the interest here. It's reasonable that someone picks up the work.

Prior to 498b98e93900 ("soc: qcom: mdt_loader: Support loading non-split images") we'd just blindly pass the entire .mdt into the SCM call.

Quoted 8 lines

So reading through your discussion and looking at the problem is that I assumed (based on the firmware files I looked at) that the hash segment is explicitly mentioned in the ELF header - i.e. segment 0 and 1 are not PT_LOAD and define the part that should be passed to init_image, and that this may or may not be packed.

And the problem we have here is that we have the packed case, but the hash segment isn't described explicitly in the ELF header.

The hash segment is still explicitly described by the presence of QCOM_MDT_TYPE_HASH, and the external file (.b01 when the program header is in slot 1) contains the hash signature. And as Shawn demonstrated, concatenating the ELF header in .b00 and this hash in .b01 results in the .mdt file. This is the case we can (and already) optimize for, by passing the entire .mdt as-is into SCM if such packing is detected.

Quoted 24 lines
Quoted 15 lines
Quoted 15 lines
Quoted 11 lines
Quoted 4 lines

The aforementioned patch series in qcom_mdt_bins_are_split (and certain firmware files from Sony phones) show that it is also possible to read PT_NULL headers from external files, as long as the header references a range that is out of bounds of the mdt file.

It's been shown that hashes can be read from .mdt or hash segment, and independently the hash segment type could be PT_NULL or PT_LOAD. With Siddharth's patch, instead of using the hash duplication in .mdt, hash will be read from hash segment. But again, to resolve my problem, the assertion that hash segment type cannot be PT_LOAD has to be dropped. And that's the only thing my patch is doing.

Correct. If I were to respin Siddharth's patchset, I'd write qcom_mdt_read_metadata as follows:

Quoted 8 lines
Quoted 8 lines

1. Find the header that contains QCOM_MDT_TYPE_HASH (allowing PT_NONE

    and PT_LOAD);
2. If `ehdr_size + hash_size == fw->size`, this is split firmware and
    the mdt file can be used as-is for metadata;

3. If the type is PT_LOAD, _or_ if the type is PT_NULL but p_offset + p_filesz does not fit inside the .mdt, this is (a variant of) split-firmware, and the hash needs to be loaded from an external file and appended to the ELF header.

Quoted 4 lines

I would expect that PT_LOAD denotes that the segment should be loaded into the final firmware region and that the hash segment would be PT_NULL regardless of being part of the .mdt, single .mbn or a separate .bNN segment.

I have two mdt files here that load the hash externally, one with PT_LOAD and one with PT_NULL annotation (QCOM_MDT_TYPE_HASH is set). Both have their p_offset and/or p_offset+p_filesz well out of bounds of the .mdt file, but the .mdt file is of the packed variant (.b00 + .b01 == .mdt).

This packing seems to be an optimization, making it possible to send the .mdt contents as-is over SCM without having to load and concatenate the header from a separate file.

The image produced by __qcom_mdt_load should still be reconstructed based on the layout specified in the program headers, of course.

Quoted 9 lines
Quoted 3 lines
Quoted 2 lines

4. If none of the above, this is a non-split firmware file. Concatenate the ELF and hash headers by reading them directly from the firmware.

I'm happy with this plan and I think Sid will be as well.

I hope that we can introduce a change that fixes Shawn's reported problem first, so that can be backported to stable, and then add Sid's need for loading the additional .bNN after that (same series is fine).

I'm totally for fixing Shawn's reported issue first, before diving into extending mdt_loader as per Sid's patches. In hindsight I'm not sure if I should be the person writing that until coming across a platform/firmware that needs this setup (hash in a different header, hash not included in the packed .mdt file).

Note that backporting needs a Fixes: tag, probably on 498b98e93900.

Shawn, do you want to send your reworded patch with the necessary safeguards as v2, or should I send it?

- Marijn

#15 lore raw
Quoted 36 lines · Marijn Suijten

Hi Bjorn,

Quoted 33 lines · Bjorn Andersson
Quoted 2 lines · Shawn Guo
Quoted 1 line · Marijn Suijten

On 8/27/21 8:24 AM, Shawn Guo wrote:

[..]

Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines

For this quick solution to be valid I propose to reject PT_LOAD in the else clause, and otherwise validate that phdrs[1].p_offset + hash_size <= fw->size.

I'm not sure what you propose here are required for non-split firmware.

Would it help if I sent a patch based on yours, with this extra validation and comments + commit message explaining the cases?

Quoted 7 lines
Quoted 3 lines

Alternatively we can try re-spinning the patches from Siddharth while taking review comments, the possible .mdt == .b00 + . b01 packing, and my suggestion to handle the headers generically into account.

I would suggest this path. It's been 8 months, so Siddharth probably lost the interest here. It's reasonable that someone picks up the work.

Prior to 498b98e93900 ("soc: qcom: mdt_loader: Support loading non-split images") we'd just blindly pass the entire .mdt into the SCM call.

So reading through your discussion and looking at the problem is that I assumed (based on the firmware files I looked at) that the hash segment is explicitly mentioned in the ELF header - i.e. segment 0 and 1 are not PT_LOAD and define the part that should be passed to init_image, and that this may or may not be packed.

Quoted 7 lines
Quoted 2 lines

And the problem we have here is that we have the packed case, but the hash segment isn't described explicitly in the ELF header.

The hash segment is still explicitly described by the presence of QCOM_MDT_TYPE_HASH, and the external file (.b01 when the program header is in slot 1) contains the hash signature.

But in Shawn's firmware the hashes follows the elf header in the .mdt and both are described in a single program header. Are you saying that this entry is of type QCOM_MDT_TYPE_HASH?

Quoted 5 lines

And as Shawn demonstrated,
concatenating the ELF header in .b00 and this hash in .b01 results in the .mdt file. This is the case we can (and already) optimize for, by passing the entire .mdt as-is into SCM if such packing is detected.

In the other case, where the hash is described by a separate program header, we need to ensure that it's concatenated directly following the ELF header.

There are three cases here:
1) It's already directly following the ELF header and we should send ehdr_size + hash_size bytes to PAS.
2) It's in the loaded file, but it's not tightly packed (this is what we see in the .mbn). In this case we need to pack up the two pieces before we send them to PAS.
3) It's not part of the loaded .mdt, in which case we need to read it from whichever program header that happens to contain it (might not be .b01).

In the case of #1 we should not do #3, because I've seen several cases where the signature in .b01 does not match the one present in the .mdt.

I do expect that passing the metadata of ELF+hash+b01 will still work, as it would ignore the tailing garbage. Replacing ELF+hash with ELF+b01 is different from what we've been doing all these years, so that will require retesting on all possible platforms...

So I think what we're looking at it the three possible operations:
1) init_image(ELF+hash)
2) init_image(pack(ELF+hash))
3) init_image(pack(ELF+load(b01)))

Let's see if Sid agrees.

Quoted 30 lines
Quoted 24 lines
Quoted 15 lines
Quoted 15 lines
Quoted 11 lines
Quoted 4 lines

The aforementioned patch series in qcom_mdt_bins_are_split (and certain firmware files from Sony phones) show that it is also possible to read PT_NULL headers from external files, as long as the header references a range that is out of bounds of the mdt file.

It's been shown that hashes can be read from .mdt or hash segment, and independently the hash segment type could be PT_NULL or PT_LOAD. With Siddharth's patch, instead of using the hash duplication in .mdt, hash will be read from hash segment. But again, to resolve my problem, the assertion that hash segment type cannot be PT_LOAD has to be dropped. And that's the only thing my patch is doing.

Correct. If I were to respin Siddharth's patchset, I'd write qcom_mdt_read_metadata as follows:

Quoted 8 lines
Quoted 8 lines

1. Find the header that contains QCOM_MDT_TYPE_HASH (allowing PT_NONE

    and PT_LOAD);
2. If `ehdr_size + hash_size == fw->size`, this is split firmware and
    the mdt file can be used as-is for metadata;

3. If the type is PT_LOAD, _or_ if the type is PT_NULL but p_offset + p_filesz does not fit inside the .mdt, this is (a variant of) split-firmware, and the hash needs to be loaded from an external file and appended to the ELF header.

Quoted 4 lines

I would expect that PT_LOAD denotes that the segment should be loaded into the final firmware region and that the hash segment would be PT_NULL regardless of being part of the .mdt, single .mbn or a separate .bNN segment.

Quoted 5 lines

I have two mdt files here that load the hash externally, one with PT_LOAD and one with PT_NULL annotation (QCOM_MDT_TYPE_HASH is set). Both have their p_offset and/or p_offset+p_filesz well out of bounds of the .mdt file, but the .mdt file is of the packed variant (.b00 + .b01 == .mdt).

Ahh, because mdt_phdr_valid() will discard them anyways as we look for PT_LOAD segments...

Quoted 4 lines

This packing seems to be an optimization, making it possible to send the .mdt contents as-is over SCM without having to load and concatenate the header from a separate file.

Yes, that's probably the case. I was surprised to see the need for repacking the two segments in the .mbn case. I can only assume that Windows does this...

Regards,
Bjorn

Quoted 26 lines

The image produced by __qcom_mdt_load should still be reconstructed based on the layout specified in the program headers, of course.

Quoted 9 lines
Quoted 3 lines
Quoted 2 lines

4. If none of the above, this is a non-split firmware file. Concatenate the ELF and hash headers by reading them directly from the firmware.

I'm happy with this plan and I think Sid will be as well.

I hope that we can introduce a change that fixes Shawn's reported problem first, so that can be backported to stable, and then add Sid's need for loading the additional .bNN after that (same series is fine).

I'm totally for fixing Shawn's reported issue first, before diving into extending mdt_loader as per Sid's patches. In hindsight I'm not sure if I should be the person writing that until coming across a platform/firmware that needs this setup (hash in a different header, hash not included in the packed .mdt file).

Note that backporting needs a Fixes: tag, probably on 498b98e93900.

Shawn, do you want to send your reworded patch with the necessary safeguards as v2, or should I send it?

- Marijn

#16 lore raw

Hi Bjorn,

Quoted 17 lines · Bjorn Andersson

On Fri 27 Aug 10:40 PDT 2021, Marijn Suijten wrote:
[..]

Quoted 15 lines · Bjorn Andersson
Quoted 12 lines

[..]
Prior to 498b98e93900 ("soc: qcom: mdt_loader: Support loading non-split images") we'd just blindly pass the entire .mdt into the SCM call.

So reading through your discussion and looking at the problem is that I assumed (based on the firmware files I looked at) that the hash segment is explicitly mentioned in the ELF header - i.e. segment 0 and 1 are not PT_LOAD and define the part that should be passed to init_image, and that this may or may not be packed.

And the problem we have here is that we have the packed case, but the hash segment isn't described explicitly in the ELF header.

Quoted 5 lines
Quoted 3 lines

The hash segment is still explicitly described by the presence of QCOM_MDT_TYPE_HASH, and the external file (.b01 when the program header is in slot 1) contains the hash signature.

But in Shawn's firmware the hashes follows the elf header in the .mdt

Apologies, I wasn't denying that the hash doesn't follow the ELF header in the .mdt file (in his example, ELF (.b00) + hash (.b01) == .mdt), simply forgot to mention it.

Quoted 1 line

and both are described in a single program header.

Only the ELF header is described in the first program header, which is merely 0x001b4 bytes in size. The hash segment is described by the second header, which loads it from an external file.

The trailing data in the .mdt file directly after the ELF header is unspecified by the program headers (all other tables are PT_LOAD after all). As discussed below this seems to be an "ease-of-use" feature of the .mdt, doubling as metadata sent directly to SCM. Its "presence" is (seemingly) detected by comparing the size of the mdt file against the summed size of the ELF header and the segment containing QCOM_MDT_TYPE_HASH.

Quoted 1 line

Are you saying that this entry is of type QCOM_MDT_TYPE_HASH?

According to Shawn's validation (matching what I see locally), the second program header starting at 0x1000 has the QCOM_MDT_TYPE_HASH bit set.

Quoted 9 lines
Quoted 5 lines

And as Shawn demonstrated,
concatenating the ELF header in .b00 and this hash in .b01 results in the .mdt file. This is the case we can (and already) optimize for, by passing the entire .mdt as-is into SCM if such packing is detected.

In the other case, where the hash is described by a separate program header, we need to ensure that it's concatenated directly following the ELF header.

Shawn's case has the hash described in a separate program header as well, but it is also appended directly after the ELF header. The program headers do not specify anything about this region in the .mdt file though.

Quoted 3 lines

There are three cases here:
1) It's already directly following the ELF header and we should send ehdr_size + hash_size bytes to PAS.

Ack.

Quoted 3 lines

2) It's in the loaded file, but it's not tightly packed (this is what we see in the .mbn). In this case we need to pack up the two pieces before we send them to PAS.

Ack.

Quoted 3 lines

3) It's not part of the loaded .mdt, in which case we need to read it from whichever program header that happens to contain it (might not be .b01).

How should we detect that it is not part of the .mdt? Will the size of ELF header + hash segment simply not equal the size of the .mdt file?

Quoted 2 lines

In the case of #1 we should not do #3, because I've seen several cases where the signature in .b01 does not match the one present in the .mdt.

Oh, that really complicates matters :(. It means that SCM will see a different signature than what we eventually load into memory (recall that the hash bit in a .mdt is not specified in any program header, and will hence never be copied to memory. The "memory size" [p_memsz] of the ELF header is zero, and will never be copied either). Maybe that's never read though.

Quoted 1 line

I do expect that passing the metadata of ELF+hash+b01 will still work,

I don't think we have an ELF+hash+b01 case. It's always ELF+hash, but that hash can come from one of three locations as described in your summation below.

Quoted 3 lines

as it would ignore the tailing garbage. Replacing ELF+hash with ELF+b01 is different from what we've been doing all these years, so that will require retesting on all possible platforms...

I don't think we should be doing this replacement (Sid's patch does, which seems to have lost this check) and simply keep the ehdr_size + hash_size == fw->size predicate for direct ELF+hash reading from a .mdt.

Quoted 4 lines

So I think what we're looking at it the three possible operations:
1) init_image(ELF+hash)
2) init_image(pack(ELF+hash))
3) init_image(pack(ELF+load(b01)))

Correct, and all three cases should be handled by the four steps I mentioned to implement qcom_mdt_read_metadata.

Quoted 21 lines

Let's see if Sid agrees.

Quoted 19 lines
Quoted 17 lines
Quoted 4 lines
Quoted 4 lines

[..]
Correct. If I were to respin Siddharth's patchset, I'd write qcom_mdt_read_metadata as follows:

Quoted 8 lines
Quoted 8 lines

1. Find the header that contains QCOM_MDT_TYPE_HASH (allowing PT_NONE

     and PT_LOAD);
2. If `ehdr_size + hash_size == fw->size`, this is split firmware and
     the mdt file can be used as-is for metadata;

3. If the type is PT_LOAD, _or_ if the type is PT_NULL but p_offset + p_filesz does not fit inside the .mdt, this is (a variant of) split-firmware, and the hash needs to be loaded from an external file and appended to the ELF header.

I would expect that PT_LOAD denotes that the segment should be loaded into the final firmware region and that the hash segment would be PT_NULL regardless of being part of the .mdt, single .mbn or a separate .bNN segment.

Quoted 8 lines
Quoted 5 lines

I have two mdt files here that load the hash externally, one with PT_LOAD and one with PT_NULL annotation (QCOM_MDT_TYPE_HASH is set). Both have their p_offset and/or p_offset+p_filesz well out of bounds of the .mdt file, but the .mdt file is of the packed variant (.b00 + .b01 == .mdt).

Ahh, because mdt_phdr_valid() will discard them anyways as we look for PT_LOAD segments...

Yes. Curious how that works for mbn files then, as those should not have externally loadable files and hence no PT_LOAD segments?

Quoted 8 lines
Quoted 4 lines

This packing seems to be an optimization, making it possible to send the .mdt contents as-is over SCM without having to load and concatenate the header from a separate file.

Yes, that's probably the case. I was surprised to see the need for repacking the two segments in the .mbn case. I can only assume that Windows does this...

I take it the .mbns also do not have a program header defining the memory contents directly after the ELF header? Is this part zero-initialized in the file, or is it pre-initialized with the hash just like .mdt?

- Marijn