lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop()
PATCH v7 1/1 lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop()
From: Heiko Stuebner <[email protected]>
When calculating rrtmp/rr rsa_gen_key_prop() tries to make (((rlen + 31) >> 5) + 1) steps in the rr uint32_t array and (((rlen + 7) >> 3) + 1) / 4 steps in uint32_t rrtmp[] with rlen being num_bits * 2
On a 4096bit key this comes down to to 257 uint32_t elements in rr and 256 elements in rrtmp but with the current allocation rr and rrtmp only have 129 uint32_t elements.
On 2048bit keys this works by chance as the defined max_rsa_size=4096 allocates a suitable number of elements, but with an actual 4096bit key this results in other memory parts getting overwritten.
So as suggested by Heinrich Schuchardt just use the actual bit-size of the key as base for the size calculation, in turn making the code compatible to any future keysizes.
rrtmp needs 2 + (((*prop)->num_bits * 2) >> 5) array elements.
---
v7:
rrtmp needs one more u32.
There is no label out. It is called err.
--- lib/rsa/rsa-keyprop.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
lib/rsa/rsa-keyprop.c +9 -5
@@ -654,14 +654,10 @@ int rsa_gen_key_prop(const void *key, uint32_t keylen, struct key_prop **prop) {struct rsa_key rsa_key;uint32_t *n =NULL , *rr =NULL , *rrtmp =NULL ; -const int max_rsa_size =4096 ;int rlen, i, ret; *prop =calloc (sizeof (**prop),1 ); - n =calloc (sizeof (uint32_t ),1 + (max_rsa_size >>5 )); - rr =calloc (sizeof (uint32_t ),1 + (max_rsa_size >>5 )); - rrtmp =calloc (sizeof (uint32_t ),1 + (max_rsa_size >>5 )); -if (!(*prop) || !n || !rr || !rrtmp) { +if (!(*prop)) { ret = -ENOMEM ; goto err; }
@@ -682,6 +678,14 @@ int rsa_gen_key_prop(const void *key, uint32_t keylen, struct key_prop **prop) }memcpy ((void *)(*prop)->modulus, &rsa_key.n[i], rsa_key.n_sz - i); + n =calloc (sizeof (uint32_t ),1 + ((*prop)->num_bits >>5 )); + rr =calloc (sizeof (uint32_t ),1 + (((*prop)->num_bits *2 ) >>5 )); + rrtmp =calloc (sizeof (uint32_t ),2 + (((*prop)->num_bits *2 ) >>5 )); +if (!n || !rr || !rrtmp) { + ret = -ENOMEM ; + goto err; + } +/* exponent */ (*prop)->public_exponent =calloc (1 ,sizeof (uint64_t ));if (!(*prop)->public_exponent) {
--
2.27.0
PATCH v7 1/1 lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop()
Quoted 19 lines · Heinrich Schuchardt
From: Heiko Stuebner <[email protected]>
When calculating rrtmp/rr
rsa_gen_key_prop()tries to make (((rlen + 31) >> 5) + 1) steps in the rruint32_tarray and (((rlen + 7) >> 3) + 1) / 4 steps inuint32_trrtmp[] with rlen beingnum_bits* 2On a 4096bit key this comes down to to 257
uint32_telements in rr and 256 elements in rrtmp but with the current allocation rr and rrtmp only have 129uint32_telements.On 2048bit keys this works by chance as the defined max_rsa_size=4096 allocates a suitable number of elements, but with an actual 4096bit key this results in other memory parts getting overwritten.
So as suggested by Heinrich Schuchardt just use the actual bit-size of the key as base for the size calculation, in turn making the code compatible to any future keysizes.
Quoted 7 lines
Suggested-by Heinrich Schuchardt <[email protected]>Signed-off-by Heiko Stuebner <[email protected]>Reviewed-by Simon Glass <[email protected]>rrtmp needs 2 + (((*prop)->num_bits * 2) >> 5) array elements.
Reviewed-by Heinrich Schuchardt <[email protected]>
Applied to u-boot/master, thanks!
Signature · 8 lines
Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 659 bytes Desc: not available URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200708/441a74bd/attachment.sig>