Skip to contentSkip to conversation

lib: zstd: Fix -Wstringop-overflow warning

PATCH In review Kees Cook 2 participants 5 messages Jan 4, 2023 – Jan 14, 2023 Zstd

lore
Share & tools
lore

Kees Cook

author
#0 lore raw

PATCH v2 1/1 lib: zstd: Fix -Wstringop-overflow warning

Fix the following -Wstringop-overflow warning when building with GCC 11+:

lib/zstd/decompress/huf_decompress.c: In function ‘HUF_readDTableX2_wksp’:
lib/zstd/decompress/huf_decompress.c:700:5: warning: ‘HUF_fillDTableX2.constprop’ accessing 624 bytes in a region of size 52 [-Wstringop-overflow=]

  700 |     HUF_fillDTableX2(dt, maxTableLog,
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  701 |                    wksp->sortedSymbol, sizeOfSort,
      |                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  702 |                    wksp->rankStart0, wksp->rankVal, maxW,
      |                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  703 |                    tableLog+1,
      |                    ~~~~~~~~~~~
  704 |                    wksp->calleeWksp, sizeof(wksp->calleeWksp) / sizeof(U32));
      |

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
lib/zstd/decompress/huf_decompress.c:700:5: note: referencing argument 6 of type ‘U32 (*)[13]’ {aka ‘unsigned int (*)[13]’} lib/zstd/decompress/huf_decompress.c:571:13: note: in a call to function ‘HUF_fillDTableX2.constprop’
571 | static void HUF_fillDTableX2(HUF_DEltX2* DTable, const U32 targetLog,

      |             ^~~~~~~~~~~~~~~~

by using pointer notation instead of array notation.

This is one of the last remaining warnings to be fixed before globally enabling -Wstringop-overflow.

---
v2: use "rankValCol_t *" instead of U32
v1: https://lore.kernel.org/lkml/20220330193352.GA119296@embeddedor/

---
 lib/zstd/decompress/huf_decompress.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/zstd/decompress/huf_decompress.c +1 -1
index 89b269a641c7..60958afebc41 100644
@@ -985,7 +985,7 @@ static void HUF_fillDTableX2Level2(HUF_DEltX2* DTable, U32 targetLog, const U32
 
 static void HUF_fillDTableX2(HUF_DEltX2* DTable, const U32 targetLog,
                            const sortedSymbol_t* sortedList,
-                           const U32* rankStart, rankVal_t rankValOrigin, const U32 maxWeight,
+                           const U32* rankStart, rankValCol_t *rankValOrigin, const U32 maxWeight,
                            const U32 nbBitsBaseline)
 {
     U32* const rankVal = rankValOrigin[0];
Signature · 1 line
2.34.1
#1 lore raw
Quoted 27 lines · Kees Cook

!-------------------------------------------------------------------|
This Message Is From an External Sender

|-------------------------------------------------------------------!

Fix the following -Wstringop-overflow warning when building with GCC 11+:

lib/zstd/decompress/huf_decompress.c: In function ‘HUF_readDTableX2_wksp’:
lib/zstd/decompress/huf_decompress.c:700:5: warning: ‘HUF_fillDTableX2.constprop’ accessing 624 bytes in a region of size 52 [-Wstringop-overflow=]

 700 |     HUF_fillDTableX2(dt, maxTableLog,
     |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 701 |                    wksp->sortedSymbol, sizeOfSort,
     |                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 702 |                    wksp->rankStart0, wksp->rankVal, maxW,
     |                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 703 |                    tableLog+1,
     |                    ~~~~~~~~~~~
 704 |                    wksp->calleeWksp, sizeof(wksp->calleeWksp) / sizeof(U32));
     |

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
lib/zstd/decompress/huf_decompress.c:700:5: note: referencing argument 6 of type ‘U32 (*)[13]’ {aka ‘unsigned int (*)[13]’} lib/zstd/decompress/huf_decompress.c:571:13: note: in a call to function ‘HUF_fillDTableX2.constprop’
571 | static void HUF_fillDTableX2(HUF_DEltX2* DTable, const U32 targetLog,

Quoted 6 lines
     |             ^~~~~~~~~~~~~~~~

by using pointer notation instead of array notation.

This is one of the last remaining warnings to be fixed before globally enabling -Wstringop-overflow.

The patch looks correct to me, thanks for reviving it. But, I was attempting to reproduce the issue, so I could better understand what's going on, and I wasn't able to reproduce it myself.

To attempt to reproduce, I applied this patch

---

lib/zstd/Makefile +2 -0
index 20f08c644b71..190d3d5ab4be 100644
@@ -12,6 +12,8 @@ obj-$(CONFIG_ZSTD_COMPRESS) += zstd_compress.o
 obj-$(CONFIG_ZSTD_DECOMPRESS) += zstd_decompress.o
 obj-$(CONFIG_ZSTD_COMMON) += zstd_common.o
 
+ccflags-y := -Wstringop-overflow=4 -Werror
+
 zstd_compress-y := \
                zstd_compress_module.o \
                compress/fse_compress.o \

---

Then compiled on x86-64 with gcc 12.2.0 on tag v6.2-rc3. I saw no errors. I also tried with just -Wstringop-overflow, and on upstream zstd. I tried to make a minimal reproducer on godbolt, so I could see if it was the gcc version, but wasn't able to make it fail with any of them https://gcc.godbolt.org/z/Exzq9arMr.

Could you please tell me how to reproduce this warning?

Best,
Nick Terrell

Quoted 27 lines

---
v2: use "rankValCol_t *" instead of U32
v1: https://lore.kernel.org/lkml/20220330193352.GA119296@embeddedor/
---
lib/zstd/decompress/huf_decompress.c | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)
lib/zstd/decompress/huf_decompress.c +1 -1
index 89b269a641c7..60958afebc41 100644
@@ -985,7 +985,7 @@ static void HUF_fillDTableX2Level2(HUF_DEltX2* DTable, U32 targetLog, const U32
 
 static void HUF_fillDTableX2(HUF_DEltX2* DTable, const U32 targetLog,
                           const sortedSymbol_t* sortedList,
-                           const U32* rankStart, rankVal_t rankValOrigin, const U32 maxWeight,
+                           const U32* rankStart, rankValCol_t *rankValOrigin, const U32 maxWeight,
                           const U32 nbBitsBaseline)
 {
    U32* const rankVal = rankValOrigin[0];
Signature · 1 line
2.34.1
author
#2 lore raw
Quoted 55 lines · Nick Terrell
Quoted 27 lines · Kees Cook

!-------------------------------------------------------------------|
This Message Is From an External Sender

|-------------------------------------------------------------------!

Fix the following -Wstringop-overflow warning when building with GCC 11+:

lib/zstd/decompress/huf_decompress.c: In function ‘HUF_readDTableX2_wksp’:
lib/zstd/decompress/huf_decompress.c:700:5: warning: ‘HUF_fillDTableX2.constprop’ accessing 624 bytes in a region of size 52 [-Wstringop-overflow=]

 700 |     HUF_fillDTableX2(dt, maxTableLog,
     |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 701 |                    wksp->sortedSymbol, sizeOfSort,
     |                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 702 |                    wksp->rankStart0, wksp->rankVal, maxW,
     |                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 703 |                    tableLog+1,
     |                    ~~~~~~~~~~~
 704 |                    wksp->calleeWksp, sizeof(wksp->calleeWksp) / sizeof(U32));
     |

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
lib/zstd/decompress/huf_decompress.c:700:5: note: referencing argument 6 of type ‘U32 (*)[13]’ {aka ‘unsigned int (*)[13]’} lib/zstd/decompress/huf_decompress.c:571:13: note: in a call to function ‘HUF_fillDTableX2.constprop’
571 | static void HUF_fillDTableX2(HUF_DEltX2* DTable, const U32 targetLog,

Quoted 6 lines
     |             ^~~~~~~~~~~~~~~~

by using pointer notation instead of array notation.

This is one of the last remaining warnings to be fixed before globally enabling -Wstringop-overflow.

The patch looks correct to me, thanks for reviving it. But, I was attempting to reproduce the issue, so I could better understand what's going on, and I wasn't able to reproduce it myself.

To attempt to reproduce, I applied this patch

---

lib/zstd/Makefile +2 -1
index 20f08c644b71..190d3d5ab4be 100644
@@ -12,6 +12,8 @@ obj-$(CONFIG_ZSTD_COMPRESS) += zstd_compress.o
 obj-$(CONFIG_ZSTD_DECOMPRESS) += zstd_decompress.o
 obj-$(CONFIG_ZSTD_COMMON) += zstd_common.o
 
+ccflags-y := -Wstringop-overflow=4 -Werror
+
 zstd_compress-y := \
                zstd_compress_module.o \
                compress/fse_compress.o \
---
Quoted 6 lines

Then compiled on x86-64 with gcc 12.2.0 on tag v6.2-rc3. I saw no errors. I also tried with just -Wstringop-overflow, and on upstream zstd. I tried to make a minimal reproducer on godbolt, so I could see if it was the gcc version, but wasn't able to make it fail with any of them https://gcc.godbolt.org/z/Exzq9arMr.

Could you please tell me how to reproduce this warning?

I saw it like so with next-20230113 on x86_64:

$ gcc --version
gcc (Ubuntu 12.2.0-3ubuntu1) 12.2.0
...
$ make KCFLAGS=-Wstringop-overflow allmodconfig lib/zstd/decompress/huf_decompress.o
Signature · 1 line
Kees Cook
Reviewed-by
#3 lore raw
Quoted 70 lines · Kees Cook

!-------------------------------------------------------------------|
This Message Is From an External Sender

|-------------------------------------------------------------------!

Quoted 60 lines · Nick Terrell
Quoted 27 lines · Kees Cook

!-------------------------------------------------------------------| This Message Is From an External Sender

|-------------------------------------------------------------------!

Fix the following -Wstringop-overflow warning when building with GCC 11+:

lib/zstd/decompress/huf_decompress.c: In function ‘HUF_readDTableX2_wksp’:
lib/zstd/decompress/huf_decompress.c:700:5: warning: ‘HUF_fillDTableX2.constprop’ accessing 624 bytes in a region of size 52 [-Wstringop-overflow=]

700 |     HUF_fillDTableX2(dt, maxTableLog,
    |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
701 |                    wksp->sortedSymbol, sizeOfSort,
    |                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
702 |                    wksp->rankStart0, wksp->rankVal, maxW,
    |                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
703 |                    tableLog+1,
    |                    ~~~~~~~~~~~
704 |                    wksp->calleeWksp, sizeof(wksp->calleeWksp) / sizeof(U32));
    |

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
lib/zstd/decompress/huf_decompress.c:700:5: note: referencing argument 6 of type ‘U32 (*)[13]’ {aka ‘unsigned int (*)[13]’} lib/zstd/decompress/huf_decompress.c:571:13: note: in a call to function ‘HUF_fillDTableX2.constprop’ 571 | static void HUF_fillDTableX2(HUF_DEltX2* DTable, const U32 targetLog,

Quoted 6 lines
    |             ^~~~~~~~~~~~~~~~

by using pointer notation instead of array notation.

This is one of the last remaining warnings to be fixed before globally enabling -Wstringop-overflow.

The patch looks correct to me, thanks for reviving it. But, I was attempting to reproduce the issue, so I could better understand what's going on, and I wasn't able to reproduce it myself.

To attempt to reproduce, I applied this patch

---

lib/zstd/Makefile +0 -0
index 20f08c644b71..190d3d5ab4be 100644
@@ -12,6 +12,8 @@ obj-$(CONFIG_ZSTD_COMPRESS) += zstd_compress.o

obj-$(CONFIG_ZSTD_DECOMPRESS) += zstd_decompress.o obj-$(CONFIG_ZSTD_COMMON) += zstd_common.o

lib/zstd/Makefile +2 -0
+ccflags-y := -Wstringop-overflow=4 -Werror
+
zstd_compress-y := \
               zstd_compress_module.o \
               compress/fse_compress.o \

---

Then compiled on x86-64 with gcc 12.2.0 on tag v6.2-rc3. I saw no errors. I also tried with just -Wstringop-overflow, and on upstream zstd. I tried to make a minimal reproducer on godbolt, so I could see if it was the gcc version, but wasn't able to make it fail with any of them https://gcc.godbolt.org/z/Exzq9arMr .

Quoted 8 lines
Quoted 1 line

Could you please tell me how to reproduce this warning?

I saw it like so with next-20230113 on x86_64:

$ gcc --version
gcc (Ubuntu 12.2.0-3ubuntu1) 12.2.0
...
$ make KCFLAGS=-Wstringop-overflow allmodconfig lib/zstd/decompress/huf_decompress.o

Thanks, I was able to repro it! I will merge this patch into my tree.

If you would like to submit the same patch upstream yourself, I will accept the PR, otherwise I can submit an upstream PR.

Just to be certain, this patch is to work around a shortcoming in -Wstringop-overflow, but the code was otherwise correct?

Reviewed-by Nick Terrell <[email protected] <mailto:[email protected]>>

Best,
Nick Terrell

Quoted 2 lines
Signature · 1 line
Kees Cook
author
#4 lore raw
Quoted 74 lines · Nick Terrell
Quoted 74 lines · Kees Cook

!-------------------------------------------------------------------|
This Message Is From an External Sender

|-------------------------------------------------------------------!

Quoted 53 lines · Nick Terrell
Quoted 27 lines · Kees Cook

!-------------------------------------------------------------------| This Message Is From an External Sender

|-------------------------------------------------------------------!

Fix the following -Wstringop-overflow warning when building with GCC 11+:

lib/zstd/decompress/huf_decompress.c: In function ‘HUF_readDTableX2_wksp’:
lib/zstd/decompress/huf_decompress.c:700:5: warning: ‘HUF_fillDTableX2.constprop’ accessing 624 bytes in a region of size 52 [-Wstringop-overflow=]

700 |     HUF_fillDTableX2(dt, maxTableLog,
    |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
701 |                    wksp->sortedSymbol, sizeOfSort,
    |                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
702 |                    wksp->rankStart0, wksp->rankVal, maxW,
    |                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
703 |                    tableLog+1,
    |                    ~~~~~~~~~~~
704 |                    wksp->calleeWksp, sizeof(wksp->calleeWksp) / sizeof(U32));
    |

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
lib/zstd/decompress/huf_decompress.c:700:5: note: referencing argument 6 of type ‘U32 (*)[13]’ {aka ‘unsigned int (*)[13]’} lib/zstd/decompress/huf_decompress.c:571:13: note: in a call to function ‘HUF_fillDTableX2.constprop’ 571 | static void HUF_fillDTableX2(HUF_DEltX2* DTable, const U32 targetLog,

Quoted 6 lines
    |             ^~~~~~~~~~~~~~~~

by using pointer notation instead of array notation.

This is one of the last remaining warnings to be fixed before globally enabling -Wstringop-overflow.

The patch looks correct to me, thanks for reviving it. But, I was attempting to reproduce the issue, so I could better understand what's going on, and I wasn't able to reproduce it myself.

To attempt to reproduce, I applied this patch

---

lib/zstd/Makefile +0 -0
index 20f08c644b71..190d3d5ab4be 100644
@@ -12,6 +12,8 @@ obj-$(CONFIG_ZSTD_COMPRESS) += zstd_compress.o

obj-$(CONFIG_ZSTD_DECOMPRESS) += zstd_decompress.o obj-$(CONFIG_ZSTD_COMMON) += zstd_common.o

lib/zstd/Makefile +2 -0
+ccflags-y := -Wstringop-overflow=4 -Werror
+
zstd_compress-y := \
               zstd_compress_module.o \
               compress/fse_compress.o \
Quoted 8 lines

---

Then compiled on x86-64 with gcc 12.2.0 on tag v6.2-rc3. I saw no errors. I also tried with just -Wstringop-overflow, and on upstream zstd. I tried to make a minimal reproducer on godbolt, so I could see if it was the gcc version, but wasn't able to make it fail with any of them https://gcc.godbolt.org/z/Exzq9arMr .

Could you please tell me how to reproduce this warning?

I saw it like so with next-20230113 on x86_64:

Quoted 6 lines
Quoted 4 lines
$ gcc --version
gcc (Ubuntu 12.2.0-3ubuntu1) 12.2.0
...
$ make KCFLAGS=-Wstringop-overflow allmodconfig lib/zstd/decompress/huf_decompress.o

Thanks, I was able to repro it! I will merge this patch into my tree.

Thanks!

Quoted 1 line

If you would like to submit the same patch upstream yourself, I will accept the PR, otherwise I can submit an upstream PR.

I don't know the process there, so if you could do it, I'd appreciate it.

Quoted 2 lines

Just to be certain, this patch is to work around a shortcoming in -Wstringop-overflow, but the code was otherwise correct?

It's the same result for the binary output. The types indicated in the function prototype meant there was cross-struct-member overflow (i.e. writing to the second array dimension when only 1 was specified), but the way to clear up intention isn't as clean here, so I'd kind of say half a code correctness issue, and half a work-around. *makes wavey hands gesture*

-Kees

Signature · 1 line
Kees Cook