Fix sign compare error when checking file size

PATCH In review Paul Elder 3 participants 5 messages Nov 11 Media (V4L/DVB)

Share & tools

Paul Elder

author

2026-11-11 01:26 UTC #0 lore raw

PATCH 1/1 Fix sign compare error when checking file size

Fix the sign compare error when checking file size by casting the file size returned by ftell to unsigned int, as it is a signed long being compared against UINT_MAX.

This fixes the following compiler error:

FAILED: [code=1] raw2rgbpnm.p/raw2rgbpnm.c.o
/home/autobuild/autobuild/instance-5/output-1/host/bin/arm-buildroot-linux-gnueabi-gcc -Iraw2rgbpnm.p -I. -I.. -I../include -fdiagnostics-color=always -Wall -Winvalid-pch -Wextra -Werror -O3 -Wno-missing-field-initializers -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -D_FORTIFY_SOURCE=1 -MD -MQ raw2rgbpnm.p/raw2rgbpnm.c.o -MF raw2rgbpnm.p/raw2rgbpnm.c.o.d -o raw2rgbpnm.p/raw2rgbpnm.c.o -c ../raw2rgbpnm.c ../raw2rgbpnm.c: In function 'read_raw_data':
../raw2rgbpnm.c:173:36: error: comparison of integer expressions of different signedness: 'unsigned int' and 'long int' [-Werror=sign-compare]
173 | if (UINT_MAX / bpp < file_size)
| ^
../raw2rgbpnm.c:185:23: error: comparison of integer expressions of different signedness: 'long int' and 'unsigned int' [-Werror=sign-compare]
185 | if (file_size > UINT_MAX)
| ^

Signed-off-by Paul Elder <[email protected]>

---
I've been getting build failure reports from buildroot for this.
https://autobuild.buildroot.org/results/f1f070bc5ac72d8b0f2ed403273635b7459a9ae5/

---
 raw2rgbpnm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

raw2rgbpnm.c +2 -2

index 0c71be321746..39b186473624 100644
@@ -170,7 +170,7 @@ static unsigned char *read_raw_data(char *filename, unsigned int *width,
 	}
 
 	if (!*height && line_length) {
-		if (UINT_MAX / bpp < file_size)
+		if (UINT_MAX / bpp < (unsigned int)file_size)
 			error("line length too large");
 		*height = file_size / line_length;
 		*width = line_length * 8 / bpp;

@@ -182,7 +182,7 @@ static unsigned char *read_raw_data(char *filename, unsigned int *width,
 	if (!line_length || UINT_MAX / line_length < *height ||
 	    line_length < *width * bpp / 8)
 		error("stride %u is bad", line_length);
-	if (file_size > UINT_MAX)
+	if ((unsigned int)file_size > UINT_MAX)
 		error("too large file");
 	if ((unsigned int)file_size < line_length * *height)
 		error("out of input data");

Signature · 1 line

2.47.3

Laurent Pinchart

Reviewed-by on v1 1/1

2026-11-11 08:50 UTC #1 lore raw

Quoted 38 lines · Paul Elder

Fix the sign compare error when checking file size by casting the file size returned by ftell to unsigned int, as it is a signed long being compared against UINT_MAX.
This fixes the following compiler error:
FAILED: [code=1] raw2rgbpnm.p/raw2rgbpnm.c.o
/home/autobuild/autobuild/instance-5/output-1/host/bin/arm-buildroot-linux-gnueabi-gcc -Iraw2rgbpnm.p -I. -I.. -I../include -fdiagnostics-color=always -Wall -Winvalid-pch -Wextra -Werror -O3 -Wno-missing-field-initializers -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -D_FORTIFY_SOURCE=1 -MD -MQ raw2rgbpnm.p/raw2rgbpnm.c.o -MF raw2rgbpnm.p/raw2rgbpnm.c.o.d -o raw2rgbpnm.p/raw2rgbpnm.c.o -c ../raw2rgbpnm.c ../raw2rgbpnm.c: In function 'read_raw_data':
../raw2rgbpnm.c:173:36: error: comparison of integer expressions of different signedness: 'unsigned int' and 'long int' [-Werror=sign-compare]
173 | if (UINT_MAX / bpp < file_size)
| ^
../raw2rgbpnm.c:185:23: error: comparison of integer expressions of different signedness: 'long int' and 'unsigned int' [-Werror=sign-compare]
185 | if (file_size > UINT_MAX)
| ^
Signed-off-by Paul Elder <[email protected]>
---
I've been getting build failure reports from buildroot for this.
https://autobuild.buildroot.org/results/f1f070bc5ac72d8b0f2ed403273635b7459a9ae5/
---
 raw2rgbpnm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
raw2rgbpnm.c +1 -1
index 0c71be321746..39b186473624 100644
@@ -170,7 +170,7 @@ static unsigned char *read_raw_data(char *filename, unsigned int *width,
 	}
 
 	if (!*height && line_length) {
-		if (UINT_MAX / bpp < file_size)
+		if (UINT_MAX / bpp < (unsigned int)file_size)
 			error("line length too large");
 		*height = file_size / line_length;
 		*width = line_length * 8 / bpp;

raw2rgbpnm.c static unsigned char *read_raw_data(char *filename, unsigned int *width, · line 185

Quoted 5 lines

raw2rgbpnm.c +1 -1

@@ -182,7 +182,7 @@ static unsigned char *read_raw_data(char *filename, unsigned int *width,
 	if (!line_length || UINT_MAX / line_length < *height ||
 	    line_length < *width * bpp / 8)
 		error("stride %u is bad", line_length);
-	if (file_size > UINT_MAX)
+	if ((unsigned int)file_size > UINT_MAX)

A better option in my opinion is to turn file_size into an unsigned variable (after checking for errors from ftell()) and drop all the casts. There are already casts to unsigned int though, the cold is old, and there's little incentive in refactoring it extensively, so I think this patch is good enough.

Reviewed-by Laurent Pinchart <[email protected]>

Quoted 3 lines

raw2rgbpnm.c +0 -0

 		error("too large file");
 	if ((unsigned int)file_size < line_length * *height)
 		error("out of input data");

Signature · 3 lines

Regards,

Laurent Pinchart

Sakari Ailus

2026-11-11 09:23 UTC #2 lore raw

Hi Paul, Laurent,

Quoted 32 lines · Laurent Pinchart

Quoted 31 lines · Paul Elder
Fix the sign compare error when checking file size by casting the file size returned by ftell to unsigned int, as it is a signed long being compared against UINT_MAX.
This fixes the following compiler error:
FAILED: [code=1] raw2rgbpnm.p/raw2rgbpnm.c.o
/home/autobuild/autobuild/instance-5/output-1/host/bin/arm-buildroot-linux-gnueabi-gcc -Iraw2rgbpnm.p -I. -I.. -I../include -fdiagnostics-color=always -Wall -Winvalid-pch -Wextra -Werror -O3 -Wno-missing-field-initializers -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -D_FORTIFY_SOURCE=1 -MD -MQ raw2rgbpnm.p/raw2rgbpnm.c.o -MF raw2rgbpnm.p/raw2rgbpnm.c.o.d -o raw2rgbpnm.p/raw2rgbpnm.c.o -c ../raw2rgbpnm.c ../raw2rgbpnm.c: In function 'read_raw_data':
../raw2rgbpnm.c:173:36: error: comparison of integer expressions of different signedness: 'unsigned int' and 'long int' [-Werror=sign-compare]
173 | if (UINT_MAX / bpp < file_size)
| ^
../raw2rgbpnm.c:185:23: error: comparison of integer expressions of different signedness: 'long int' and 'unsigned int' [-Werror=sign-compare]
185 | if (file_size > UINT_MAX)
| ^
Signed-off-by Paul Elder <[email protected]>
---
I've been getting build failure reports from buildroot for this.
https://autobuild.buildroot.org/results/f1f070bc5ac72d8b0f2ed403273635b7459a9ae5/
---
 raw2rgbpnm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
raw2rgbpnm.c +0 -0
index 0c71be321746..39b186473624 100644
@@ -170,7 +170,7 @@ static unsigned char *read_raw_data(char *filename, unsigned int *width,
 	}
 

Quoted 12 lines

raw2rgbpnm.c static unsigned char *read_raw_data(char *filename, unsigned int *width, · line 185

Quoted 12 lines

raw2rgbpnm.c +2 -2

 	if (!*height && line_length) {
-		if (UINT_MAX / bpp < file_size)
+		if (UINT_MAX / bpp < (unsigned int)file_size)
 			error("line length too large");
 		*height = file_size / line_length;
 		*width = line_length * 8 / bpp;

@@ -182,7 +182,7 @@ static unsigned char *read_raw_data(char *filename, unsigned int *width,
 	if (!line_length || UINT_MAX / line_length < *height ||
 	    line_length < *width * bpp / 8)
 		error("stride %u is bad", line_length);
-	if (file_size > UINT_MAX)
+	if ((unsigned int)file_size > UINT_MAX)

This will never be true, as an unsigned int won't be larger than UINT_MAX. file_size is long so unsigned long would seem reasonable here. Although it seems there was no bug here, it's just a compiler warning on what could have been a bug...

Quoted 13 lines

A better option in my opinion is to turn file_size into an unsigned variable (after checking for errors from ftell()) and drop all the casts. There are already casts to unsigned int though, the cold is old, and there's little incentive in refactoring it extensively, so I think this patch is good enough.
Reviewed-by Laurent Pinchart <[email protected]>
Quoted 3 lines
raw2rgbpnm.c +0 -0
 		error("too large file");
 	if ((unsigned int)file_size < line_length * *height)
 		error("out of input data");

Signature · 3 lines

Regards,

Sakari Ailus

Laurent Pinchart

2026-11-11 09:28 UTC #3 lore raw

Quoted 42 lines · Sakari Ailus

Quoted 41 lines · Laurent Pinchart
Quoted 40 lines · Paul Elder
Fix the sign compare error when checking file size by casting the file size returned by ftell to unsigned int, as it is a signed long being compared against UINT_MAX.
This fixes the following compiler error:
FAILED: [code=1] raw2rgbpnm.p/raw2rgbpnm.c.o
/home/autobuild/autobuild/instance-5/output-1/host/bin/arm-buildroot-linux-gnueabi-gcc -Iraw2rgbpnm.p -I. -I.. -I../include -fdiagnostics-color=always -Wall -Winvalid-pch -Wextra -Werror -O3 -Wno-missing-field-initializers -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -D_FORTIFY_SOURCE=1 -MD -MQ raw2rgbpnm.p/raw2rgbpnm.c.o -MF raw2rgbpnm.p/raw2rgbpnm.c.o.d -o raw2rgbpnm.p/raw2rgbpnm.c.o -c ../raw2rgbpnm.c ../raw2rgbpnm.c: In function 'read_raw_data':
../raw2rgbpnm.c:173:36: error: comparison of integer expressions of different signedness: 'unsigned int' and 'long int' [-Werror=sign-compare]
173 | if (UINT_MAX / bpp < file_size)
| ^
../raw2rgbpnm.c:185:23: error: comparison of integer expressions of different signedness: 'long int' and 'unsigned int' [-Werror=sign-compare]
185 | if (file_size > UINT_MAX)
| ^
Signed-off-by Paul Elder <[email protected]>
---
I've been getting build failure reports from buildroot for this.
https://autobuild.buildroot.org/results/f1f070bc5ac72d8b0f2ed403273635b7459a9ae5/
---
 raw2rgbpnm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
raw2rgbpnm.c +1 -1
index 0c71be321746..39b186473624 100644
@@ -170,7 +170,7 @@ static unsigned char *read_raw_data(char *filename, unsigned int *width,
 	}
 
 	if (!*height && line_length) {
-		if (UINT_MAX / bpp < file_size)
+		if (UINT_MAX / bpp < (unsigned int)file_size)
 			error("line length too large");
 		*height = file_size / line_length;
 		*width = line_length * 8 / bpp;
@@ -182,7 +182,7 @@ static unsigned char *read_raw_data(char *filename, unsigned int *width,
 	if (!line_length || UINT_MAX / line_length < *height ||
 	    line_length < *width * bpp / 8)

Quoted 8 lines

Quoted 3 lines
raw2rgbpnm.c static unsigned char *read_raw_data(char *filename, unsigned int *width, · line 185
Quoted 3 lines
raw2rgbpnm.c +1 -1
 		error("stride %u is bad", line_length);
-	if (file_size > UINT_MAX)
+	if ((unsigned int)file_size > UINT_MAX)
This will never be true, as an unsigned int won't be larger than UINT_MAX. file_size is long so unsigned long would seem reasonable here. Although it seems there was no bug here, it's just a compiler warning on what could have been a bug...

What's your preference to silence this warning ? Would you like a minimal change, or a larger cleanup ?

Quoted 11 lines

Quoted 11 lines
A better option in my opinion is to turn file_size into an unsigned variable (after checking for errors from ftell()) and drop all the casts. There are already casts to unsigned int though, the cold is old, and there's little incentive in refactoring it extensively, so I think this patch is good enough.
Reviewed-by Laurent Pinchart <[email protected]>
Quoted 3 lines
raw2rgbpnm.c +0 -0
 		error("too large file");
 	if ((unsigned int)file_size < line_length * *height)
 		error("out of input data");

Signature · 3 lines

Regards,

Laurent Pinchart

Sakari Ailus

2026-11-11 09:31 UTC #4 lore raw

Quoted 52 lines · Laurent Pinchart

Quoted 42 lines · Sakari Ailus
Quoted 41 lines · Laurent Pinchart
Quoted 40 lines · Paul Elder
Fix the sign compare error when checking file size by casting the file size returned by ftell to unsigned int, as it is a signed long being compared against UINT_MAX.
This fixes the following compiler error:
FAILED: [code=1] raw2rgbpnm.p/raw2rgbpnm.c.o
/home/autobuild/autobuild/instance-5/output-1/host/bin/arm-buildroot-linux-gnueabi-gcc -Iraw2rgbpnm.p -I. -I.. -I../include -fdiagnostics-color=always -Wall -Winvalid-pch -Wextra -Werror -O3 -Wno-missing-field-initializers -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -g0 -D_FORTIFY_SOURCE=1 -MD -MQ raw2rgbpnm.p/raw2rgbpnm.c.o -MF raw2rgbpnm.p/raw2rgbpnm.c.o.d -o raw2rgbpnm.p/raw2rgbpnm.c.o -c ../raw2rgbpnm.c ../raw2rgbpnm.c: In function 'read_raw_data':
../raw2rgbpnm.c:173:36: error: comparison of integer expressions of different signedness: 'unsigned int' and 'long int' [-Werror=sign-compare]
173 | if (UINT_MAX / bpp < file_size)
| ^
../raw2rgbpnm.c:185:23: error: comparison of integer expressions of different signedness: 'long int' and 'unsigned int' [-Werror=sign-compare]
185 | if (file_size > UINT_MAX)
| ^
Signed-off-by Paul Elder <[email protected]>
---
I've been getting build failure reports from buildroot for this.
https://autobuild.buildroot.org/results/f1f070bc5ac72d8b0f2ed403273635b7459a9ae5/
---
 raw2rgbpnm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
raw2rgbpnm.c +1 -1
index 0c71be321746..39b186473624 100644
@@ -170,7 +170,7 @@ static unsigned char *read_raw_data(char *filename, unsigned int *width,
 	}
 
 	if (!*height && line_length) {
-		if (UINT_MAX / bpp < file_size)
+		if (UINT_MAX / bpp < (unsigned int)file_size)
 			error("line length too large");
 		*height = file_size / line_length;
 		*width = line_length * 8 / bpp;
@@ -182,7 +182,7 @@ static unsigned char *read_raw_data(char *filename, unsigned int *width,
 	if (!line_length || UINT_MAX / line_length < *height ||
 	    line_length < *width * bpp / 8)
Quoted 8 lines
Quoted 3 lines
raw2rgbpnm.c static unsigned char *read_raw_data(char *filename, unsigned int *width, · line 185
Quoted 3 lines
raw2rgbpnm.c +1 -1
 		error("stride %u is bad", line_length);
-	if (file_size > UINT_MAX)
+	if ((unsigned int)file_size > UINT_MAX)
This will never be true, as an unsigned int won't be larger than UINT_MAX. file_size is long so unsigned long would seem reasonable here. Although it seems there was no bug here, it's just a compiler warning on what could have been a bug...

Quoted 2 lines

What's your preference to silence this warning ? Would you like a minimal change, or a larger cleanup ?

My suggestion was to use unsigned long instead of unsigned int.

In fact I'd move the file_size check above its first use while at it.

Signature · 1 line

Sakari Ailus