Skip to contentSkip to conversation

fixdep: use fflush() and ferror() to ensure successful write to files

PATCH Superseded Masahiro Yamada 3 participants 5 messages Feb 25, 2022 – Mar 1, 2022 2 messages not in the archive Kbuild

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 fixdep: use fflush() and ferror() to ensure successful write to files

Checking the return value of (v)printf does not ensure the successful write to the .cmd file.

Call fflush() and ferror() to make sure that everything has been written to the file.

---

Changes in v2:
- add error message

 scripts/basic/fixdep.c | 46 +++++++++++++++++-------------------------
 1 file changed, 19 insertions(+), 27 deletions(-)
scripts/basic/fixdep.c +19 -27
index 44e887cff49b..2328f9a641da 100644
@@ -105,25 +105,6 @@ static void usage(void)
 	exit(1);
 }
 
-/*
- * In the intended usage of this program, the stdout is redirected to .*.cmd
- * files. The return value of printf() must be checked to catch any error,
- * e.g. "No space left on device".
- */
-static void xprintf(const char *format, ...)
-{
-	va_list ap;
-	int ret;
-
-	va_start(ap, format);
-	ret = vprintf(format, ap);
-	if (ret < 0) {
-		perror("fixdep");
-		exit(1);
-	}
-	va_end(ap);
-}
-
 struct item {
 	struct item	*next;
 	unsigned int	len;
@@ -189,7 +170,7 @@ static void use_config(const char *m, int slen)
 
 	define_config(m, slen, hash);
 	/* Print out a dependency path from a symbol name. */
-	xprintf("    $(wildcard include/config/%.*s) \\\n", slen, m);
+	printf("    $(wildcard include/config/%.*s) \\\n", slen, m);
 }
 
 /* test if s ends in sub */
@@ -318,13 +299,13 @@ static void parse_dep_file(char *m, const char *target)
 				 */
 				if (!saw_any_target) {
 					saw_any_target = 1;
-					xprintf("source_%s := %s\n\n",
-						target, m);
-					xprintf("deps_%s := \\\n", target);
+					printf("source_%s := %s\n\n",
+					       target, m);
+					printf("deps_%s := \\\n", target);
 				}
 				is_first_dep = 0;
 			} else {
-				xprintf("  %s \\\n", m);
+				printf("  %s \\\n", m);
 			}
 
 			buf = read_file(m);
@@ -347,8 +328,8 @@ static void parse_dep_file(char *m, const char *target)
 		exit(1);
 	}
 
-	xprintf("\n%s: $(deps_%s)\n\n", target, target);
-	xprintf("$(deps_%s):\n", target);
+	printf("\n%s: $(deps_%s)\n\n", target, target);
+	printf("$(deps_%s):\n", target);
 }
 
 int main(int argc, char *argv[])
@@ -363,11 +344,22 @@ int main(int argc, char *argv[])
 	target = argv[2];
 	cmdline = argv[3];
 
-	xprintf("cmd_%s := %s\n\n", target, cmdline);
+	printf("cmd_%s := %s\n\n", target, cmdline);
 
 	buf = read_file(depfile);
 	parse_dep_file(buf, target);
 	free(buf);
 
+	fflush(stdout);
+
+	/*
+	 * In the intended usage, the stdout is redirected to .*.cmd files.
+	 * Call ferror() to catch errors such as "No space left on device".
+	 */
+	if (ferror(stdout)) {
+		fprintf(stderr, "fixdep: not all data was written to the output\n");
+		exit(1);
+	}
+
 	return 0;
 }
Signature · 1 line
2.32.0

A message that is not in the archive (sent off-list or not yet received): CAKwvOd=WjnHSHKLVRJifHxV2tyDsLTkek80NWU=do=FSHhNLug@mail.gmail.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw

David answered most of the questions from Nick.

Let me answer this question:
"Why call ferror as opposed to checking the return code of fflush? Reading the man page closer:"

When fprintf() happens to need to write data to the end device, the internal buffer is cleared anyway (even if the writing to the end device fails).
(We do not notice the failure of this because this patch is removing xprintf().)

If the buffer has been cleared by the previous fprintf() call, fflush() succeeds because there is no data in the internal buffer.

So, checking the return value of fflush() is not enough.

That's my understanding.
(I ran a small piece of test code under "ulimit -f")

So, we have two options:

[1] Check the return values in *all* the call-sites of fprintf() and fflush().

[2] Call fprintf() and fflush() or whatever without checking their return values.
And, check ferror() at the last moment.

If you are really sure that all the return values are checked, you can go with [1], then you do not need to call ferror(), but it is generally less fragile than [2].

Quoted 191 lines · David Laight

Someone send HTML mail – outlook is broken – only lets you top post :-(

The return value from fprintf() is normally the number of bytes written to

the internal buffer (8k in glibc?)

Only if the buffer is full and an actual write() is done do you get any indication of an error.

So you can use the error return from fprintf() to terminate a loop – but it usually

just isn’t worth the effort.

The error status returned by ferror() is ‘sticky’, so you need only check once.

But you need to check before fclose().

Since fclose() has to write out the buffer – that write can also fail.

I’m not sure whether fclose() returns and error in that case, but adding fflush()

makes the coding easier.

So if you have lots of fprintf() adding data to a file (which is often the case)

almost all of them always succeed – even if the disk is full.

Adding the error paths that can never really happen just makes the

code harder to read and clutters things up.

David

From: Nick Desaulniers <[email protected]>
Sent: 28 February 2022 23:01
To: Masahiro Yamada <[email protected]>
Cc: Linux Kbuild mailing list <[email protected]>; David Laight <[email protected]>; LKML <[email protected]>; Michal Marek <[email protected]>
Subject: Re: [PATCH v2] fixdep: use fflush() and ferror() to ensure successful write to files

Quoted 3 lines · Masahiro Yamada

Checking the return value of (v)printf does not ensure the successful write to the .cmd file.

Masahiro,
Apologies for my delay reviewing; I was on vacation last week.

Do you have more context for why this change is necessary? Perhaps you might describe further in the commit message the use case you're trying to support?

Reading the man pages for vprintf(3), fflush(3), and ferror(3), I'm curious why checking the return value of ferror(3) after not doing so for vprintf and fflush is preferred?

Why not simply unconditionally add a call to fflush while leaving the existing return code checking on vprintf?

Quoted 26 lines

Call fflush() and ferror() to make sure that everything has been written to the file.

---

Changes in v2:
- add error message

 scripts/basic/fixdep.c | 46 +++++++++++++++++-------------------------
 1 file changed, 19 insertions(+), 27 deletions(-)
scripts/basic/fixdep.c +0 -5
index 44e887cff49b..2328f9a641da 100644
@@ -105,25 +105,6 @@ static void usage(void)
        exit(1);
 }
 
-/*
- * In the intended usage of this program, the stdout is redirected to .*.cmd
- * files. The return value of printf() must be checked to catch any error,
- * e.g. "No space left on device".
- */

scripts/basic/fixdep.c static void usage(void) · line 122

Quoted 10 lines
scripts/basic/fixdep.c +0 -10
-static void xprintf(const char *format, ...)
-{
-       va_list ap;
-       int ret;
-
-       va_start(ap, format);
-       ret = vprintf(format, ap);
-       if (ret < 0) {
-               perror("fixdep");
-               exit(1);

Wouldn't the existing approach abort sooner if there was an error encountered?

Quoted 53 lines
scripts/basic/fixdep.c +8 -12
-       }
-       va_end(ap);
-}
-
 struct item {
        struct item     *next;
        unsigned int    len;
@@ -189,7 +170,7 @@ static void use_config(const char *m, int slen)
 
        define_config(m, slen, hash);
        /* Print out a dependency path from a symbol name. */
-       xprintf("    $(wildcard include/config/%.*s) \\\n", slen, m);
+       printf("    $(wildcard include/config/%.*s) \\\n", slen, m);
 }
 
 /* test if s ends in sub */
@@ -318,13 +299,13 @@ static void parse_dep_file(char *m, const char *target)
                                 */
                                if (!saw_any_target) {
                                        saw_any_target = 1;
-                                       xprintf("source_%s := %s\n\n",
-                                               target, m);
-                                       xprintf("deps_%s := \\\n", target);
+                                       printf("source_%s := %s\n\n",
+                                              target, m);
+                                       printf("deps_%s := \\\n", target);
                                }
                                is_first_dep = 0;
                        } else {
-                               xprintf("  %s \\\n", m);
+                               printf("  %s \\\n", m);
                        }
 
                        buf = read_file(m);
@@ -347,8 +328,8 @@ static void parse_dep_file(char *m, const char *target)
                exit(1);
        }
 
-       xprintf("\n%s: $(deps_%s)\n\n", target, target);
-       xprintf("$(deps_%s):\n", target);
+       printf("\n%s: $(deps_%s)\n\n", target, target);
+       printf("$(deps_%s):\n", target);
 }
 
 int main(int argc, char *argv[])
@@ -363,11 +344,22 @@ int main(int argc, char *argv[])
        target = argv[2];
        cmdline = argv[3];
 
-       xprintf("cmd_%s := %s\n\n", target, cmdline);
+       printf("cmd_%s := %s\n\n", target, cmdline);
 
        buf = read_file(depfile);

scripts/basic/fixdep.c int main(int argc, char *argv[]) · line 359

Quoted 10 lines
scripts/basic/fixdep.c +7 -0
        parse_dep_file(buf, target);
        free(buf);
 
+       fflush(stdout);
+
+       /*
+        * In the intended usage, the stdout is redirected to .*.cmd files.
+        * Call ferror() to catch errors such as "No space left on device".
+        */
+       if (ferror(stdout)) {

Why call ferror as opposed to checking the return code of fflush? Reading the man page closer:

The function feof() tests the end-of-file indicator for the stream pointed to by stream, returning nonzero if it is set. The end-of-file indicator can be cleared only by the function clearerr().

The function ferror() tests the error indicator for the stream pointed to by stream, returning nonzero if it is set. The error indicator can be reset only by the clearerr() function.

Does that imply that "the end-of-file indicator" is distinct from "the error indicator?"

Quoted 9 lines
scripts/basic/fixdep.c +4 -1
+               fprintf(stderr, "fixdep: not all data was written to the output\n");
+               exit(1);
+       }
+
        return 0;
 }
--

2.32.0

Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

P Please consider the environment and don't print this e-mail unless you really need to

Signature · 2 lines
Best Regards
Masahiro Yamada
#4 lore raw
Quoted 38 lines · Masahiro Yamada

David answered most of the questions from Nick.

Let me answer this question:
"Why call ferror as opposed to checking the return code of fflush? Reading the man page closer:"

When fprintf() happens to need to write data to the end device, the internal buffer is cleared anyway (even if the writing to the end device fails).
(We do not notice the failure of this because this patch is removing xprintf().)

If the buffer has been cleared by the previous fprintf() call, fflush() succeeds because there is no data in the internal buffer.

So, checking the return value of fflush() is not enough.

That's my understanding.
(I ran a small piece of test code under "ulimit -f")

So, we have two options:

[1] Check the return values in *all* the call-sites of fprintf() and fflush().

[2] Call fprintf() and fflush() or whatever without checking their return values.
And, check ferror() at the last moment.

Quoted 3 lines

If you are really sure that all the return values are checked, you can go with [1], then you do not need to call ferror(), but it is generally less fragile than [2].

Sure, doesn't matter much to me; was more curious "why."

Quoted 204 lines · David Laight
Quoted 187 lines

Someone send HTML mail – outlook is broken – only lets you top post :-(

The return value from fprintf() is normally the number of bytes written to

the internal buffer (8k in glibc?)

Only if the buffer is full and an actual write() is done do you get any indication of an error.

So you can use the error return from fprintf() to terminate a loop – but it usually

just isn’t worth the effort.

The error status returned by ferror() is ‘sticky’, so you need only check once.

But you need to check before fclose().

Since fclose() has to write out the buffer – that write can also fail.

I’m not sure whether fclose() returns and error in that case, but adding fflush()

makes the coding easier.

So if you have lots of fprintf() adding data to a file (which is often the case)

almost all of them always succeed – even if the disk is full.

Adding the error paths that can never really happen just makes the

code harder to read and clutters things up.

David

From: Nick Desaulniers <[email protected]>
Sent: 28 February 2022 23:01
To: Masahiro Yamada <[email protected]>
Cc: Linux Kbuild mailing list <[email protected]>; David Laight <[email protected]>; LKML <[email protected]>; Michal Marek <[email protected]>
Subject: Re: [PATCH v2] fixdep: use fflush() and ferror() to ensure successful write to files

Quoted 3 lines · Masahiro Yamada

Checking the return value of (v)printf does not ensure the successful write to the .cmd file.

Masahiro,
Apologies for my delay reviewing; I was on vacation last week.

Do you have more context for why this change is necessary? Perhaps you might describe further in the commit message the use case you're trying to support?

Reading the man pages for vprintf(3), fflush(3), and ferror(3), I'm curious why checking the return value of ferror(3) after not doing so for vprintf and fflush is preferred?

Why not simply unconditionally add a call to fflush while leaving the existing return code checking on vprintf?

Quoted 26 lines

Call fflush() and ferror() to make sure that everything has been written to the file.

---

Changes in v2:
- add error message

 scripts/basic/fixdep.c | 46 +++++++++++++++++-------------------------
 1 file changed, 19 insertions(+), 27 deletions(-)
scripts/basic/fixdep.c +0 -5
index 44e887cff49b..2328f9a641da 100644
@@ -105,25 +105,6 @@ static void usage(void)
        exit(1);
 }
 
-/*
- * In the intended usage of this program, the stdout is redirected to .*.cmd
- * files. The return value of printf() must be checked to catch any error,
- * e.g. "No space left on device".
- */

scripts/basic/fixdep.c static void usage(void) · line 122

Quoted 10 lines
scripts/basic/fixdep.c +0 -10
-static void xprintf(const char *format, ...)
-{
-       va_list ap;
-       int ret;
-
-       va_start(ap, format);
-       ret = vprintf(format, ap);
-       if (ret < 0) {
-               perror("fixdep");
-               exit(1);

Wouldn't the existing approach abort sooner if there was an error encountered?

Quoted 53 lines
scripts/basic/fixdep.c +8 -12
-       }
-       va_end(ap);
-}
-
 struct item {
        struct item     *next;
        unsigned int    len;
@@ -189,7 +170,7 @@ static void use_config(const char *m, int slen)
 
        define_config(m, slen, hash);
        /* Print out a dependency path from a symbol name. */
-       xprintf("    $(wildcard include/config/%.*s) \\\n", slen, m);
+       printf("    $(wildcard include/config/%.*s) \\\n", slen, m);
 }
 
 /* test if s ends in sub */
@@ -318,13 +299,13 @@ static void parse_dep_file(char *m, const char *target)
                                 */
                                if (!saw_any_target) {
                                        saw_any_target = 1;
-                                       xprintf("source_%s := %s\n\n",
-                                               target, m);
-                                       xprintf("deps_%s := \\\n", target);
+                                       printf("source_%s := %s\n\n",
+                                              target, m);
+                                       printf("deps_%s := \\\n", target);
                                }
                                is_first_dep = 0;
                        } else {
-                               xprintf("  %s \\\n", m);
+                               printf("  %s \\\n", m);
                        }
 
                        buf = read_file(m);
@@ -347,8 +328,8 @@ static void parse_dep_file(char *m, const char *target)
                exit(1);
        }
 
-       xprintf("\n%s: $(deps_%s)\n\n", target, target);
-       xprintf("$(deps_%s):\n", target);
+       printf("\n%s: $(deps_%s)\n\n", target, target);
+       printf("$(deps_%s):\n", target);
 }
 
 int main(int argc, char *argv[])
@@ -363,11 +344,22 @@ int main(int argc, char *argv[])
        target = argv[2];
        cmdline = argv[3];
 
-       xprintf("cmd_%s := %s\n\n", target, cmdline);
+       printf("cmd_%s := %s\n\n", target, cmdline);
 
        buf = read_file(depfile);

scripts/basic/fixdep.c int main(int argc, char *argv[]) · line 359

Quoted 10 lines
scripts/basic/fixdep.c +7 -0
        parse_dep_file(buf, target);
        free(buf);
 
+       fflush(stdout);
+
+       /*
+        * In the intended usage, the stdout is redirected to .*.cmd files.
+        * Call ferror() to catch errors such as "No space left on device".
+        */
+       if (ferror(stdout)) {

Why call ferror as opposed to checking the return code of fflush? Reading the man page closer:

The function feof() tests the end-of-file indicator for the stream pointed to by stream, returning nonzero if it is set. The end-of-file indicator can be cleared only by the function clearerr().

The function ferror() tests the error indicator for the stream pointed to by stream, returning nonzero if it is set. The error indicator can be reset only by the clearerr() function.

Does that imply that "the end-of-file indicator" is distinct from "the error indicator?"

Quoted 9 lines
scripts/basic/fixdep.c +4 -1
+               fprintf(stderr, "fixdep: not all data was written to the output\n");
+               exit(1);
+       }
+
        return 0;
 }
--

2.32.0

Quoted 4 lines

Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

P Please consider the environment and don't print this e-mail unless you really need to

--
Best Regards
Masahiro Yamada

Signature · 2 lines
Thanks,
~Nick Desaulniers
#5 lore raw
In reply to a message not in the archive
Quoted 17 lines · David Laight

Someone send HTML mail – outlook is broken – only lets you top post :-(

The return value from fprintf() is normally the number of bytes written to

the internal buffer (8k in glibc?)

Only if the buffer is full and an actual write() is done do you get any indication of an error.

So you can use the error return from fprintf() to terminate a loop – but it usually

just isn’t worth the effort.

The error status returned by ferror() is ‘sticky’, so you need only check once.

Quoted 7 lines

But you need to check before fclose().

Since fclose() has to write out the buffer – that write can also fail.

I’m not sure whether fclose() returns and error in that case, but adding fflush()

makes the coding easier.

I just checked this.

fclose() returns -1 if it fails to flush the buffer.

[ test code ]
#include <stdio.h>
int main(void)

{
        char buf[2049];
        int ret, i;

        for (i = 0; i < 2048; i++)
                buf[i] = 'a';

        buf[2048] = 0;

        ret = printf("%s", buf);
        fprintf(stderr, "printf() returned: %d\n", ret);

        ret = fclose(stdout);
        fprintf(stderr, "fclose() returned %d\n", ret);

        return 0;
}

I tested this on Debian buster.

I created a very small partition with 1K size,
then write data to that partition.

root@buster:~# lsblk  /dev/vdb1
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vdb1 254:17   0   1K  0 part
root@buster:~# ./a.out  > /dev/vdb1
printf() returned: 2048
fclose() returned -1

The buffer size seems 4k
as far as I tested on Debian.

Signature · 2 lines
Best Regards
Masahiro Yamada
#6 lore raw

From: Masahiro Yamada

Quoted 18 lines

Sent: 01 March 2022 09:06

Quoted 15 lines · David Laight

The return value from fprintf() is normally the number of bytes written to the internal buffer (8k in glibc?)

Only if the buffer is full and an actual write() is done do you get any indication of an error.

So you can use the error return from fprintf() to terminate a loop – but it usually just isn’t worth the effort.

The error status returned by ferror() is ‘sticky’, so you need only check once.

But you need to check before fclose().

Since fclose() has to write out the buffer – that write can also fail.

Quoted 7 lines
Quoted 2 lines

I’m not sure whether fclose() returns and error in that case, but adding fflush() makes the coding easier.

I just checked this.

fclose() returns -1 if it fails to flush the buffer.

But you still need to check ferror() before it. So you might as well do fflush(); if (ferror()); fclose();

Then there is only one error exit path.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)