fixdep: use fflush() and ferror() to ensure successful write to files
PATCH v2 1/1 fixdep: use fflush() and ferror() to ensure successful write to files
Checking the return value of (v)printf does not ensure the successful write to the .cmd file.
Call fflush() and ferror() to make sure that everything has been written to the file.
---
Changes in v2:
- add error message
scripts/basic/fixdep.c | 46 +++++++++++++++++------------------------- 1 file changed, 19 insertions(+), 27 deletions(-)
scripts/basic/fixdep.c +19 -27
@@ -105,25 +105,6 @@ static void usage(void)exit (1 ); } -/* -* In the intended usage of this program, the stdout is redirected to .*.cmd -* files. The return value of printf() must be checked to catch any error, -* e.g. "No space left on device". -*/ -static void xprintf (const char *format, ...) -{ -va_list ap; -int ret; - -va_start (ap, format); - ret =vprintf (format, ap); -if (ret <0 ) { -perror ("fixdep" ); -exit (1 ); - } -va_end (ap); -} -struct item {struct item *next;unsigned int len;
@@ -189,7 +170,7 @@ static void use_config(const char *m, int slen)define_config (m, slen, hash);/* Print out a dependency path from a symbol name. */ -xprintf (" $(wildcard include/config/%.*s) \\\n" , slen, m); +printf (" $(wildcard include/config/%.*s) \\\n" , slen, m); }/* test if s ends in sub */
@@ -318,13 +299,13 @@ static void parse_dep_file(char *m, const char *target)*/ if (!saw_any_target) { saw_any_target =1 ; -xprintf ("source_%s := %s\n\n" , - target, m); -xprintf ("deps_%s := \\\n" , target); +printf ("source_%s := %s\n\n" , + target, m); +printf ("deps_%s := \\\n" , target); } is_first_dep =0 ; }else { -xprintf (" %s \\\n" , m); +printf (" %s \\\n" , m); } buf =read_file (m);
@@ -347,8 +328,8 @@ static void parse_dep_file(char *m, const char *target)exit (1 ); } -xprintf ("\n%s: $(deps_%s)\n\n" , target, target); -xprintf ("$(deps_%s):\n" , target); +printf ("\n%s: $(deps_%s)\n\n" , target, target); +printf ("$(deps_%s):\n" , target); }int main (int argc,char *argv[])
@@ -363,11 +344,22 @@ int main(int argc, char *argv[]) target = argv[2 ]; cmdline = argv[3 ]; -xprintf ("cmd_%s := %s\n\n" , target, cmdline); +printf ("cmd_%s := %s\n\n" , target, cmdline); buf =read_file (depfile);parse_dep_file (buf, target);free (buf); +fflush (stdout); + +/* +* In the intended usage, the stdout is redirected to .*.cmd files. +* Call ferror() to catch errors such as "No space left on device". +*/ +if (ferror (stdout)) { +fprintf (stderr,"fixdep: not all data was written to the output\n" ); +exit (1 ); + } +return 0 ; }
Signature · 1 line
2.32.0
A message that is not in the archive (sent off-list or not yet received): CAKwvOd=WjnHSHKLVRJifHxV2tyDsLTkek80NWU=do=FSHhNLug@mail.gmail.com
A message that is not in the archive (sent off-list or not yet received): [email protected]
David answered most of the questions from Nick.
Let me answer this question:
"Why call ferror as opposed to checking the return code of fflush? Reading the man page closer:"
When fprintf() happens to need to write data to the end device, the internal buffer is cleared anyway (even if the writing to the end device fails).
(We do not notice the failure of this because this patch is removing xprintf().)
If the buffer has been cleared by the previous fprintf() call, fflush() succeeds because there is no data in the internal buffer.
So, checking the return value of fflush() is not enough.
That's my understanding.
(I ran a small piece of test code under "ulimit -f")
So, we have two options:
[1] Check the return values in *all* the call-sites of fprintf() and fflush().
[2] Call fprintf() and fflush() or whatever without checking their return values.
And, check ferror() at the last moment.
If you are really sure that all the return values are checked, you can go with [1], then you do not need to call ferror(), but it is generally less fragile than [2].
Quoted 191 lines · David Laight
Someone send HTML mail – outlook is broken – only lets you top post :-(
The return value from
fprintf()is normally the number of bytes written tothe internal buffer (8k in glibc?)
Only if the buffer is full and an actual
write()is done do you get any indication of an error.So you can use the error return from
fprintf()to terminate a loop – but it usuallyjust isn’t worth the effort.
The error status returned by
ferror()is ‘sticky’, so you need only check once.But you need to check before
fclose().Since
fclose()has to write out the buffer – that write can also fail.I’m not sure whether
fclose()returns and error in that case, but addingfflush()makes the coding easier.
So if you have lots of
fprintf()adding data to a file (which is often the case)almost all of them always succeed – even if the disk is full.
Adding the error paths that can never really happen just makes the
code harder to read and clutters things up.
David
From: Nick Desaulniers <[email protected]>
Sent: 28 February 2022 23:01
To: Masahiro Yamada <[email protected]>
Cc: LinuxKbuildmailing list <[email protected]>; David Laight <[email protected]>; LKML <[email protected]>; Michal Marek <[email protected]>
Subject: Re: [PATCH v2] fixdep: usefflush()andferror()to ensure successful write to filesQuoted 3 lines · Masahiro Yamada
Checking the return value of (v)printf does not ensure the successful write to the .cmd file.
Masahiro,
Apologies for my delay reviewing; I was on vacation last week.Do you have more context for why this change is necessary? Perhaps you might describe further in the commit message the use case you're trying to support?
Reading the man pages for vprintf(3), fflush(3), and ferror(3), I'm curious why checking the return value of ferror(3) after not doing so for
vprintfandfflushis preferred?Why not simply unconditionally add a call to fflush while leaving the existing return code checking on vprintf?
Quoted 26 lines
Call
fflush()andferror()to make sure that everything has been written to the file.Signed-off-by Masahiro Yamada <[email protected]>---
Changes in v2:
- add error messagescripts/basic/fixdep.c | 46 +++++++++++++++++------------------------- 1 file changed, 19 insertions(+), 27 deletions(-)scripts/basic/fixdep.c +0
-5@@ -105,25 +105,6 @@ static void usage(void)exit (1 ); } -/* -* In the intended usage of this program, the stdout is redirected to .*.cmd -* files. The return value of printf() must be checked to catch any error, -* e.g. "No space left on device". -*/ scripts/basic/fixdep.c static void usage(void) · line 122
Quoted 10 lines
scripts/basic/fixdep.c +0
-10-static void xprintf (const char *format, ...) -{ -va_list ap; -int ret; - -va_start (ap, format); - ret =vprintf (format, ap); -if (ret <0 ) { -perror ("fixdep" ); -exit (1 );Wouldn't the existing approach abort sooner if there was an error encountered?
Quoted 53 lines
scripts/basic/fixdep.c +8
-12- } -va_end (ap); -} -struct item {struct item *next;unsigned int len;@@ -189,7 +170,7 @@ static void use_config(const char *m, int slen)define_config (m, slen, hash);/* Print out a dependency path from a symbol name. */ -xprintf (" $(wildcard include/config/%.*s) \\\n" , slen, m); +printf (" $(wildcard include/config/%.*s) \\\n" , slen, m); }/* test if s ends in sub */ @@ -318,13 +299,13 @@ static void parse_dep_file(char *m, const char *target)*/ if (!saw_any_target) { saw_any_target =1 ; -xprintf ("source_%s := %s\n\n" , - target, m); -xprintf ("deps_%s := \\\n" , target); +printf ("source_%s := %s\n\n" , + target, m); +printf ("deps_%s := \\\n" , target); } is_first_dep =0 ; }else { -xprintf (" %s \\\n" , m); +printf (" %s \\\n" , m); } buf =read_file (m);@@ -347,8 +328,8 @@ static void parse_dep_file(char *m, const char *target)exit (1 ); } -xprintf ("\n%s: $(deps_%s)\n\n" , target, target); -xprintf ("$(deps_%s):\n" , target); +printf ("\n%s: $(deps_%s)\n\n" , target, target); +printf ("$(deps_%s):\n" , target); }int main (int argc,char *argv[])@@ -363,11 +344,22 @@ int main(int argc, char *argv[]) target = argv[2 ]; cmdline = argv[3 ]; -xprintf ("cmd_%s := %s\n\n" , target, cmdline); +printf ("cmd_%s := %s\n\n" , target, cmdline); buf =read_file (depfile);scripts/basic/fixdep.c int main(int argc, char *argv[]) · line 359
Quoted 10 lines
scripts/basic/fixdep.c +7
-0parse_dep_file (buf, target);free (buf); +fflush (stdout); + +/* +* In the intended usage, the stdout is redirected to .*.cmd files. +* Call ferror() to catch errors such as "No space left on device". +*/ +if (ferror (stdout)) {Why call ferror as opposed to checking the return code of fflush? Reading the man page closer:
The function
feof()tests the end-of-file indicator for the stream pointed to by stream, returning nonzero if it is set. The end-of-file indicator can be cleared only by the functionclearerr().The function
ferror()tests the error indicator for the stream pointed to by stream, returning nonzero if it is set. The error indicator can be reset only by theclearerr()function.Does that imply that "the end-of-file indicator" is distinct from "the error indicator?"
Quoted 9 lines
scripts/basic/fixdep.c +4
-1+fprintf (stderr,"fixdep: not all data was written to the output\n" ); +exit (1 ); + } +return 0 ; } --2.32.0
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
P Please consider the environment and don't print this e-mail unless you really need to
Signature · 2 lines
Best Regards Masahiro Yamada
Quoted 38 lines · Masahiro Yamada
David answered most of the questions from Nick.
Let me answer this question:
"Why call ferror as opposed to checking the return code of fflush? Reading the man page closer:"When
fprintf()happens to need to write data to the end device, the internal buffer is cleared anyway (even if the writing to the end device fails).
(We do not notice the failure of this because this patch is removingxprintf().)If the buffer has been cleared by the previous
fprintf()call,fflush()succeeds because there is no data in the internal buffer.So, checking the return value of
fflush()is not enough.That's my understanding.
(I ran a small piece of test code under "ulimit -f")So, we have two options:
[1] Check the return values in *all* the call-sites of
fprintf()andfflush().[2] Call
fprintf()andfflush()or whatever without checking their return values.
And, checkferror()at the last moment.
Quoted 3 lines
If you are really sure that all the return values are checked, you can go with [1], then you do not need to call
ferror(), but it is generally less fragile than [2].
Sure, doesn't matter much to me; was more curious "why."
Quoted 204 lines · David Laight
Quoted 187 lines
Someone send HTML mail – outlook is broken – only lets you top post :-(
The return value from
fprintf()is normally the number of bytes written tothe internal buffer (8k in glibc?)
Only if the buffer is full and an actual
write()is done do you get any indication of an error.So you can use the error return from
fprintf()to terminate a loop – but it usuallyjust isn’t worth the effort.
The error status returned by
ferror()is ‘sticky’, so you need only check once.But you need to check before
fclose().Since
fclose()has to write out the buffer – that write can also fail.I’m not sure whether
fclose()returns and error in that case, but addingfflush()makes the coding easier.
So if you have lots of
fprintf()adding data to a file (which is often the case)almost all of them always succeed – even if the disk is full.
Adding the error paths that can never really happen just makes the
code harder to read and clutters things up.
David
From: Nick Desaulniers <[email protected]>
Sent: 28 February 2022 23:01
To: Masahiro Yamada <[email protected]>
Cc: LinuxKbuildmailing list <[email protected]>; David Laight <[email protected]>; LKML <[email protected]>; Michal Marek <[email protected]>
Subject: Re: [PATCH v2] fixdep: usefflush()andferror()to ensure successful write to filesQuoted 3 lines · Masahiro Yamada
Checking the return value of (v)printf does not ensure the successful write to the .cmd file.
Masahiro,
Apologies for my delay reviewing; I was on vacation last week.Do you have more context for why this change is necessary? Perhaps you might describe further in the commit message the use case you're trying to support?
Reading the man pages for vprintf(3), fflush(3), and ferror(3), I'm curious why checking the return value of ferror(3) after not doing so for
vprintfandfflushis preferred?Why not simply unconditionally add a call to fflush while leaving the existing return code checking on vprintf?
Quoted 26 lines
Call
fflush()andferror()to make sure that everything has been written to the file.Signed-off-by Masahiro Yamada <[email protected]>---
Changes in v2:
- add error messagescripts/basic/fixdep.c | 46 +++++++++++++++++------------------------- 1 file changed, 19 insertions(+), 27 deletions(-)scripts/basic/fixdep.c +0
-5@@ -105,25 +105,6 @@ static void usage(void)exit (1 ); } -/* -* In the intended usage of this program, the stdout is redirected to .*.cmd -* files. The return value of printf() must be checked to catch any error, -* e.g. "No space left on device". -*/ scripts/basic/fixdep.c static void usage(void) · line 122
Quoted 10 lines
scripts/basic/fixdep.c +0
-10-static void xprintf (const char *format, ...) -{ -va_list ap; -int ret; - -va_start (ap, format); - ret =vprintf (format, ap); -if (ret <0 ) { -perror ("fixdep" ); -exit (1 );Wouldn't the existing approach abort sooner if there was an error encountered?
Quoted 53 lines
scripts/basic/fixdep.c +8
-12- } -va_end (ap); -} -struct item {struct item *next;unsigned int len;@@ -189,7 +170,7 @@ static void use_config(const char *m, int slen)define_config (m, slen, hash);/* Print out a dependency path from a symbol name. */ -xprintf (" $(wildcard include/config/%.*s) \\\n" , slen, m); +printf (" $(wildcard include/config/%.*s) \\\n" , slen, m); }/* test if s ends in sub */ @@ -318,13 +299,13 @@ static void parse_dep_file(char *m, const char *target)*/ if (!saw_any_target) { saw_any_target =1 ; -xprintf ("source_%s := %s\n\n" , - target, m); -xprintf ("deps_%s := \\\n" , target); +printf ("source_%s := %s\n\n" , + target, m); +printf ("deps_%s := \\\n" , target); } is_first_dep =0 ; }else { -xprintf (" %s \\\n" , m); +printf (" %s \\\n" , m); } buf =read_file (m);@@ -347,8 +328,8 @@ static void parse_dep_file(char *m, const char *target)exit (1 ); } -xprintf ("\n%s: $(deps_%s)\n\n" , target, target); -xprintf ("$(deps_%s):\n" , target); +printf ("\n%s: $(deps_%s)\n\n" , target, target); +printf ("$(deps_%s):\n" , target); }int main (int argc,char *argv[])@@ -363,11 +344,22 @@ int main(int argc, char *argv[]) target = argv[2 ]; cmdline = argv[3 ]; -xprintf ("cmd_%s := %s\n\n" , target, cmdline); +printf ("cmd_%s := %s\n\n" , target, cmdline); buf =read_file (depfile);scripts/basic/fixdep.c int main(int argc, char *argv[]) · line 359
Quoted 10 lines
scripts/basic/fixdep.c +7
-0parse_dep_file (buf, target);free (buf); +fflush (stdout); + +/* +* In the intended usage, the stdout is redirected to .*.cmd files. +* Call ferror() to catch errors such as "No space left on device". +*/ +if (ferror (stdout)) {Why call ferror as opposed to checking the return code of fflush? Reading the man page closer:
The function
feof()tests the end-of-file indicator for the stream pointed to by stream, returning nonzero if it is set. The end-of-file indicator can be cleared only by the functionclearerr().The function
ferror()tests the error indicator for the stream pointed to by stream, returning nonzero if it is set. The error indicator can be reset only by theclearerr()function.Does that imply that "the end-of-file indicator" is distinct from "the error indicator?"
Quoted 9 lines
scripts/basic/fixdep.c +4
-1+fprintf (stderr,"fixdep: not all data was written to the output\n" ); +exit (1 ); + } +return 0 ; } --2.32.0
Quoted 4 lines
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
P Please consider the environment and don't print this e-mail unless you really need to
--
Best Regards
Masahiro Yamada
Signature · 2 lines
Thanks, ~Nick Desaulniers
Quoted 17 lines · David Laight
Someone send HTML mail – outlook is broken – only lets you top post :-(
The return value from
fprintf()is normally the number of bytes written tothe internal buffer (8k in glibc?)
Only if the buffer is full and an actual
write()is done do you get any indication of an error.So you can use the error return from
fprintf()to terminate a loop – but it usuallyjust isn’t worth the effort.
The error status returned by
ferror()is ‘sticky’, so you need only check once.
Quoted 7 lines
But you need to check before
fclose().Since
fclose()has to write out the buffer – that write can also fail.I’m not sure whether
fclose()returns and error in that case, but addingfflush()makes the coding easier.
I just checked this.
fclose() returns -1 if it fails to flush the buffer.
[ test code ]
#include <stdio.h>
int main(void)
{
char buf[2049];
int ret, i;
for (i = 0; i < 2048; i++)
buf[i] = 'a';
buf[2048] = 0;
ret = printf("%s", buf);
fprintf(stderr, "printf() returned: %d\n", ret);
ret = fclose(stdout);
fprintf(stderr, "fclose() returned %d\n", ret);
return 0;
}I tested this on Debian buster.
I created a very small partition with 1K size,
then write data to that partition.
root@buster:~# lsblk /dev/vdb1 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vdb1 254:17 0 1K 0 part root@buster:~# ./a.out > /dev/vdb1 printf() returned: 2048 fclose() returned -1
The buffer size seems 4k
as far as I tested on Debian.
Signature · 2 lines
Best Regards Masahiro Yamada
From: Masahiro Yamada
Quoted 18 lines
Sent: 01 March 2022 09:06
Quoted 15 lines · David Laight
The return value from
fprintf()is normally the number of bytes written to the internal buffer (8k in glibc?)Only if the buffer is full and an actual
write()is done do you get any indication of an error.So you can use the error return from
fprintf()to terminate a loop – but it usually just isn’t worth the effort.The error status returned by
ferror()is ‘sticky’, so you need only check once.But you need to check before
fclose().Since
fclose()has to write out the buffer – that write can also fail.
Quoted 7 lines
Quoted 2 lines
I’m not sure whether
fclose()returns and error in that case, but addingfflush()makes the coding easier.I just checked this.
fclose()returns -1 if it fails to flush the buffer.
But you still need to check ferror() before it. So you might as well do fflush(); if (ferror()); fclose();
Then there is only one error exit path.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)