Skip to contentSkip to conversation

libfrog: make cmn_err() emit each message atomically to avoid torn output

RFC PATCH Superseded Avinesh Kumar 3 participants 7 messages Jul 15 – Jul 20

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 libfrog: make cmn_err() emit each message atomically to avoid torn output

From: Avinesh Kumar <[email protected]>

fstests xfs/033 fails sporadically with a spurious blank line in the xfs_repair output:

- output mismatch (see /opt/xfstests/results//xfs/033.out.bad)

    --- tests/xfs/033.out	2026-06-24 15:52:51.000000000 -0400
    +++ /opt/xfstests/results//xfs/033.out.bad	2026-07-14 18:54:46.582495041 -0400
    @@ -103,6 +103,7 @@
     Phase 3 - for each AG...
             - scan and clear agi unlinked lists...
             - process known inodes and perform inode discovery...
    +
     bad magic number 0xffff on inode INO
     bad version number 0xffffffff on inode INO
     inode identifier 18446744073709551615 mismatch on inode INO

Root cause is in cmn_err() (libfrog/util.c), which emits a message and its newline as two separate writes to unbuffered stderr. xfs_repair's threads all share stderr. If one is preempted between the two writes, another thread's line lands in between:

(snips from cat -A 033.raw) -
Phase 3 - for each AG...$
- scan and clear agi unlinked lists...$
- process known inodes and perform inode discovery...$
Metadata corruption detected at 0x445dd3, xfs_inode block 0x80/0x4000 - agno = 0$
$
bad CRC for inode 128$
bad magic number 0x0 on inode 128$

which should be like -

Phase 3 - for each AG...$
- scan and clear agi unlinked lists...$
- process known inodes and perform inode discovery...$
Metadata corruption detected at 0x445dd3, xfs_inode block 0x80/0x4000$
- agno = 0$
bad CRC for inode 130$
bad magic number 0x0 on inode 130$

_filter_repair() strips the noise line it glued onto but not the lone newline, which then fails the golden diff.

Make the two writes atomic.

---

Hi, please give feedback if this should be fixed in a different way. Thanks!

 libfrog/util.c | 2 ++
 1 file changed, 2 insertions(+)
libfrog/util.c +2 -0
index 5bae5bab..3b6df917 100644
@@ -116,8 +116,10 @@ cmn_err(int level, char *fmt, ...)
 	va_list	ap;
 
 	va_start(ap, fmt);
+	flockfile(stderr);
 	vfprintf(stderr, fmt, ap);
 	fputs("\n", stderr);
+	funlockfile(stderr);
 	va_end(ap);
 }
 
Signature · 1 line
2.55.0
#1 lore raw
Quoted 60 lines · Avinesh Kumar

From: Avinesh Kumar <[email protected]>

fstests xfs/033 fails sporadically with a spurious blank line in the xfs_repair output:

- output mismatch (see /opt/xfstests/results//xfs/033.out.bad)

    --- tests/xfs/033.out	2026-06-24 15:52:51.000000000 -0400
    +++ /opt/xfstests/results//xfs/033.out.bad	2026-07-14 18:54:46.582495041 -0400
    @@ -103,6 +103,7 @@
     Phase 3 - for each AG...
             - scan and clear agi unlinked lists...
             - process known inodes and perform inode discovery...
    +
     bad magic number 0xffff on inode INO
     bad version number 0xffffffff on inode INO
     inode identifier 18446744073709551615 mismatch on inode INO

Root cause is in cmn_err() (libfrog/util.c), which emits a message and its newline as two separate writes to unbuffered stderr. xfs_repair's threads all share stderr. If one is preempted between the two writes, another thread's line lands in between:

(snips from cat -A 033.raw) -
Phase 3 - for each AG...$
- scan and clear agi unlinked lists...$
- process known inodes and perform inode discovery...$
Metadata corruption detected at 0x445dd3, xfs_inode block 0x80/0x4000 - agno = 0$
$
bad CRC for inode 128$
bad magic number 0x0 on inode 128$

which should be like -

Phase 3 - for each AG...$
- scan and clear agi unlinked lists...$
- process known inodes and perform inode discovery...$
Metadata corruption detected at 0x445dd3, xfs_inode block 0x80/0x4000$
- agno = 0$
bad CRC for inode 130$
bad magic number 0x0 on inode 130$

_filter_repair() strips the noise line it glued onto but not the lone newline, which then fails the golden diff.

Make the two writes atomic.

---

Hi, please give feedback if this should be fixed in a different way. Thanks!

 libfrog/util.c | 2 ++
 1 file changed, 2 insertions(+)
libfrog/util.c +0 -0
index 5bae5bab..3b6df917 100644

libfrog/util.c cmn_err(int level, char *fmt, ...) · line 122

Quoted 7 lines
libfrog/util.c +2 -0
@@ -116,8 +116,10 @@ cmn_err(int level, char *fmt, ...)
 	va_list	ap;
 
 	va_start(ap, fmt);
+	flockfile(stderr);
 	vfprintf(stderr, fmt, ap);
 	fputs("\n", stderr);
+	funlockfile(stderr);

I think do_error() in xfs_repair.c is going to need the same treatment but otherwise this chunk looks ok to me.

--D

Quoted 7 lines
libfrog/util.c +0 -0
 	va_end(ap);
 }
 
Signature · 1 line
2.55.0
#2 lore raw

Hello Darrick,

Quoted 63 lines · Darrick J. Wong
Quoted 62 lines · Avinesh Kumar

From: Avinesh Kumar <[email protected]>

fstests xfs/033 fails sporadically with a spurious blank line in the xfs_repair output:

- output mismatch (see /opt/xfstests/results//xfs/033.out.bad)

     --- tests/xfs/033.out	2026-06-24 15:52:51.000000000 -0400
     +++ /opt/xfstests/results//xfs/033.out.bad	2026-07-14 18:54:46.582495041 -0400
     @@ -103,6 +103,7 @@
      Phase 3 - for each AG...
              - scan and clear agi unlinked lists...
              - process known inodes and perform inode discovery...
     +
      bad magic number 0xffff on inode INO
      bad version number 0xffffffff on inode INO
      inode identifier 18446744073709551615 mismatch on inode INO

Root cause is in cmn_err() (libfrog/util.c), which emits a message and its newline as two separate writes to unbuffered stderr. xfs_repair's threads all share stderr. If one is preempted between the two writes, another thread's line lands in between:

(snips from cat -A 033.raw) -
Phase 3 - for each AG...$
- scan and clear agi unlinked lists...$
- process known inodes and perform inode discovery...$
Metadata corruption detected at 0x445dd3, xfs_inode block 0x80/0x4000 - agno = 0$
$
bad CRC for inode 128$
bad magic number 0x0 on inode 128$

which should be like -

Phase 3 - for each AG...$
- scan and clear agi unlinked lists...$
- process known inodes and perform inode discovery...$
Metadata corruption detected at 0x445dd3, xfs_inode block 0x80/0x4000$
- agno = 0$
bad CRC for inode 130$
bad magic number 0x0 on inode 130$

_filter_repair() strips the noise line it glued onto but not the lone newline, which then fails the golden diff.

Make the two writes atomic.

---

Hi, please give feedback if this should be fixed in a different way. Thanks!

  libfrog/util.c | 2 ++
  1 file changed, 2 insertions(+)
libfrog/util.c +0 -0
index 5bae5bab..3b6df917 100644
@@ -116,8 +116,10 @@ cmn_err(int level, char *fmt, ...)
  	va_list	ap;
 
Quoted 9 lines

libfrog/util.c cmn_err(int level, char *fmt, ...) · line 122

Quoted 5 lines
libfrog/util.c +2 -0
  	va_start(ap, fmt);
+	flockfile(stderr);
  	vfprintf(stderr, fmt, ap);
  	fputs("\n", stderr);
+	funlockfile(stderr);

I think do_error() in xfs_repair.c is going to need the same treatment but otherwise this chunk looks ok to me.

Thanks for the review. I can send a follow up patch for do_error(), or do you want these to go in the same patch?

Regards,
Avinesh

Quoted 9 lines

--D

Quoted 7 lines
libfrog/util.c +0 -0
  	va_end(ap);
  }
 
Signature · 1 line
2.55.0
#3 lore raw
Quoted 75 lines · Avinesh Kumar

Hello Darrick,

Quoted 56 lines · Darrick J. Wong
Quoted 55 lines · Avinesh Kumar

From: Avinesh Kumar <[email protected]>

fstests xfs/033 fails sporadically with a spurious blank line in the xfs_repair output:

- output mismatch (see /opt/xfstests/results//xfs/033.out.bad)

     --- tests/xfs/033.out	2026-06-24 15:52:51.000000000 -0400
     +++ /opt/xfstests/results//xfs/033.out.bad	2026-07-14 18:54:46.582495041 -0400
     @@ -103,6 +103,7 @@
      Phase 3 - for each AG...
              - scan and clear agi unlinked lists...
              - process known inodes and perform inode discovery...
     +
      bad magic number 0xffff on inode INO
      bad version number 0xffffffff on inode INO
      inode identifier 18446744073709551615 mismatch on inode INO

Root cause is in cmn_err() (libfrog/util.c), which emits a message and its newline as two separate writes to unbuffered stderr. xfs_repair's threads all share stderr. If one is preempted between the two writes, another thread's line lands in between:

(snips from cat -A 033.raw) -
Phase 3 - for each AG...$
- scan and clear agi unlinked lists...$
- process known inodes and perform inode discovery...$
Metadata corruption detected at 0x445dd3, xfs_inode block 0x80/0x4000 - agno = 0$
$
bad CRC for inode 128$
bad magic number 0x0 on inode 128$

which should be like -

Phase 3 - for each AG...$
- scan and clear agi unlinked lists...$
- process known inodes and perform inode discovery...$
Metadata corruption detected at 0x445dd3, xfs_inode block 0x80/0x4000$
- agno = 0$
bad CRC for inode 130$
bad magic number 0x0 on inode 130$

_filter_repair() strips the noise line it glued onto but not the lone newline, which then fails the golden diff.

Make the two writes atomic.

---

Hi, please give feedback if this should be fixed in a different way. Thanks!

  libfrog/util.c | 2 ++
  1 file changed, 2 insertions(+)
Quoted 15 lines

libfrog/util.c cmn_err(int level, char *fmt, ...) · line 121

Quoted 11 lines
libfrog/util.c +2 -0
index 5bae5bab..3b6df917 100644
@@ -116,8 +116,10 @@ cmn_err(int level, char *fmt, ...)
  	va_list	ap;
  	va_start(ap, fmt);
+	flockfile(stderr);
  	vfprintf(stderr, fmt, ap);
  	fputs("\n", stderr);
+	funlockfile(stderr);

I think do_error() in xfs_repair.c is going to need the same treatment but otherwise this chunk looks ok to me.

Quoted 2 lines

Thanks for the review. I can send a follow up patch for do_error(), or do you want these to go in the same patch?

They probably ought to be in the same patch.

--D

Quoted 13 lines

Regards,
Avinesh

Quoted 8 lines

--D

Quoted 6 lines
libfrog/util.c +0 -0
  	va_end(ap);
  }
Signature · 1 line
2.55.0
#4 lore raw

PATCH v2 1/1 libfrog: make cmn_err() emit each message atomically to avoid torn output

From: Avinesh Kumar <[email protected]>

fstests xfs/033 fails sporadically with a spurious blank line in the xfs_repair output:

- output mismatch (see /opt/xfstests/results//xfs/033.out.bad)

    --- tests/xfs/033.out	2026-06-24 15:52:51.000000000 -0400
    +++ /opt/xfstests/results//xfs/033.out.bad	2026-07-14 18:54:46.582495041 -0400
    @@ -103,6 +103,7 @@
     Phase 3 - for each AG...
             - scan and clear agi unlinked lists...
             - process known inodes and perform inode discovery...
    +
     bad magic number 0xffff on inode INO
     bad version number 0xffffffff on inode INO
     inode identifier 18446744073709551615 mismatch on inode INO

Root cause is in cmn_err() (libfrog/util.c), which emits a message and its newline as two separate writes to unbuffered stderr. xfs_repair's threads all share stderr. If one is preempted between the two writes, another thread's line lands in between:

(snips from cat -A 033.raw) -
Phase 3 - for each AG...$
- scan and clear agi unlinked lists...$
- process known inodes and perform inode discovery...$
Metadata corruption detected at 0x445dd3, xfs_inode block 0x80/0x4000 - agno = 0$
$
bad CRC for inode 128$
bad magic number 0x0 on inode 128$

which should be like -

Phase 3 - for each AG...$
- scan and clear agi unlinked lists...$
- process known inodes and perform inode discovery...$
Metadata corruption detected at 0x445dd3, xfs_inode block 0x80/0x4000$
- agno = 0$
bad CRC for inode 130$
bad magic number 0x0 on inode 130$

_filter_repair() strips the noise line it glued onto but not the lone newline, which then fails the golden diff.

Make the two writes atomic.

do_error() in xfs_repair.c also has same issue, fix that too.

---
v2: fix the do_error() in xfs_repair.c too.

 libfrog/util.c      | 2 ++
 repair/xfs_repair.c | 2 ++
 2 files changed, 4 insertions(+)
libfrog/util.c +2 -0
index 5bae5bab..3b6df917 100644
@@ -116,8 +116,10 @@ cmn_err(int level, char *fmt, ...)
 	va_list	ap;
 
 	va_start(ap, fmt);
+	flockfile(stderr);
 	vfprintf(stderr, fmt, ap);
 	fputs("\n", stderr);
+	funlockfile(stderr);
 	va_end(ap);
 }
 
repair/xfs_repair.c +2 -0
index 6b97a806..dd758ebc 100644
@@ -458,10 +458,12 @@ do_error(char const *msg, ...)
 {
 	va_list args;
 
+	flockfile(stderr);
 	fprintf(stderr, _("\nfatal error -- "));
 
 	va_start(args, msg);
 	vfprintf(stderr, msg, args);
+	funlockfile(stderr);
 	if (dumpcore)
 		abort();
 	exit(1);
Signature · 1 line
2.55.0
Reviewed-by on v2 1/1
#5 lore raw
Quoted 41 lines · Avinesh Kumar

From: Avinesh Kumar <[email protected]>

fstests xfs/033 fails sporadically with a spurious blank line in the xfs_repair output:

- output mismatch (see /opt/xfstests/results//xfs/033.out.bad)

    --- tests/xfs/033.out	2026-06-24 15:52:51.000000000 -0400
    +++ /opt/xfstests/results//xfs/033.out.bad	2026-07-14 18:54:46.582495041 -0400
    @@ -103,6 +103,7 @@
     Phase 3 - for each AG...
             - scan and clear agi unlinked lists...
             - process known inodes and perform inode discovery...
    +
     bad magic number 0xffff on inode INO
     bad version number 0xffffffff on inode INO
     inode identifier 18446744073709551615 mismatch on inode INO

Root cause is in cmn_err() (libfrog/util.c), which emits a message and its newline as two separate writes to unbuffered stderr. xfs_repair's threads all share stderr. If one is preempted between the two writes, another thread's line lands in between:

(snips from cat -A 033.raw) -
Phase 3 - for each AG...$
- scan and clear agi unlinked lists...$
- process known inodes and perform inode discovery...$
Metadata corruption detected at 0x445dd3, xfs_inode block 0x80/0x4000 - agno = 0$
$
bad CRC for inode 128$
bad magic number 0x0 on inode 128$

which should be like -

Phase 3 - for each AG...$
- scan and clear agi unlinked lists...$
- process known inodes and perform inode discovery...$
Metadata corruption detected at 0x445dd3, xfs_inode block 0x80/0x4000$
- agno = 0$
bad CRC for inode 130$
bad magic number 0x0 on inode 130$

Quoted 8 lines

_filter_repair() strips the noise line it glued onto but not the lone newline, which then fails the golden diff.

Make the two writes atomic.

do_error() in xfs_repair.c also has same issue, fix that too.

Looks good to me, hopefully this will cut down on the random xfs_repair errors in fstests.

--D

Quoted 42 lines

---
v2: fix the do_error() in xfs_repair.c too.

 libfrog/util.c      | 2 ++
 repair/xfs_repair.c | 2 ++
 2 files changed, 4 insertions(+)
libfrog/util.c +2 -0
index 5bae5bab..3b6df917 100644
@@ -116,8 +116,10 @@ cmn_err(int level, char *fmt, ...)
 	va_list	ap;
 
 	va_start(ap, fmt);
+	flockfile(stderr);
 	vfprintf(stderr, fmt, ap);
 	fputs("\n", stderr);
+	funlockfile(stderr);
 	va_end(ap);
 }
 
repair/xfs_repair.c +2 -0
index 6b97a806..dd758ebc 100644
@@ -458,10 +458,12 @@ do_error(char const *msg, ...)
 {
 	va_list args;
 
+	flockfile(stderr);
 	fprintf(stderr, _("\nfatal error -- "));
 
 	va_start(args, msg);
 	vfprintf(stderr, msg, args);
+	funlockfile(stderr);
 	if (dumpcore)
 		abort();
 	exit(1);
Signature · 1 line
2.55.0
#6 lore raw
In reply to Avinesh Kumar
Quoted 2 lines · Avinesh Kumar

From: Avinesh Kumar <[email protected]>

Adding Andrey to the thread as he's maintaining xfsprogs.

Quoted 90 lines

fstests xfs/033 fails sporadically with a spurious blank line in the xfs_repair output:

- output mismatch (see /opt/xfstests/results//xfs/033.out.bad)

    --- tests/xfs/033.out	2026-06-24 15:52:51.000000000 -0400
    +++ /opt/xfstests/results//xfs/033.out.bad	2026-07-14 18:54:46.582495041 -0400
    @@ -103,6 +103,7 @@
     Phase 3 - for each AG...
             - scan and clear agi unlinked lists...
             - process known inodes and perform inode discovery...
    +
     bad magic number 0xffff on inode INO
     bad version number 0xffffffff on inode INO
     inode identifier 18446744073709551615 mismatch on inode INO

Root cause is in cmn_err() (libfrog/util.c), which emits a message and its newline as two separate writes to unbuffered stderr. xfs_repair's threads all share stderr. If one is preempted between the two writes, another thread's line lands in between:

(snips from cat -A 033.raw) -
Phase 3 - for each AG...$
- scan and clear agi unlinked lists...$
- process known inodes and perform inode discovery...$
Metadata corruption detected at 0x445dd3, xfs_inode block 0x80/0x4000 - agno = 0$
$
bad CRC for inode 128$
bad magic number 0x0 on inode 128$

which should be like -

Phase 3 - for each AG...$
- scan and clear agi unlinked lists...$
- process known inodes and perform inode discovery...$
Metadata corruption detected at 0x445dd3, xfs_inode block 0x80/0x4000$
- agno = 0$
bad CRC for inode 130$
bad magic number 0x0 on inode 130$

_filter_repair() strips the noise line it glued onto but not the lone newline, which then fails the golden diff.

Make the two writes atomic.

do_error() in xfs_repair.c also has same issue, fix that too.

---
v2: fix the do_error() in xfs_repair.c too.

 libfrog/util.c      | 2 ++
 repair/xfs_repair.c | 2 ++
 2 files changed, 4 insertions(+)
libfrog/util.c +2 -0
index 5bae5bab..3b6df917 100644
@@ -116,8 +116,10 @@ cmn_err(int level, char *fmt, ...)
 	va_list	ap;
 
 	va_start(ap, fmt);
+	flockfile(stderr);
 	vfprintf(stderr, fmt, ap);
 	fputs("\n", stderr);
+	funlockfile(stderr);
 	va_end(ap);
 }
 
repair/xfs_repair.c +2 -0
index 6b97a806..dd758ebc 100644
@@ -458,10 +458,12 @@ do_error(char const *msg, ...)
 {
 	va_list args;
 
+	flockfile(stderr);
 	fprintf(stderr, _("\nfatal error -- "));
 
 	va_start(args, msg);
 	vfprintf(stderr, msg, args);
+	funlockfile(stderr);
 	if (dumpcore)
 		abort();
 	exit(1);
Signature · 1 line
2.55.0