Possible regression after NFS eof page pollution fix (ext4 checksum errors)
Subsystems (2)
Hi Trond,
We’ve recently started seeing filesystem issues in our internal regression runs, and we were able to bisect the problem down to the following commit:
commit b1817b18ff20e69f5accdccefaf78bf5454bede2
Author: Trond Myklebust <[email protected]> Date: Thu Sep 4 18:46:16 2025 -0400
NFS: Protect against 'eof page pollution'
This commit fixes the failing xfstest 'generic/363'.
When the user mmaps() an area that extends beyond the end of file, and proceeds to write data into the folio that straddles that eof, we're required to discard that folio data if the user calls some function that extends the file length.
After this change, we intermittently see EXT4 checksum-related errors during boot. A representative dmesg excerpt is below:
[ 1908.365537] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #263414: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1908.375449] EXT4-fs error (device vda2): __ext4_find_entry:1624: inode #263414: comm updatedb: checksumming directory block 0 [ 1908.382985] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #263414: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1908.389289] EXT4-fs error (device vda2): __ext4_find_entry:1624: inode #263414: comm updatedb: checksumming directory block 0 [ 1909.598811] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423753: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1909.604308] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423753: comm updatedb: Directory block failed checksum [ 1909.958470] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423759: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1909.963825] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423759: comm updatedb: Directory block failed checksum [ 1909.985956] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #303617: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1909.991371] EXT4-fs error (device vda2): __ext4_find_entry:1624: inode #303617: comm updatedb: checksumming directory block 0 [ 1910.156415] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423761: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1910.161959] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423761: comm updatedb: Directory block failed checksum [ 1910.171364] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423735: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1910.177292] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423735: comm updatedb: Directory block failed checksum [ 1910.267721] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423744: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1910.281838] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423744: comm updatedb: Directory block failed checksum [ 1910.476906] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423751: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1910.482403] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423751: comm updatedb: Directory block failed checksum
The issue has so far only been observed in tests that use a nested VM setup. It does not reproduce deterministically, roughly half of the nested VM boots trigger the problem.
Would you mind taking a look or pointing us in the right direction? Please let us know if additional information, testing, or instrumentation would be helpful.
Thanks,
Mark
Trond Myklebust
maintainerQuoted 86 lines · Mark Bloch
Hi Trond,
We’ve recently started seeing filesystem issues in our internal regression runs, and we were able to bisect the problem down to the following commit:
commit
b1817b18ff20e69f5accdccefaf78bf5454bede2
Author: Trond Myklebust <[email protected]> Date: Thu Sep 4 18:46:16 2025 -0400NFS: Protect against 'eof page pollution'
This commit fixes the failing xfstest 'generic/363'.
When the user
mmaps()an area that extends beyond the end of file, and
proceeds to write data into the folio that straddles that eof, we're
required to discard that folio data if the user calls some function that
extends the file length.Signed-off-by Trond Myklebust <[email protected]>After this change, we intermittently see EXT4 checksum-related errors during boot.
A representative dmesg excerpt is below:[ 1908.365537] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #263414: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1908.375449] EXT4-fs error (device vda2): __ext4_find_entry:1624: inode #263414: comm updatedb: checksumming directory block 0 [ 1908.382985] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #263414: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1908.389289] EXT4-fs error (device vda2): __ext4_find_entry:1624: inode #263414: comm updatedb: checksumming directory block 0 [ 1909.598811] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423753: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1909.604308] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423753: comm updatedb: Directory block failed checksum [ 1909.958470] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423759: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1909.963825] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423759: comm updatedb: Directory block failed checksum [ 1909.985956] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #303617: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1909.991371] EXT4-fs error (device vda2): __ext4_find_entry:1624: inode #303617: comm updatedb: checksumming directory block 0 [ 1910.156415] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423761: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1910.161959] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423761: comm updatedb: Directory block failed checksum [ 1910.171364] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423735: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1910.177292] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423735: comm updatedb: Directory block failed checksum [ 1910.267721] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423744: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1910.281838] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423744: comm updatedb: Directory block failed checksum [ 1910.476906] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423751: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1910.482403] EXT4-fs error (device vda2):htree_dirblock_to_tree:1051: inode #423751: comm updatedb: Directory block failed checksum
The issue has so far only been observed in tests that use a nested VM setup.
It does not reproduce deterministically, roughly half of the nested VM boots trigger the problem.
Quoted 6 lines
Would you mind taking a look or pointing us in the right direction? Please let us know if additional information, testing, or instrumentation would be helpful.
Thanks,
Mark
I'm having trouble seeing how those issues can be related unless ext4 and NFS are somehow sharing the same folios. Does reverting just commit b1817b18ff20 and b2036bb65114 actually fix the ext4 problem?
What does "nested VM" mean in this situation, and what is the storage for the ext4 filesystem that is being corrupted?
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected], [email protected]
Quoted 91 lines · Trond Myklebust
Quoted 90 lines · Mark Bloch
Hi Trond,
We’ve recently started seeing filesystem issues in our internal regression runs, and we were able to bisect the problem down to the following commit:
commit
b1817b18ff20e69f5accdccefaf78bf5454bede2
Author: Trond Myklebust <[email protected]> Date: Thu Sep 4 18:46:16 2025 -0400NFS: Protect against 'eof page pollution'
This commit fixes the failing xfstest 'generic/363'.
When the user
mmaps()an area that extends beyond the end of file, and
proceeds to write data into the folio that straddles that eof, we're
required to discard that folio data if the user calls some function that
extends the file length.Signed-off-by Trond Myklebust <[email protected]>After this change, we intermittently see EXT4 checksum-related errors during boot.
A representative dmesg excerpt is below:[ 1908.365537] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #263414: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1908.375449] EXT4-fs error (device vda2): __ext4_find_entry:1624: inode #263414: comm updatedb: checksumming directory block 0 [ 1908.382985] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #263414: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1908.389289] EXT4-fs error (device vda2): __ext4_find_entry:1624: inode #263414: comm updatedb: checksumming directory block 0 [ 1909.598811] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423753: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1909.604308] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423753: comm updatedb: Directory block failed checksum [ 1909.958470] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423759: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1909.963825] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423759: comm updatedb: Directory block failed checksum [ 1909.985956] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #303617: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1909.991371] EXT4-fs error (device vda2): __ext4_find_entry:1624: inode #303617: comm updatedb: checksumming directory block 0 [ 1910.156415] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423761: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1910.161959] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423761: comm updatedb: Directory block failed checksum [ 1910.171364] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423735: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1910.177292] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423735: comm updatedb: Directory block failed checksum [ 1910.267721] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423744: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1910.281838] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423744: comm updatedb: Directory block failed checksum [ 1910.476906] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423751: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1910.482403] EXT4-fs error (device vda2):htree_dirblock_to_tree:1051: inode #423751: comm updatedb: Directory block failed checksum
The issue has so far only been observed in tests that use a nested VM setup.
It does not reproduce deterministically, roughly half of the nested VM boots trigger the problem.Would you mind taking a look or pointing us in the right direction? Please let us know if additional information, testing, or instrumentation would be helpful.
Quoted 6 lines
Quoted 2 lines
Thanks,
MarkI'm having trouble seeing how those issues can be related unless ext4 and NFS are somehow sharing the same folios. Does reverting just commit
b1817b18ff20andb2036bb65114actually fix the ext4 problem?
Yes, after reverting those two commits we no longer can reproduce it.
Quoted 4 lines
What does "nested VM" mean in this situation, and what is the storage for the ext4 filesystem that is being corrupted?
Probably should have explained better, let me do that now. Say we have host A.
On host A we run VM B.
Inside VM B we run VM C.
Inside VM B we have a mount (nfs one)
X:/images/.libvirt on /images/.libvirt type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=Y,local_lock=none,addr=X)
which holds the .img files. We launce QEMU with something like this:
{"driver":"file","filename":"/images/.libvirt/linux-VAGRANTSLASH-upstream_Z.img","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":true,"driver":"qcow2","file":"libvirt-2-storage","backing":null} -blockdev {"driver":"file","filename":"/images/.libvirt/Y.img","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}
inside VM C, it's a regular ext4 mount:/dev/vda2 on / type ext4 (rw,relatime)
Mark
Trond Myklebust
maintainerQuoted 155 lines · Mark Bloch
Quoted 115 lines · Trond Myklebust
Quoted 114 lines · Mark Bloch
Hi Trond,
We’ve recently started seeing filesystem issues in our internal regression runs, and we were able to bisect the problem down to the following commit:
commit
b1817b18ff20e69f5accdccefaf78bf5454bede2
Author: Trond Myklebust <[email protected]> Date: Thu Sep 4 18:46:16 2025 -0400NFS: Protect against 'eof page pollution'
This commit fixes the failing xfstest 'generic/363'.
When the user
mmaps()an area that extends beyond the end of file, and
proceeds to write data into the folio that straddles that eof,
we're
required to discard that folio data if the user calls some function that
extends the file length.Signed-off-by: Trond Myklebust
<[email protected]>After this change, we intermittently see EXT4 checksum-related errors
during boot.
A representative dmesg excerpt is below:[ 1908.365537] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #263414: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1908.375449] EXT4-fs error (device vda2): __ext4_find_entry:1624: inode #263414: comm updatedb: checksumming directory block 0 [ 1908.382985] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #263414: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1908.389289] EXT4-fs error (device vda2): __ext4_find_entry:1624: inode #263414: comm updatedb: checksumming directory block 0 [ 1909.598811] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423753: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1909.604308] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423753: comm updatedb: Directory block failed checksum [ 1909.958470] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423759: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1909.963825] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423759: comm updatedb: Directory block failed checksum [ 1909.985956] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #303617: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1909.991371] EXT4-fs error (device vda2): __ext4_find_entry:1624: inode #303617: comm updatedb: checksumming directory block 0 [ 1910.156415] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423761: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1910.161959] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423761: comm updatedb: Directory block failed checksum [ 1910.171364] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423735: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1910.177292] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423735: comm updatedb: Directory block failed checksum [ 1910.267721] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423744: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1910.281838] EXT4-fs error (device vda2): htree_dirblock_to_tree:1051: inode #423744: comm updatedb: Directory block failed checksum [ 1910.476906] EXT4-fs warning (device vda2): ext4_dirblock_csum_verify:375: inode #423751: comm updatedb: No space for directory leaf checksum. Please run e2fsck -D. [ 1910.482403] EXT4-fs error (device vda2):htree_dirblock_to_tree:1051: inode #423751: comm updatedb:
Directory
block failed checksumThe issue has so far only been observed in tests that use a nested VM
setup.
It does not reproduce deterministically, roughly half of the nested
VM boots trigger the problem.Would you mind taking a look or pointing us in the right direction?
Please let us know if additional information, testing, or instrumentation would be helpful.Quoted 7 lines
Quoted 2 lines
Thanks,
MarkI'm having trouble seeing how those issues can be related unless ext4
and NFS are somehow sharing the same folios. Does reverting just commitb1817b18ff20andb2036bb65114actually fix the ext4 problem?Yes, after reverting those two commits we no longer can reproduce it.
Quoted 5 lines
What does "nested VM" mean in this situation, and what is the storage
for the ext4 filesystem that is being corrupted?Probably should have explained better, let me do that now. Say we have host A.
On host A we run VM B.
Inside VM B we run VM C.Inside VM B we have a mount (nfs one)
X:/images/.libvirt on /images/.libvirt type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fat al_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=Y, local_lock=none,addr=X)which holds the .img files. We launce QEMU with something like this:
{"driver":"file","filename":"/images/.libvirt/linux-VAGRANTSLASH- upstream_Z.img","node-name":"libvirt-2-storage","auto-read- only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2- format","read-only":true,"driver":"qcow2","file":"libvirt-2- storage","backing":null} -blockdev {"driver":"file","filename":"/images/.libvirt/Y.img","node- name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}
Quoted 5 lines
inside VM C, it's a regular ext4 mount:
/dev/vda2on / type ext4 (rw,relatime)Mark
OK so if I'm understanding correctly, this is organised as ext4 partitions that are stored in qcow2 images that are again stored on a NFSv4.2 partition.
Do these qcow2 images have a file size that is fixed at creation time, or is the file size dynamic?
Also, does changing the "discard" option from "unmap" to "ignore" make any difference to the outcome?
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected], [email protected]
+@Riad Abu Ra Ed
________________________________________
From: Trond Myklebust <[email protected]>
Sent: Monday, January 5, 2026 5:20 PM
To: Mark Bloch <[email protected]>
Cc: Linoy Ganti <[email protected]>; Bar Friedman <[email protected]>; [email protected] <[email protected]>; Maor Gottlieb <[email protected]> Subject: Re: Possible regression after NFS eof page pollution fix (ext4 checksum errors)
External email: Use caution opening links or attachments
Quoted 310 lines · Mark Bloch
Quoted 236 lines · Trond Myklebust
Quoted 224 lines · Mark Bloch
Hi Trond,
We’ve recently started seeing filesystem issues in our internal
regression runs, and we were able to bisect the problem down to
the following commit:
commit
b1817b18ff20e69f5accdccefaf78bf5454bede2Author: Trond Myklebust <[email protected]>
Date: Thu Sep 4 18:46:16 2025 -0400
NFS: Protect against 'eof page pollution'
This commit fixes the failing xfstest 'generic/363'.
When the user
mmaps()an area that extends beyond the end offile, and
proceeds to write data into the folio that straddles that
eof,
we're
required to discard that folio data if the user calls some
function that
extends the file length.
Signed-off-by Trond MyklebustAfter this change, we intermittently see EXT4 checksum-related
errors
during boot.
A representative dmesg excerpt is below:
[ 1908.365537] EXT4-fs warning (device vda2):ext4_dirblock_csum_verify:375: inode #263414: comm updatedb: No
space
for directory leaf checksum. Please run e2fsck -D.
[ 1908.375449] EXT4-fs error (device vda2):__ext4_find_entry:1624:
inode #263414: comm updatedb: checksumming directory block 0
[ 1908.382985] EXT4-fs warning (device vda2):ext4_dirblock_csum_verify:375: inode #263414: comm updatedb: No
space
for directory leaf checksum. Please run e2fsck -D.
[ 1908.389289] EXT4-fs error (device vda2):__ext4_find_entry:1624:
inode #263414: comm updatedb: checksumming directory block 0
[ 1909.598811] EXT4-fs warning (device vda2):ext4_dirblock_csum_verify:375: inode #423753: comm updatedb: No
space
for directory leaf checksum. Please run e2fsck -D.
[ 1909.604308] EXT4-fs error (device vda2):htree_dirblock_to_tree:1051: inode #423753: comm updatedb:
Directory
block failed checksum
[ 1909.958470] EXT4-fs warning (device vda2):ext4_dirblock_csum_verify:375: inode #423759: comm updatedb: No
space
for directory leaf checksum. Please run e2fsck -D.
[ 1909.963825] EXT4-fs error (device vda2):htree_dirblock_to_tree:1051: inode #423759: comm updatedb:
Directory
block failed checksum
[ 1909.985956] EXT4-fs warning (device vda2):ext4_dirblock_csum_verify:375: inode #303617: comm updatedb: No
space
for directory leaf checksum. Please run e2fsck -D.
[ 1909.991371] EXT4-fs error (device vda2):__ext4_find_entry:1624:
inode #303617: comm updatedb: checksumming directory block 0
[ 1910.156415] EXT4-fs warning (device vda2):ext4_dirblock_csum_verify:375: inode #423761: comm updatedb: No
space
for directory leaf checksum. Please run e2fsck -D.
[ 1910.161959] EXT4-fs error (device vda2):htree_dirblock_to_tree:1051: inode #423761: comm updatedb:
Directory
block failed checksum
[ 1910.171364] EXT4-fs warning (device vda2):ext4_dirblock_csum_verify:375: inode #423735: comm updatedb: No
space
for directory leaf checksum. Please run e2fsck -D.
[ 1910.177292] EXT4-fs error (device vda2):htree_dirblock_to_tree:1051: inode #423735: comm updatedb:
Directory
block failed checksum
[ 1910.267721] EXT4-fs warning (device vda2):ext4_dirblock_csum_verify:375: inode #423744: comm updatedb: No
space
for directory leaf checksum. Please run e2fsck -D.
[ 1910.281838] EXT4-fs error (device vda2):htree_dirblock_to_tree:1051: inode #423744: comm updatedb:
Directory
block failed checksum
[ 1910.476906] EXT4-fs warning (device vda2):ext4_dirblock_csum_verify:375: inode #423751: comm updatedb: No
space
for directory leaf checksum. Please run e2fsck -D.
[ 1910.482403] EXT4-fs error (device vda2):htree_dirblock_to_tree:1051: inode #423751: comm updatedb:
Directory
block failed checksum
The issue has so far only been observed in tests that use a
nested VM
setup.
It does not reproduce deterministically, roughly half of the
nested
VM boots trigger the problem.
Would you mind taking a look or pointing us in the right
direction?
Please let us know if additional information, testing,
Quoted 7 lines
or instrumentation would be helpful.
Thanks,
Mark
Quoted 7 lines
I'm having trouble seeing how those issues can be related unless
ext4
and NFS are somehow sharing the same folios. Does reverting just
commit
b1817b18ff20andb2036bb65114actually fix the ext4 problem?Yes, after reverting those two commits we no longer can reproduce it.
Quoted 9 lines
What does "nested VM" mean in this situation, and what is the
storage
for the ext4 filesystem that is being corrupted?
Probably should have explained better, let me do that now.
Say we have host A.
On host A we run VM B.
Inside VM B we run VM C.
Inside VM B we have a mount (nfs one)
X:/images/.libvirt on /images/.libvirt type nfs4
(rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fat
al_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=Y,
local_lock=none,addr=X)
which holds the .img files. We launce QEMU with something like this:
{"driver":"file","filename":"/images/.libvirt/linux-VAGRANTSLASH-
upstream_Z.img","node-name":"libvirt-2-storage","auto-read-
only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-
format","read-only":true,"driver":"qcow2","file":"libvirt-2-
storage","backing":null} -blockdev
{"driver":"file","filename":"/images/.libvirt/Y.img","node-
name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}
Quoted 9 lines
inside VM C, it's a regular ext4 mount:
/dev/vda2on / type ext4 (rw,relatime)Mark
OK so if I'm understanding correctly, this is organised as ext4
partitions that are stored in qcow2 images that are again stored on a
NFSv4.2 partition.
Do these qcow2 images have a file size that is fixed at creation time,
or is the file size dynamic?
Also, does changing the "discard" option from "unmap" to "ignore" make
any difference to the outcome?
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
Trond Myklebust
maintainerHi Mark,
Quoted 11 lines · Trond Myklebust
OK so if I'm understanding correctly, this is organised as ext4 partitions that are stored in qcow2 images that are again stored on a NFSv4.2 partition.
Do these qcow2 images have a file size that is fixed at creation time,
or is the file size dynamic?
Also, does changing the "discard" option from "unmap" to "ignore" make
any difference to the outcome?
I've been staring at this for several days now, and the only candidate for a bug in the NFS client that I can see is this one. Can you please check if the following patch helps?
Thanks
Trond
8<------------------------------------------------------------------
From 18acd9e2652d44bcb8a48bc4643ab006787b809a Mon Sep 17 00:00:00 2001
Message-ID: <18acd9e2652d44bcb8a48bc4643ab006787b809a.1768091015.git.trond.myklebust@hammerspace.com>
From: Trond Myklebust <[email protected]>
Date: Sat, 10 Jan 2026 18:53:34 -0500
Subject: [PATCH] NFSv4.2: Fix size read races in fallocate and copy offload
If the pre-operation file size is read before locking the inode and quiescing O_DIRECT writes, then nfs_truncate_last_folio() might end up overwriting valid file data.
--- fs/nfs/io.c | 2 ++ fs/nfs/nfs42proc.c | 29 +++++++++++++++++++---------- 2 files changed, 21 insertions(+), 10 deletions(-)
fs/nfs/io.c +2 -0
@@ -84,6 +84,7 @@ nfs_start_io_write(struct inode *inode)nfs_file_block_o_direct (NFS_I (inode));return err; } +EXPORT_SYMBOL_GPL (nfs_start_io_write);/** * nfs_end_io_write - declare that the buffered write operation is done
@@ -97,6 +98,7 @@ nfs_end_io_write(struct inode *inode) {up_write (&inode->i_rwsem); } +EXPORT_SYMBOL_GPL (nfs_end_io_write);/* Call with exclusively locked inode->i_rwsem */ static void nfs_block_buffered (struct nfs_inode *nfsi,struct inode *inode)
fs/nfs/nfs42proc.c +19 -10
@@ -114,7 +114,6 @@ static int nfs42_proc_fallocate(struct rpc_message *msg, struct file *filep, exception.inode = inode; exception.state = lock->open_context->state; -nfs_file_block_o_direct (NFS_I (inode)); err =nfs_sync_inode (inode);if (err) goto out;
@@ -138,13 +137,17 @@ int nfs42_proc_allocate(struct file *filep, loff_t offset, loff_t len) .rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_ALLOCATE], };struct inode *inode =file_inode (filep); -loff_t oldsize =i_size_read (inode); +loff_t oldsize;int err;if (!nfs_server_capable (inode,NFS_CAP_ALLOCATE ))return -EOPNOTSUPP ; -inode_lock (inode); + err =nfs_start_io_write (inode); +if (err) +return err; + + oldsize =i_size_read (inode); err =nfs42_proc_fallocate (&msg, filep, offset, len);
@@ -155,7 +158,7 @@ int nfs42_proc_allocate(struct file *filep, loff_t offset, loff_t len)NFS_SERVER (inode)->caps &= ~(NFS_CAP_ALLOCATE |NFS_CAP_ZERO_RANGE ); -inode_unlock (inode); +nfs_end_io_write (inode);return err; }
@@ -170,7 +173,9 @@ int nfs42_proc_deallocate(struct file *filep, loff_t offset, loff_t len)if (!nfs_server_capable (inode,NFS_CAP_DEALLOCATE ))return -EOPNOTSUPP ; -inode_lock (inode); + err =nfs_start_io_write (inode); +if (err) +return err; err =nfs42_proc_fallocate (&msg, filep, offset, len);if (err ==0 )
@@ -179,7 +184,7 @@ int nfs42_proc_deallocate(struct file *filep, loff_t offset, loff_t len)NFS_SERVER (inode)->caps &= ~(NFS_CAP_DEALLOCATE |NFS_CAP_ZERO_RANGE ); -inode_unlock (inode); +nfs_end_io_write (inode);return err; }
@@ -189,14 +194,17 @@ int nfs42_proc_zero_range(struct file *filep, loff_t offset, loff_t len) .rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_ZERO_RANGE], };struct inode *inode =file_inode (filep); -loff_t oldsize =i_size_read (inode); +loff_t oldsize;int err;if (!nfs_server_capable (inode,NFS_CAP_ZERO_RANGE ))return -EOPNOTSUPP ; -inode_lock (inode); + err =nfs_start_io_write (inode); +if (err) +return err; + oldsize =i_size_read (inode); err =nfs42_proc_fallocate (&msg, filep, offset, len);if (err ==0 ) { nfs_truncate_last_folio(inode->i_mapping, oldsize,
@@ -205,7 +213,7 @@ int nfs42_proc_zero_range(struct file *filep, loff_t offset, loff_t len) }else if (err == -EOPNOTSUPP )NFS_SERVER (inode)->caps &= ~NFS_CAP_ZERO_RANGE ; -inode_unlock (inode); +nfs_end_io_write (inode);return err; }
@@ -416,7 +424,7 @@ static ssize_t _nfs42_proc_copy(struct file *src,struct nfs_server *src_server =NFS_SERVER (src_inode);loff_t pos_src = args->src_pos;loff_t pos_dst = args->dst_pos; -loff_t oldsize_dst =i_size_read (dst_inode); +loff_t oldsize_dst;size_t count = args->count;ssize_t status;
@@ -461,6 +469,7 @@ static ssize_t _nfs42_proc_copy(struct file *src, &src_lock->open_context->state->flags);set_bit (NFS_CLNT_DST_SSC_COPY_STATE , &dst_lock->open_context->state->flags); + oldsize_dst =i_size_read (dst_inode); status =nfs4_call_sync (dst_server->client, dst_server, &msg, &args->seq_args, &res->seq_res,0 );
Signature · 1 line
2.52.0
Hi Trond,
Quoted 11 lines · Trond Myklebust
Hi Mark,
Quoted 8 lines · Trond Myklebust
OK so if I'm understanding correctly, this is organised as ext4 partitions that are stored in qcow2 images that are again stored on a NFSv4.2 partition.
Do these qcow2 images have a file size that is fixed at creation time,
or is the file size dynamic?
The file size is dynamic (with a fixed maximum of 35 GB).
Quoted 3 lines
Quoted 3 lines
Also, does changing the "discard" option from "unmap" to "ignore" make
any difference to the outcome?
The discard option is already set to "ignore" in the image. Do you want us to test the other options just to see if it makes a difference?
Quoted 4 lines
I've been staring at this for several days now, and the only candidate for a bug in the NFS client that I can see is this one. Can you please check if the following patch helps?
Thanks for the patch, I'll let the team dealing with the issue know and let them test the patch.
I'll update once I know anything.
Mark
Quoted 149 lines
Thanks
Trond8<------------------------------------------------------------------
From18acd9e2652d44bcb8a48bc4643ab006787b809aMon Sep 17 00:00:00 2001
Message-ID: <18acd9e2652d44bcb8a48bc4643ab006787b809a.1768091015.git.trond.myklebust@hammerspace.com>
From: Trond Myklebust <[email protected]>
Date: Sat, 10 Jan 2026 18:53:34 -0500
Subject: [PATCH] NFSv4.2: Fix size read races in fallocate and copy offloadIf the pre-operation file size is read before locking the inode and quiescing
O_DIRECTwrites, thennfs_truncate_last_folio()might end up overwriting valid file data.Fixes b1817b18ff20 ("NFS: Protect against 'eof page pollution'")Signed-off-by Trond Myklebust <[email protected]>--- fs/nfs/io.c | 2 ++ fs/nfs/nfs42proc.c | 29 +++++++++++++++++++---------- 2 files changed, 21 insertions(+), 10 deletions(-)fs/nfs/io.c +2
-0@@ -84,6 +84,7 @@ nfs_start_io_write(struct inode *inode)nfs_file_block_o_direct (NFS_I (inode));return err; } +EXPORT_SYMBOL_GPL (nfs_start_io_write);/** * nfs_end_io_write - declare that the buffered write operation is done @@ -97,6 +98,7 @@ nfs_end_io_write(struct inode *inode) {up_write (&inode->i_rwsem); } +EXPORT_SYMBOL_GPL (nfs_end_io_write);/* Call with exclusively locked inode->i_rwsem */ static void nfs_block_buffered (struct nfs_inode *nfsi,struct inode *inode)fs/nfs/nfs42proc.c +19
-10@@ -114,7 +114,6 @@ static int nfs42_proc_fallocate(struct rpc_message *msg, struct file *filep, exception.inode = inode; exception.state = lock->open_context->state; -nfs_file_block_o_direct (NFS_I (inode)); err =nfs_sync_inode (inode);if (err) goto out;@@ -138,13 +137,17 @@ int nfs42_proc_allocate(struct file *filep, loff_t offset, loff_t len) .rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_ALLOCATE], };struct inode *inode =file_inode (filep); -loff_t oldsize =i_size_read (inode); +loff_t oldsize;int err;if (!nfs_server_capable (inode,NFS_CAP_ALLOCATE ))return -EOPNOTSUPP ; -inode_lock (inode); + err =nfs_start_io_write (inode); +if (err) +return err; + + oldsize =i_size_read (inode); err =nfs42_proc_fallocate (&msg, filep, offset, len);@@ -155,7 +158,7 @@ int nfs42_proc_allocate(struct file *filep, loff_t offset, loff_t len)NFS_SERVER (inode)->caps &= ~(NFS_CAP_ALLOCATE |NFS_CAP_ZERO_RANGE ); -inode_unlock (inode); +nfs_end_io_write (inode);return err; }@@ -170,7 +173,9 @@ int nfs42_proc_deallocate(struct file *filep, loff_t offset, loff_t len)if (!nfs_server_capable (inode,NFS_CAP_DEALLOCATE ))return -EOPNOTSUPP ; -inode_lock (inode); + err =nfs_start_io_write (inode); +if (err) +return err; err =nfs42_proc_fallocate (&msg, filep, offset, len);if (err ==0 )@@ -179,7 +184,7 @@ int nfs42_proc_deallocate(struct file *filep, loff_t offset, loff_t len)NFS_SERVER (inode)->caps &= ~(NFS_CAP_DEALLOCATE |NFS_CAP_ZERO_RANGE ); -inode_unlock (inode); +nfs_end_io_write (inode);return err; }@@ -189,14 +194,17 @@ int nfs42_proc_zero_range(struct file *filep, loff_t offset, loff_t len) .rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_ZERO_RANGE], };struct inode *inode =file_inode (filep); -loff_t oldsize =i_size_read (inode); +loff_t oldsize;int err;if (!nfs_server_capable (inode,NFS_CAP_ZERO_RANGE ))return -EOPNOTSUPP ; -inode_lock (inode); + err =nfs_start_io_write (inode); +if (err) +return err; + oldsize =i_size_read (inode); err =nfs42_proc_fallocate (&msg, filep, offset, len);if (err ==0 ) { nfs_truncate_last_folio(inode->i_mapping, oldsize,@@ -205,7 +213,7 @@ int nfs42_proc_zero_range(struct file *filep, loff_t offset, loff_t len) }else if (err == -EOPNOTSUPP )NFS_SERVER (inode)->caps &= ~NFS_CAP_ZERO_RANGE ; -inode_unlock (inode); +nfs_end_io_write (inode);return err; }@@ -416,7 +424,7 @@ static ssize_t _nfs42_proc_copy(struct file *src,struct nfs_server *src_server =NFS_SERVER (src_inode);loff_t pos_src = args->src_pos;loff_t pos_dst = args->dst_pos; -loff_t oldsize_dst =i_size_read (dst_inode); +loff_t oldsize_dst;size_t count = args->count;ssize_t status;@@ -461,6 +469,7 @@ static ssize_t _nfs42_proc_copy(struct file *src, &src_lock->open_context->state->flags);set_bit (NFS_CLNT_DST_SSC_COPY_STATE , &dst_lock->open_context->state->flags); + oldsize_dst =i_size_read (dst_inode); status =nfs4_call_sync (dst_server->client, dst_server, &msg, &args->seq_args, &res->seq_res,0 );
Trond Myklebust
maintainerQuoted 18 lines · Mark Bloch
Hi Trond,
Quoted 12 lines · Trond Myklebust
Hi Mark,
Quoted 9 lines · Trond Myklebust
OK so if I'm understanding correctly, this is organised as ext4 partitions that are stored in qcow2 images that are again stored on a
NFSv4.2 partition.Do these qcow2 images have a file size that is fixed at creation time,
or is the file size dynamic?The file size is dynamic (with a fixed maximum of 35 GB).
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Also, does changing the "discard" option from "unmap" to "ignore" make
any difference to the outcome?The discard option is already set to "ignore" in the image. Do you want us to test the other options just to see if it makes a difference?
I believe in your previous email you had it set as "unmap":
{"driver":"file","filename":"/images/.libvirt/linux-VAGRANTSLASH- upstream_Z.img","node-name":"libvirt-2-storage","auto-read- only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2- format","read-only":true,"driver":"qcow2","file":"libvirt-2- storage","backing":null} -blockdev {"driver":"file","filename":"/images/.libvirt/Y.img","node- name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}
However if you've already tested with "ignore" and it didn't make a difference then let's not worry about it.
Quoted 11 lines
Quoted 7 lines
I've been staring at this for several days now, and the only candidate
for a bug in the NFS client that I can see is this one. Can you please
check if the following patch helps?Thanks for the patch, I'll let the team dealing with the issue know and let them test the patch.
I'll update once I know anything.
Thanks!
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected], [email protected]
Mark Bloch
Tested-by Bar FriedmanQuoted 26 lines · Mark Bloch
Hi Trond,
Quoted 11 lines · Trond Myklebust
Hi Mark,
Quoted 8 lines · Trond Myklebust
OK so if I'm understanding correctly, this is organised as ext4 partitions that are stored in qcow2 images that are again stored on a NFSv4.2 partition.
Do these qcow2 images have a file size that is fixed at creation time,
or is the file size dynamic?The file size is dynamic (with a fixed maximum of 35 GB).
Quoted 3 lines
Quoted 3 lines
Also, does changing the "discard" option from "unmap" to "ignore" make
any difference to the outcome?The discard option is already set to "ignore" in the image. Do you want us to test the other options just to see if it makes a difference?
Quoted 1 line
Quoted 7 lines
Quoted 3 lines
I've been staring at this for several days now, and the only candidate for a bug in the NFS client that I can see is this one. Can you please check if the following patch helps?
Thanks for the patch, I'll let the team dealing with the issue know and let them test the patch.
I'll update once I know anything.
We've been testing your patch for some time now and didn't hit the issue. Feel free to add Bar's tested by tag as she was the one
that actually tested the fix. Thanks for looking into this.
Mark
Quoted 153 lines
Mark
Quoted 149 lines
Thanks
Trond8<------------------------------------------------------------------
From18acd9e2652d44bcb8a48bc4643ab006787b809aMon Sep 17 00:00:00 2001
Message-ID: <18acd9e2652d44bcb8a48bc4643ab006787b809a.1768091015.git.trond.myklebust@hammerspace.com>
From: Trond Myklebust <[email protected]>
Date: Sat, 10 Jan 2026 18:53:34 -0500
Subject: [PATCH] NFSv4.2: Fix size read races in fallocate and copy offloadIf the pre-operation file size is read before locking the inode and quiescing
O_DIRECTwrites, thennfs_truncate_last_folio()might end up overwriting valid file data.Fixes b1817b18ff20 ("NFS: Protect against 'eof page pollution'")Signed-off-by Trond Myklebust <[email protected]>--- fs/nfs/io.c | 2 ++ fs/nfs/nfs42proc.c | 29 +++++++++++++++++++---------- 2 files changed, 21 insertions(+), 10 deletions(-)fs/nfs/io.c +2
-0@@ -84,6 +84,7 @@ nfs_start_io_write(struct inode *inode)nfs_file_block_o_direct (NFS_I (inode));return err; } +EXPORT_SYMBOL_GPL (nfs_start_io_write);/** * nfs_end_io_write - declare that the buffered write operation is done @@ -97,6 +98,7 @@ nfs_end_io_write(struct inode *inode) {up_write (&inode->i_rwsem); } +EXPORT_SYMBOL_GPL (nfs_end_io_write);/* Call with exclusively locked inode->i_rwsem */ static void nfs_block_buffered (struct nfs_inode *nfsi,struct inode *inode)fs/nfs/nfs42proc.c +19
-10@@ -114,7 +114,6 @@ static int nfs42_proc_fallocate(struct rpc_message *msg, struct file *filep, exception.inode = inode; exception.state = lock->open_context->state; -nfs_file_block_o_direct (NFS_I (inode)); err =nfs_sync_inode (inode);if (err) goto out;@@ -138,13 +137,17 @@ int nfs42_proc_allocate(struct file *filep, loff_t offset, loff_t len) .rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_ALLOCATE], };struct inode *inode =file_inode (filep); -loff_t oldsize =i_size_read (inode); +loff_t oldsize;int err;if (!nfs_server_capable (inode,NFS_CAP_ALLOCATE ))return -EOPNOTSUPP ; -inode_lock (inode); + err =nfs_start_io_write (inode); +if (err) +return err; + + oldsize =i_size_read (inode); err =nfs42_proc_fallocate (&msg, filep, offset, len);@@ -155,7 +158,7 @@ int nfs42_proc_allocate(struct file *filep, loff_t offset, loff_t len)NFS_SERVER (inode)->caps &= ~(NFS_CAP_ALLOCATE |NFS_CAP_ZERO_RANGE ); -inode_unlock (inode); +nfs_end_io_write (inode);return err; }@@ -170,7 +173,9 @@ int nfs42_proc_deallocate(struct file *filep, loff_t offset, loff_t len)if (!nfs_server_capable (inode,NFS_CAP_DEALLOCATE ))return -EOPNOTSUPP ; -inode_lock (inode); + err =nfs_start_io_write (inode); +if (err) +return err; err =nfs42_proc_fallocate (&msg, filep, offset, len);if (err ==0 )@@ -179,7 +184,7 @@ int nfs42_proc_deallocate(struct file *filep, loff_t offset, loff_t len)NFS_SERVER (inode)->caps &= ~(NFS_CAP_DEALLOCATE |NFS_CAP_ZERO_RANGE ); -inode_unlock (inode); +nfs_end_io_write (inode);return err; }@@ -189,14 +194,17 @@ int nfs42_proc_zero_range(struct file *filep, loff_t offset, loff_t len) .rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_ZERO_RANGE], };struct inode *inode =file_inode (filep); -loff_t oldsize =i_size_read (inode); +loff_t oldsize;int err;if (!nfs_server_capable (inode,NFS_CAP_ZERO_RANGE ))return -EOPNOTSUPP ; -inode_lock (inode); + err =nfs_start_io_write (inode); +if (err) +return err; + oldsize =i_size_read (inode); err =nfs42_proc_fallocate (&msg, filep, offset, len);if (err ==0 ) { nfs_truncate_last_folio(inode->i_mapping, oldsize,@@ -205,7 +213,7 @@ int nfs42_proc_zero_range(struct file *filep, loff_t offset, loff_t len) }else if (err == -EOPNOTSUPP )NFS_SERVER (inode)->caps &= ~NFS_CAP_ZERO_RANGE ; -inode_unlock (inode); +nfs_end_io_write (inode);return err; }@@ -416,7 +424,7 @@ static ssize_t _nfs42_proc_copy(struct file *src,struct nfs_server *src_server =NFS_SERVER (src_inode);loff_t pos_src = args->src_pos;loff_t pos_dst = args->dst_pos; -loff_t oldsize_dst =i_size_read (dst_inode); +loff_t oldsize_dst;size_t count = args->count;ssize_t status;@@ -461,6 +469,7 @@ static ssize_t _nfs42_proc_copy(struct file *src, &src_lock->open_context->state->flags);set_bit (NFS_CLNT_DST_SSC_COPY_STATE , &dst_lock->open_context->state->flags); + oldsize_dst =i_size_read (dst_inode); status =nfs4_call_sync (dst_server->client, dst_server, &msg, &args->seq_args, &res->seq_res,0 );
Trond Myklebust
maintainerQuoted 42 lines · Mark Bloch
Quoted 35 lines · Mark Bloch
Hi Trond,
Quoted 13 lines · Trond Myklebust
Hi Mark,
Quoted 10 lines · Trond Myklebust
OK so if I'm understanding correctly, this is organised as ext4 partitions that are stored in qcow2 images that are again stored on a
NFSv4.2 partition.Do these qcow2 images have a file size that is fixed at creation
time,
or is the file size dynamic?The file size is dynamic (with a fixed maximum of 35 GB).
Quoted 4 lines
Quoted 4 lines
Also, does changing the "discard" option from "unmap" to "ignore"
make
any difference to the outcome?The discard option is already set to "ignore" in the image. Do you want us to test the other options just to see if it makes a difference?
Quoted 6 lines
I've been staring at this for several days now, and the only candidate
for a bug in the NFS client that I can see is this one. Can you please
check if the following patch helps?Quoted 3 lines
Thanks for the patch, I'll let the team dealing with the issue know and let them test the patch.
I'll update once I know anything.
Quoted 9 lines
We've been testing your patch for some time now and didn't hit the issue.
Feel free to add Bar's tested by tag as she was the one that actually tested the fix. Thanks for looking into this.Tested-by Bar Friedman <[email protected]>Mark
Thank you very much for testing, Bar! I unfortunately already sent the patch upstream as it was clearly a necessary fix (even though it was not obvious to me that it would be sufficient to fix your reported problem). I'm therefore hoping it will hit the 6.18.x stable kernels soon.
Quoted 1 line
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected], [email protected]