Skip to contentSkip to conversation

ovl: skip stale entries in merge dir cache iteration

PATCH In review Amir Goldstein 2 participants 10 messages Apr 26, 2021 – Jul 21, 2021 Overlay filesystem

lore
Share & tools
lore

Amir Goldstein

author maintainer
#0 lore raw

PATCH 1/1 ovl: skip stale entries in merge dir cache iteration

On the first getdents call, ovl_iterate() populates the readdir cache with a list of entries, but for upper entries with origin lower inode, p->ino remains zero.

Following getdents calls traverse the readdir cache list and call ovl_cache_update_ino() for entries with zero p->ino to lookup the entry in the overlay and return d_ino that is consistent with st_ino.

If the upper file was unlinked between the first getdents call and the getdents call that lists the file entry, ovl_cache_update_ino() will not find the entry and fall back to setting d_ino to the upper real st_ino, which is inconsistent with how this object was presented to users.

Instead of listing a stale entry with inconsistent d_ino, simply skip the stale entry, which is better for users.

---
 fs/overlayfs/readdir.c | 5 +++++
 1 file changed, 5 insertions(+)
fs/overlayfs/readdir.c +5 -0
index cc1e80257064..10b7780e4bdc 100644
@@ -481,6 +481,8 @@ static int ovl_cache_update_ino(struct path *path, struct ovl_cache_entry *p)
 	}
 	this = lookup_one_len(p->name, dir, p->len);
 	if (IS_ERR_OR_NULL(this) || !this->d_inode) {
+		/* Mark a stale entry */
+		p->is_whiteout = true;
 		if (IS_ERR(this)) {
 			err = PTR_ERR(this);
 			this = NULL;
@@ -776,6 +778,9 @@ static int ovl_iterate(struct file *file, struct dir_context *ctx)
 				if (err)
 					goto out;
 			}
+		}
+		/* ovl_cache_update_ino() sets is_whiteout on stale entry */
+		if (!p->is_whiteout) {
 			if (!dir_emit(ctx, p->name, p->len, p->ino, p->type))
 				break;
 		}
Signature · 1 line
2.25.1
author maintainer
#1 lore raw
Quoted 9 lines · Amir Goldstein

On the first getdents call, ovl_iterate() populates the readdir cache with a list of entries, but for upper entries with origin lower inode, p->ino remains zero.

Following getdents calls traverse the readdir cache list and call ovl_cache_update_ino() for entries with zero p->ino to lookup the entry in the overlay and return d_ino that is consistent with st_ino.

Quoted 8 lines

If the upper file was unlinked between the first getdents call and the getdents call that lists the file entry, ovl_cache_update_ino() will not find the entry and fall back to setting d_ino to the upper real st_ino, which is inconsistent with how this object was presented to users.

Instead of listing a stale entry with inconsistent d_ino, simply skip the stale entry, which is better for users.

Miklos,

I forgot to follow up on this patch.
Upstream xfstest overlay/077 is failing without this patch. Please add:

Thanks,
Amir.

Quoted 31 lines
---
 fs/overlayfs/readdir.c | 5 +++++
 1 file changed, 5 insertions(+)
fs/overlayfs/readdir.c +5 -1
index cc1e80257064..10b7780e4bdc 100644
@@ -481,6 +481,8 @@ static int ovl_cache_update_ino(struct path *path, struct ovl_cache_entry *p)
        }
        this = lookup_one_len(p->name, dir, p->len);
        if (IS_ERR_OR_NULL(this) || !this->d_inode) {
+               /* Mark a stale entry */
+               p->is_whiteout = true;
                if (IS_ERR(this)) {
                        err = PTR_ERR(this);
                        this = NULL;
@@ -776,6 +778,9 @@ static int ovl_iterate(struct file *file, struct dir_context *ctx)
                                if (err)
                                        goto out;
                        }
+               }
+               /* ovl_cache_update_ino() sets is_whiteout on stale entry */
+               if (!p->is_whiteout) {
                        if (!dir_emit(ctx, p->name, p->len, p->ino, p->type))
                                break;
                }
--

2.25.1

maintainer
#2 lore raw
Quoted 16 lines · Amir Goldstein
Quoted 14 lines · Amir Goldstein

On the first getdents call, ovl_iterate() populates the readdir cache with a list of entries, but for upper entries with origin lower inode, p->ino remains zero.

Following getdents calls traverse the readdir cache list and call ovl_cache_update_ino() for entries with zero p->ino to lookup the entry in the overlay and return d_ino that is consistent with st_ino.

If the upper file was unlinked between the first getdents call and the getdents call that lists the file entry, ovl_cache_update_ino() will not find the entry and fall back to setting d_ino to the upper real st_ino, which is inconsistent with how this object was presented to users.

Quoted 8 lines
Quoted 3 lines

Instead of listing a stale entry with inconsistent d_ino, simply skip the stale entry, which is better for users.

Miklos,

I forgot to follow up on this patch.
Upstream xfstest overlay/077 is failing without this patch.

Can't reproduce (on ext4/xfs and "-oxino=on").

Is there some trick?

Thanks,
Miklos

author maintainer
#3 lore raw
Quoted 22 lines · Miklos Szeredi
Quoted 20 lines · Amir Goldstein
Quoted 17 lines · Amir Goldstein

On the first getdents call, ovl_iterate() populates the readdir cache with a list of entries, but for upper entries with origin lower inode, p->ino remains zero.

Following getdents calls traverse the readdir cache list and call ovl_cache_update_ino() for entries with zero p->ino to lookup the entry in the overlay and return d_ino that is consistent with st_ino.

If the upper file was unlinked between the first getdents call and the getdents call that lists the file entry, ovl_cache_update_ino() will not find the entry and fall back to setting d_ino to the upper real st_ino, which is inconsistent with how this object was presented to users.

Instead of listing a stale entry with inconsistent d_ino, simply skip the stale entry, which is better for users.

Quoted 8 lines
Quoted 4 lines

Miklos,

I forgot to follow up on this patch.
Upstream xfstest overlay/077 is failing without this patch.

Can't reproduce (on ext4/xfs and "-oxino=on").

Is there some trick?

Not sure. overlay/077 fails for me on v5.14.0-rc2 on ext4/xfs.

QA output created by 077
+entry m100 has inconsistent d_ino (234 != 232) +entry f100 has inconsistent d_ino (335 != 16777542)
Silence is golden

Maybe you need to build src/t_dir_offset2?

Thanks,
Amir.

maintainer
#4 lore raw
Quoted 33 lines · Amir Goldstein
Quoted 22 lines · Miklos Szeredi
Quoted 20 lines · Amir Goldstein
Quoted 17 lines · Amir Goldstein

On the first getdents call, ovl_iterate() populates the readdir cache with a list of entries, but for upper entries with origin lower inode, p->ino remains zero.

Following getdents calls traverse the readdir cache list and call ovl_cache_update_ino() for entries with zero p->ino to lookup the entry in the overlay and return d_ino that is consistent with st_ino.

If the upper file was unlinked between the first getdents call and the getdents call that lists the file entry, ovl_cache_update_ino() will not find the entry and fall back to setting d_ino to the upper real st_ino, which is inconsistent with how this object was presented to users.

Instead of listing a stale entry with inconsistent d_ino, simply skip the stale entry, which is better for users.

Quoted 8 lines
Quoted 4 lines

Miklos,

I forgot to follow up on this patch.
Upstream xfstest overlay/077 is failing without this patch.

Can't reproduce (on ext4/xfs and "-oxino=on").

Is there some trick?

Quoted 8 lines

Not sure. overlay/077 fails for me on v5.14.0-rc2 on ext4/xfs.

QA output created by 077
+entry m100 has inconsistent d_ino (234 != 232) +entry f100 has inconsistent d_ino (335 != 16777542)
Silence is golden

Maybe you need to build src/t_dir_offset2?

root@kvm:/opt/xfstests-dev# git log -1 --pretty=%h
10f6b231
root@kvm:/opt/xfstests-dev# cat local.config
export TEST_DEV=/dev/vdb1
export TEST_DIR=/test
export SCRATCH_DEV=/dev/vdb2
export SCRATCH_MNT=/scratch
export FSTYP=ext4
export OVERLAY_MOUNT_OPTIONS="-o xino=on"
root@kvm:/opt/xfstests-dev# make src/t_dir_offset2
make: 'src/t_dir_offset2' is up to date.
root@kvm:/opt/xfstests-dev# ./check -overlay overlay/077
FSTYP         -- overlay
PLATFORM      -- Linux/x86_64 kvm 5.14.0-rc2 #276 SMP Tue Jul 20
05:54:44 CEST 2021
MKFS_OPTIONS  -- /scratch
MOUNT_OPTIONS -- -o xino=on /scratch /scratch/ovl-mnt

overlay/077 1s ... 1s
Ran: overlay/077
Passed all 1 tests

root@kvm:/opt/xfstests-dev# cat results/overlay/077.full

Create file in pure upper dir:
getdents at offset 0 returned 192 bytes
created entry p0
entry p0 found as expected

Remove file in pure upper dir:
getdents at offset 0 returned 192 bytes
unlinked entry p100
entry p100 not found as expected

Create file in impure upper dir:
getdents at offset 0 returned 192 bytes
created entry o0
entry o0 found as expected

Remove file in impure upper dir:
getdents at offset 0 returned 192 bytes
unlinked entry o100
entry o100 not found as expected

Create file in merge dir:
getdents at offset 0 returned 192 bytes
created entry m0
entry m0 found as expected

Remove file in merge dir:
getdents at offset 0 returned 192 bytes
unlinked entry m100
entry m100 not found as expected

Create file in former merge dir:
getdents at offset 0 returned 192 bytes
created entry f0
entry f0 found as expected

Remove file in former merge dir:
getdents at offset 0 returned 192 bytes
unlinked entry f100
entry f100 not found as expected

Ideas for further debugging why this test isn't failing for v4.12-rc2?

Thanks,
Miklos

maintainer
#5 lore raw
Quoted 103 lines · Miklos Szeredi
Quoted 33 lines · Amir Goldstein
Quoted 22 lines · Miklos Szeredi
Quoted 20 lines · Amir Goldstein
Quoted 17 lines · Amir Goldstein

On the first getdents call, ovl_iterate() populates the readdir cache with a list of entries, but for upper entries with origin lower inode, p->ino remains zero.

Following getdents calls traverse the readdir cache list and call ovl_cache_update_ino() for entries with zero p->ino to lookup the entry in the overlay and return d_ino that is consistent with st_ino.

If the upper file was unlinked between the first getdents call and the getdents call that lists the file entry, ovl_cache_update_ino() will not find the entry and fall back to setting d_ino to the upper real st_ino, which is inconsistent with how this object was presented to users.

Instead of listing a stale entry with inconsistent d_ino, simply skip the stale entry, which is better for users.

Quoted 8 lines
Quoted 4 lines

Miklos,

I forgot to follow up on this patch.
Upstream xfstest overlay/077 is failing without this patch.

Can't reproduce (on ext4/xfs and "-oxino=on").

Is there some trick?

Quoted 8 lines

Not sure. overlay/077 fails for me on v5.14.0-rc2 on ext4/xfs.

QA output created by 077
+entry m100 has inconsistent d_ino (234 != 232) +entry f100 has inconsistent d_ino (335 != 16777542)
Silence is golden

Maybe you need to build src/t_dir_offset2?

root@kvm:/opt/xfstests-dev# git log -1 --pretty=%h
10f6b231
root@kvm:/opt/xfstests-dev# cat local.config
export TEST_DEV=/dev/vdb1
export TEST_DIR=/test
export SCRATCH_DEV=/dev/vdb2
export SCRATCH_MNT=/scratch
export FSTYP=ext4
export OVERLAY_MOUNT_OPTIONS="-o xino=on"
root@kvm:/opt/xfstests-dev# make src/t_dir_offset2
make: 'src/t_dir_offset2' is up to date.
root@kvm:/opt/xfstests-dev# ./check -overlay overlay/077
FSTYP         -- overlay
PLATFORM      -- Linux/x86_64 kvm 5.14.0-rc2 #276 SMP Tue Jul 20
05:54:44 CEST 2021
MKFS_OPTIONS  -- /scratch
MOUNT_OPTIONS -- -o xino=on /scratch /scratch/ovl-mnt

overlay/077 1s ... 1s
Ran: overlay/077
Passed all 1 tests

root@kvm:/opt/xfstests-dev# cat results/overlay/077.full

Create file in pure upper dir:
getdents at offset 0 returned 192 bytes
created entry p0
entry p0 found as expected

Remove file in pure upper dir:
getdents at offset 0 returned 192 bytes
unlinked entry p100
entry p100 not found as expected

Create file in impure upper dir:
getdents at offset 0 returned 192 bytes
created entry o0
entry o0 found as expected

Remove file in impure upper dir:
getdents at offset 0 returned 192 bytes
unlinked entry o100
entry o100 not found as expected

Create file in merge dir:
getdents at offset 0 returned 192 bytes
created entry m0
entry m0 found as expected

Remove file in merge dir:
getdents at offset 0 returned 192 bytes
unlinked entry m100
entry m100 not found as expected

Create file in former merge dir:
getdents at offset 0 returned 192 bytes
created entry f0
entry f0 found as expected

Quoted 6 lines

Remove file in former merge dir:
getdents at offset 0 returned 192 bytes
unlinked entry f100
entry f100 not found as expected

Ideas for further debugging why this test isn't failing for v4.12-rc2?

v5.14-rc2, that is (got the -rc2 part right the first time, though).

Thanks,
Miklos

author maintainer
#6 lore raw
Quoted 105 lines · Miklos Szeredi
Quoted 103 lines · Miklos Szeredi
Quoted 33 lines · Amir Goldstein
Quoted 22 lines · Miklos Szeredi
Quoted 20 lines · Amir Goldstein
Quoted 17 lines · Amir Goldstein

On the first getdents call, ovl_iterate() populates the readdir cache with a list of entries, but for upper entries with origin lower inode, p->ino remains zero.

Following getdents calls traverse the readdir cache list and call ovl_cache_update_ino() for entries with zero p->ino to lookup the entry in the overlay and return d_ino that is consistent with st_ino.

If the upper file was unlinked between the first getdents call and the getdents call that lists the file entry, ovl_cache_update_ino() will not find the entry and fall back to setting d_ino to the upper real st_ino, which is inconsistent with how this object was presented to users.

Instead of listing a stale entry with inconsistent d_ino, simply skip the stale entry, which is better for users.

Quoted 8 lines
Quoted 4 lines

Miklos,

I forgot to follow up on this patch.
Upstream xfstest overlay/077 is failing without this patch.

Can't reproduce (on ext4/xfs and "-oxino=on").

Is there some trick?

Quoted 8 lines

Not sure. overlay/077 fails for me on v5.14.0-rc2 on ext4/xfs.

QA output created by 077
+entry m100 has inconsistent d_ino (234 != 232) +entry f100 has inconsistent d_ino (335 != 16777542)
Silence is golden

Maybe you need to build src/t_dir_offset2?

root@kvm:/opt/xfstests-dev# git log -1 --pretty=%h
10f6b231
root@kvm:/opt/xfstests-dev# cat local.config
export TEST_DEV=/dev/vdb1
export TEST_DIR=/test
export SCRATCH_DEV=/dev/vdb2
export SCRATCH_MNT=/scratch
export FSTYP=ext4
export OVERLAY_MOUNT_OPTIONS="-o xino=on"
root@kvm:/opt/xfstests-dev# make src/t_dir_offset2
make: 'src/t_dir_offset2' is up to date.
root@kvm:/opt/xfstests-dev# ./check -overlay overlay/077
FSTYP         -- overlay
PLATFORM      -- Linux/x86_64 kvm 5.14.0-rc2 #276 SMP Tue Jul 20
05:54:44 CEST 2021
MKFS_OPTIONS  -- /scratch
MOUNT_OPTIONS -- -o xino=on /scratch /scratch/ovl-mnt

overlay/077 1s ... 1s
Ran: overlay/077
Passed all 1 tests

root@kvm:/opt/xfstests-dev# cat results/overlay/077.full

Create file in pure upper dir:
getdents at offset 0 returned 192 bytes
created entry p0
entry p0 found as expected

Remove file in pure upper dir:
getdents at offset 0 returned 192 bytes
unlinked entry p100
entry p100 not found as expected

Create file in impure upper dir:
getdents at offset 0 returned 192 bytes
created entry o0
entry o0 found as expected

Remove file in impure upper dir:
getdents at offset 0 returned 192 bytes
unlinked entry o100
entry o100 not found as expected

Create file in merge dir:
getdents at offset 0 returned 192 bytes
created entry m0
entry m0 found as expected

Remove file in merge dir:
getdents at offset 0 returned 192 bytes
unlinked entry m100
entry m100 not found as expected

Create file in former merge dir:
getdents at offset 0 returned 192 bytes
created entry f0
entry f0 found as expected

Quoted 7 lines
Quoted 6 lines

Remove file in former merge dir:
getdents at offset 0 returned 192 bytes
unlinked entry f100
entry f100 not found as expected

Ideas for further debugging why this test isn't failing for v4.12-rc2?

It's not you, it's me ;-)

The failure was lost during cleanup of t_dir_offset2 patches for submission and it is I who was running an older version of t_dir_offset2. Let me figure this out and get back to you with a working test.

Thanks,
Amir.

author maintainer
#7 lore raw
Quoted 115 lines · Amir Goldstein
Quoted 105 lines · Miklos Szeredi
Quoted 103 lines · Miklos Szeredi
Quoted 33 lines · Amir Goldstein
Quoted 22 lines · Miklos Szeredi
Quoted 20 lines · Amir Goldstein
Quoted 17 lines · Amir Goldstein

On the first getdents call, ovl_iterate() populates the readdir cache with a list of entries, but for upper entries with origin lower inode, p->ino remains zero.

Following getdents calls traverse the readdir cache list and call ovl_cache_update_ino() for entries with zero p->ino to lookup the entry in the overlay and return d_ino that is consistent with st_ino.

If the upper file was unlinked between the first getdents call and the getdents call that lists the file entry, ovl_cache_update_ino() will not find the entry and fall back to setting d_ino to the upper real st_ino, which is inconsistent with how this object was presented to users.

Instead of listing a stale entry with inconsistent d_ino, simply skip the stale entry, which is better for users.

Quoted 8 lines
Quoted 4 lines

Miklos,

I forgot to follow up on this patch.
Upstream xfstest overlay/077 is failing without this patch.

Can't reproduce (on ext4/xfs and "-oxino=on").

Is there some trick?

Quoted 8 lines

Not sure. overlay/077 fails for me on v5.14.0-rc2 on ext4/xfs.

QA output created by 077
+entry m100 has inconsistent d_ino (234 != 232) +entry f100 has inconsistent d_ino (335 != 16777542)
Silence is golden

Maybe you need to build src/t_dir_offset2?

root@kvm:/opt/xfstests-dev# git log -1 --pretty=%h
10f6b231
root@kvm:/opt/xfstests-dev# cat local.config
export TEST_DEV=/dev/vdb1
export TEST_DIR=/test
export SCRATCH_DEV=/dev/vdb2
export SCRATCH_MNT=/scratch
export FSTYP=ext4
export OVERLAY_MOUNT_OPTIONS="-o xino=on"
root@kvm:/opt/xfstests-dev# make src/t_dir_offset2
make: 'src/t_dir_offset2' is up to date.
root@kvm:/opt/xfstests-dev# ./check -overlay overlay/077
FSTYP         -- overlay
PLATFORM      -- Linux/x86_64 kvm 5.14.0-rc2 #276 SMP Tue Jul 20
05:54:44 CEST 2021
MKFS_OPTIONS  -- /scratch
MOUNT_OPTIONS -- -o xino=on /scratch /scratch/ovl-mnt

overlay/077 1s ... 1s
Ran: overlay/077
Passed all 1 tests

root@kvm:/opt/xfstests-dev# cat results/overlay/077.full

Create file in pure upper dir:
getdents at offset 0 returned 192 bytes
created entry p0
entry p0 found as expected

Remove file in pure upper dir:
getdents at offset 0 returned 192 bytes
unlinked entry p100
entry p100 not found as expected

Create file in impure upper dir:
getdents at offset 0 returned 192 bytes
created entry o0
entry o0 found as expected

Remove file in impure upper dir:
getdents at offset 0 returned 192 bytes
unlinked entry o100
entry o100 not found as expected

Create file in merge dir:
getdents at offset 0 returned 192 bytes
created entry m0
entry m0 found as expected

Remove file in merge dir:
getdents at offset 0 returned 192 bytes
unlinked entry m100
entry m100 not found as expected

Create file in former merge dir:
getdents at offset 0 returned 192 bytes
created entry f0
entry f0 found as expected

Quoted 7 lines
Quoted 6 lines

Remove file in former merge dir:
getdents at offset 0 returned 192 bytes
unlinked entry f100
entry f100 not found as expected

Ideas for further debugging why this test isn't failing for v4.12-rc2?

Quoted 7 lines

It's not you, it's me ;-)

The failure was lost during cleanup of t_dir_offset2 patches for submission and it is I who was running an older version of t_dir_offset2. Let me figure this out and get back to you with a working test.

How about you try again with a version of the test that actually has the check and the error print that I reported.... ;-)

Thanks,
Amir.

From 250615da6f01e99c055b7fc3bd9a2c1deac644e0 Mon Sep 17 00:00:00 2001
From: Amir Goldstein <[email protected]>
Date: Tue, 20 Jul 2021 17:45:43 +0300
Subject: [PATCH] src/t_dir_offset2: Check for inconsistent d_ino/st_ino

After unlink of a directory entry, that entry may still apear in getdents results of an already open directory fd, but it should return a d_ino value that is consistent with the already observed st_ino of that entry.

This is a regression test for kernel commit .... ("ovl: skip stale entries in merge dir cache iteration")

---
 src/t_dir_offset2.c | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
src/t_dir_offset2.c +12 -6
index 75b41c1a..026bc8f3 100644
@@ -44,6 +44,7 @@ int main(int argc, char *argv[])
 	char buf[BUF_SIZE];
 	int nread, bufsize = BUF_SIZE;
 	struct linux_dirent64 *d;
+	struct stat st = {};
 	int bpos, total, i;
 	off_t lret;
 	int retval = EXIT_SUCCESS;
@@ -81,9 +82,9 @@ int main(int argc, char *argv[])
 	}
 
 	if (filename) {
-		exists = !faccessat(fd, filename, F_OK, AT_SYMLINK_NOFOLLOW);
+		exists = !fstatat(fd, filename, &st, AT_SYMLINK_NOFOLLOW);
 		if (!exists && errno != ENOENT) {
-			perror("faccessat");
+			perror("fstatat");
 			exit(EXIT_FAILURE);
 		}
 	}
@@ -139,9 +140,6 @@ int main(int argc, char *argv[])
 			continue;
 		}
 
-		if (nread == 0)
-			break;
-
 		for (bpos = 0; bpos < nread; total++) {
 			d = (struct linux_dirent64 *) (buf + bpos);
 
@@ -165,8 +163,16 @@ int main(int argc, char *argv[])
 					printf("entry #%d: %s (d_ino=%lld, d_off=%lld)\n",
 					       i, d->d_name, (long long int)d->d_ino,
 					       (long long int)d->d_off);
-				if (!strcmp(filename, d->d_name))
+				if (!strcmp(filename, d->d_name)) {
 					found = 1;
+					if (st.st_ino && d->d_ino != st.st_ino) {
+						fprintf(stderr, "entry %s has inconsistent d_ino (%lld != %lld)\n",
+								filename,
+								(long long int)d->d_ino,
+								(long long int)st.st_ino);
+					}
+				}
+
 			}
 			bpos += d->d_reclen;
 		}
Signature · 1 line
2.32.0
author maintainer
#8 lore raw
Quoted 119 lines · Amir Goldstein
Quoted 117 lines · Amir Goldstein
Quoted 105 lines · Miklos Szeredi
Quoted 103 lines · Miklos Szeredi
Quoted 33 lines · Amir Goldstein
Quoted 22 lines · Miklos Szeredi
Quoted 20 lines · Amir Goldstein
Quoted 17 lines · Amir Goldstein

On the first getdents call, ovl_iterate() populates the readdir cache with a list of entries, but for upper entries with origin lower inode, p->ino remains zero.

Following getdents calls traverse the readdir cache list and call ovl_cache_update_ino() for entries with zero p->ino to lookup the entry in the overlay and return d_ino that is consistent with st_ino.

If the upper file was unlinked between the first getdents call and the getdents call that lists the file entry, ovl_cache_update_ino() will not find the entry and fall back to setting d_ino to the upper real st_ino, which is inconsistent with how this object was presented to users.

Instead of listing a stale entry with inconsistent d_ino, simply skip the stale entry, which is better for users.

Quoted 8 lines
Quoted 4 lines

Miklos,

I forgot to follow up on this patch.
Upstream xfstest overlay/077 is failing without this patch.

Can't reproduce (on ext4/xfs and "-oxino=on").

Is there some trick?

Quoted 8 lines

Not sure. overlay/077 fails for me on v5.14.0-rc2 on ext4/xfs.

QA output created by 077
+entry m100 has inconsistent d_ino (234 != 232) +entry f100 has inconsistent d_ino (335 != 16777542)
Silence is golden

Maybe you need to build src/t_dir_offset2?

root@kvm:/opt/xfstests-dev# git log -1 --pretty=%h
10f6b231
root@kvm:/opt/xfstests-dev# cat local.config
export TEST_DEV=/dev/vdb1
export TEST_DIR=/test
export SCRATCH_DEV=/dev/vdb2
export SCRATCH_MNT=/scratch
export FSTYP=ext4
export OVERLAY_MOUNT_OPTIONS="-o xino=on"
root@kvm:/opt/xfstests-dev# make src/t_dir_offset2
make: 'src/t_dir_offset2' is up to date.
root@kvm:/opt/xfstests-dev# ./check -overlay overlay/077
FSTYP         -- overlay
PLATFORM      -- Linux/x86_64 kvm 5.14.0-rc2 #276 SMP Tue Jul 20
05:54:44 CEST 2021
MKFS_OPTIONS  -- /scratch
MOUNT_OPTIONS -- -o xino=on /scratch /scratch/ovl-mnt

overlay/077 1s ... 1s
Ran: overlay/077
Passed all 1 tests

root@kvm:/opt/xfstests-dev# cat results/overlay/077.full

Create file in pure upper dir:
getdents at offset 0 returned 192 bytes
created entry p0
entry p0 found as expected

Remove file in pure upper dir:
getdents at offset 0 returned 192 bytes
unlinked entry p100
entry p100 not found as expected

Create file in impure upper dir:
getdents at offset 0 returned 192 bytes
created entry o0
entry o0 found as expected

Remove file in impure upper dir:
getdents at offset 0 returned 192 bytes
unlinked entry o100
entry o100 not found as expected

Create file in merge dir:
getdents at offset 0 returned 192 bytes
created entry m0
entry m0 found as expected

Remove file in merge dir:
getdents at offset 0 returned 192 bytes
unlinked entry m100
entry m100 not found as expected

Create file in former merge dir:
getdents at offset 0 returned 192 bytes
created entry f0
entry f0 found as expected

Quoted 7 lines
Quoted 6 lines

Remove file in former merge dir:
getdents at offset 0 returned 192 bytes
unlinked entry f100
entry f100 not found as expected

Ideas for further debugging why this test isn't failing for v4.12-rc2?

It's not you, it's me ;-)

Quoted 9 lines
Quoted 5 lines

The failure was lost during cleanup of t_dir_offset2 patches for submission and it is I who was running an older version of t_dir_offset2. Let me figure this out and get back to you with a working test.

How about you try again with a version of the test that actually has the check and the error print that I reported.... ;-)

P.S.1:
xino=on is futile for this test as it uses all layers on same fs

P.S.2:
The attached patch to t_dir_offset2 only reproduces the first inconsistency I reported:
entry m100 has inconsistent d_ino (234 != 232)

The second inconsistency requires another small patch to overlay/077 (attached). I will post both those patches once the kernel fix is in overlayfs-next.

Thanks,
Amir.

From e64a7b0c1ce697745c0bdb3ef8f43f0764510ef8 Mon Sep 17 00:00:00 2001
From: Amir Goldstein <[email protected]>
Date: Tue, 20 Jul 2021 19:11:03 +0300
Subject: [PATCH] overlay/077: test inconsistent d_ino in former merge dir

For testing of inconsistent d_ino/st_ino we need to unlink an entry whose st_ino is not that of the upper inode.

In the former merge dir setup we unlink all the files in the lower dir after copyup, so they all use st_ino of the upper inode.

Let the unlinked file f100 reside in a lower path that is not being unlinked so it will have the st_ino of the lower inode.

---
 tests/overlay/077 | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
tests/overlay/077 +3 -2
index 58c0f3b5..04cb3881 100755
@@ -61,8 +61,8 @@ mkdir -p $lowerdir/merge $lowerdir/former $upperdir/pure $upperdir/impure
 create_files $lowerdir/merge m
 # Files to be moved into impure upper dir
 create_files $lowerdir o
-# File to be copied up to make former merge dir impure
-touch $lowerdir/former/f100
+# File to be moved into former merge dir to make it impure
+touch $lowerdir/f100
 
 _scratch_mount
 
@@ -72,6 +72,7 @@ create_files $SCRATCH_MNT/former f
 touch $SCRATCH_MNT/merge/m100
 # Move copied up files so readdir will need to lookup origin d_ino
 mv $SCRATCH_MNT/o* $SCRATCH_MNT/impure/
+mv $SCRATCH_MNT/f100 $SCRATCH_MNT/former/
 
 # Remove the lower directory and mount overlay again to create
 # a "former merge dir"
Signature · 1 line
2.32.0
maintainer
#9 lore raw
Quoted 134 lines · Amir Goldstein
Quoted 119 lines · Amir Goldstein
Quoted 117 lines · Amir Goldstein
Quoted 105 lines · Miklos Szeredi
Quoted 103 lines · Miklos Szeredi
Quoted 33 lines · Amir Goldstein
Quoted 22 lines · Miklos Szeredi
Quoted 20 lines · Amir Goldstein
Quoted 17 lines · Amir Goldstein

On the first getdents call, ovl_iterate() populates the readdir cache with a list of entries, but for upper entries with origin lower inode, p->ino remains zero.

Following getdents calls traverse the readdir cache list and call ovl_cache_update_ino() for entries with zero p->ino to lookup the entry in the overlay and return d_ino that is consistent with st_ino.

If the upper file was unlinked between the first getdents call and the getdents call that lists the file entry, ovl_cache_update_ino() will not find the entry and fall back to setting d_ino to the upper real st_ino, which is inconsistent with how this object was presented to users.

Instead of listing a stale entry with inconsistent d_ino, simply skip the stale entry, which is better for users.

Quoted 8 lines
Quoted 4 lines

Miklos,

I forgot to follow up on this patch.
Upstream xfstest overlay/077 is failing without this patch.

Can't reproduce (on ext4/xfs and "-oxino=on").

Is there some trick?

Quoted 8 lines

Not sure. overlay/077 fails for me on v5.14.0-rc2 on ext4/xfs.

QA output created by 077
+entry m100 has inconsistent d_ino (234 != 232) +entry f100 has inconsistent d_ino (335 != 16777542)
Silence is golden

Maybe you need to build src/t_dir_offset2?

root@kvm:/opt/xfstests-dev# git log -1 --pretty=%h
10f6b231
root@kvm:/opt/xfstests-dev# cat local.config
export TEST_DEV=/dev/vdb1
export TEST_DIR=/test
export SCRATCH_DEV=/dev/vdb2
export SCRATCH_MNT=/scratch
export FSTYP=ext4
export OVERLAY_MOUNT_OPTIONS="-o xino=on"
root@kvm:/opt/xfstests-dev# make src/t_dir_offset2
make: 'src/t_dir_offset2' is up to date.
root@kvm:/opt/xfstests-dev# ./check -overlay overlay/077
FSTYP         -- overlay
PLATFORM      -- Linux/x86_64 kvm 5.14.0-rc2 #276 SMP Tue Jul 20
05:54:44 CEST 2021
MKFS_OPTIONS  -- /scratch
MOUNT_OPTIONS -- -o xino=on /scratch /scratch/ovl-mnt

overlay/077 1s ... 1s
Ran: overlay/077
Passed all 1 tests

root@kvm:/opt/xfstests-dev# cat results/overlay/077.full

Create file in pure upper dir:
getdents at offset 0 returned 192 bytes
created entry p0
entry p0 found as expected

Remove file in pure upper dir:
getdents at offset 0 returned 192 bytes
unlinked entry p100
entry p100 not found as expected

Create file in impure upper dir:
getdents at offset 0 returned 192 bytes
created entry o0
entry o0 found as expected

Remove file in impure upper dir:
getdents at offset 0 returned 192 bytes
unlinked entry o100
entry o100 not found as expected

Create file in merge dir:
getdents at offset 0 returned 192 bytes
created entry m0
entry m0 found as expected

Remove file in merge dir:
getdents at offset 0 returned 192 bytes
unlinked entry m100
entry m100 not found as expected

Create file in former merge dir:
getdents at offset 0 returned 192 bytes
created entry f0
entry f0 found as expected

Quoted 7 lines
Quoted 6 lines

Remove file in former merge dir:
getdents at offset 0 returned 192 bytes
unlinked entry f100
entry f100 not found as expected

Ideas for further debugging why this test isn't failing for v4.12-rc2?

It's not you, it's me ;-)

Quoted 9 lines
Quoted 5 lines

The failure was lost during cleanup of t_dir_offset2 patches for submission and it is I who was running an older version of t_dir_offset2. Let me figure this out and get back to you with a working test.

How about you try again with a version of the test that actually has the check and the error print that I reported.... ;-)

P.S.1:
xino=on is futile for this test as it uses all layers on same fs

Quoted 8 lines

P.S.2:
The attached patch to t_dir_offset2 only reproduces the first inconsistency I reported:
entry m100 has inconsistent d_ino (234 != 232)

The second inconsistency requires another small patch to overlay/077 (attached). I will post both those patches once the kernel fix is in overlayfs-next.

Thanks.

Verified and pushed.

Miklos