Skip to contentSkip to conversation

ovl: relax lookup error on mismatch origin ftype

PATCH In review Amir Goldstein 2 participants 8 messages Apr 27, 2021 – Jul 21, 2021 3 messages not in the archive Overlay filesystem

lore
Share & tools
lore

Amir Goldstein

author maintainer
#0 lore raw

PATCH 1/1 ovl: relax lookup error on mismatch origin ftype

We get occasional reports of lookup errors due to mismatched origin ftype from users that re-format a lower squashfs image.

Commit 13c6ad0f45fd ("ovl: document lower modification caveats") tries to discourage the practice of re-formating lower layers and describes the expected behavior as undefined.

Commit b0e0f69731cd ("ovl: restrict lower null uuid for "xino=auto"") limits the configurations in which origin file handles are followed.

In addition to these measures, change the behavior in case of detecting a mismatch origin ftype in lookup to issue a warning, not follow origin, but not fail the lookup operation either.

That should make overall more users happy without any big consequences.

---

Miklos,

I am getting tired of dealing with lower squashfs related reports. How about this?

It passes the xfstests quick tests and no, I do not have a reproducer for origin mismatch, so will wait for Byron to test the patch.

Thanks,
Amir.

 fs/overlayfs/namei.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/overlayfs/namei.c +2 -2
index 71e264e2f16b..850c0a37f1f0 100644
@@ -392,7 +392,7 @@ int ovl_check_origin_fh(struct ovl_fs *ofs, struct ovl_fh *fh, bool connected,
 			    upperdentry, d_inode(upperdentry)->i_mode & S_IFMT,
 			    d_inode(origin)->i_mode & S_IFMT);
 	dput(origin);
-	return -EIO;
+	return -EINVAL;
 }
 
 static int ovl_check_origin(struct ovl_fs *ofs, struct dentry *upperdentry,
@@ -408,7 +408,7 @@ static int ovl_check_origin(struct ovl_fs *ofs, struct dentry *upperdentry,
 	kfree(fh);
 
 	if (err) {
-		if (err == -ESTALE)
+		if (err == -ESTALE || err == -EINVAL)
 			return 0;
 		return err;
 	}
Signature · 1 line
2.25.1

A message that is not in the archive (sent off-list or not yet received): [email protected]

author maintainer
#2 lore raw
Quoted 11 lines · www

Hi Amir,

Thank you very much for your help. I have another question to clarify.

Quoted 4 lines
Quoted 1 line

3. If we upgrade overlayfs separately, we are not very good at verifying that we have solved this problem, because the recurrence probability of this problem is very low. So I want to ask, how can we quickly reproduce this problem?

Re-creating a lower squashfs after files have been copied to upper should reproduce the problem quite often.

Quoted 8 lines

Does the re-creating lower squashfs here mean that remount squashfs?
I've tested dozens of times in the remount way, but I haven't found this problem again?
My test steps are:
1). umount lower squashfs;
2). modify the file in upper dir;
3). mount lower squanshfs;
4). restart service(it will re-parse the modified file) or reboot the system and the problem didn't happen.

No. That's not what I mean by re-creating lower fs.
What I mean is that overlayfs is the file is question is in the squashfs image and has been copied up.

I don't know where the squashfs image you are using comes from but I am guessing you have replaced it with a new squashfs image. In the new squashfs image, files have different inode numbers.

I reckon this behavior is common for OpenWRT where the system image is being upgraded.

Quoted 12 lines
Quoted 10 lines · Amir Goldstein

We get occasional reports of lookup errors due to mismatched origin ftype from users that re-format a lower squashfs image.

Commit 13c6ad0f45fd ("ovl: document lower modification caveats") tries to discourage the practice of re-formating lower layers and describes the expected behavior as undefined.

Commit b0e0f69731cd ("ovl: restrict lower null uuid for "xino=auto"") limits the configurations in which origin file handles are followed.

Quoted 8 lines
Quoted 6 lines

In addition to these measures, change the behavior in case of detecting a mismatch origin ftype in lookup to issue a warning, not follow origin, but not fail the lookup operation either.

That should make overall more users happy without any big consequences.

We can't do that. When overlayfs reports an error, there may be some very important files that cannot be correctly parsed, resulting in some important services can not be started smoothly, which directly affects the use of the whole system.

Well, that is the purpose of my patch -
When overlayfs reports the warning it will NOT prevent reading the file, so it should NOT make the whole system unusable.

Thanks,
Amir.

A message that is not in the archive (sent off-list or not yet received): [email protected]

author maintainer
#4 lore raw
Quoted 38 lines · www

Hi Amir,

Quoted 25 lines · Amir Goldstein
Quoted 11 lines · www

Hi Amir,

Thank you very much for your help. I have another question to clarify.

Quoted 4 lines
Quoted 1 line

3. If we upgrade overlayfs separately, we are not very good at verifying that we have solved this problem, because the recurrence probability of this problem is very low. So I want to ask, how can we quickly reproduce this problem?

Re-creating a lower squashfs after files have been copied to upper should reproduce the problem quite often.

Quoted 8 lines

Does the re-creating lower squashfs here mean that remount squashfs?
I've tested dozens of times in the remount way, but I haven't found this problem again?
My test steps are:
1). umount lower squashfs;
2). modify the file in upper dir;
3). mount lower squanshfs;
4). restart service(it will re-parse the modified file) or reboot the system and the problem didn't happen.

No. That's not what I mean by re-creating lower fs.
What I mean is that overlayfs is the file is question is in the squashfs image and has been copied up.

Quoted 7 lines

I don't know where the squashfs image you are using comes from but I am guessing you have replaced it with a new squashfs image. In the new squashfs image, files have different inode numbers.

I reckon this behavior is common for OpenWRT where the system image is being upgraded.

Quoted 11 lines

https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-phosphor/initrdscripts/files/obmc-init.sh (init file)
...
line 376: mount /run/image-rofs /run/initramfs/ro -t squanshfs -o ro,loop (line 358, set copy-base-filesystem-to-ram=y)
...
line 416: mkdir -p $upper/var/log --- new add line, not in the file
line 417: mount -t overlayfs -o lowerdir=run/initramfs/ro,upperdir=run/initramfs/rw/cow,workdir=run/initramfs/rw/work cow /root
...
I would like to ask if the newly added line(line 416) or "set set copy-base-filesystem-to-ram=y" causes this problem?(This folder is created and the value is set every time the system starts, but the probability of problems is not high) There are no other changes.

I don't know.

I am not sure what wasn't clear in my answer.

The lowerfs image, namely /run/image-rofs in your script can be downloaded from tftp or from the web. I have no idea what actually happens in your setup.

Every time you run this init script this lower fs image may change, while the upper f2fs rwfs does not change.
When that happens, it MAY cause the reported issue.

Thanks,
Amir.

A message that is not in the archive (sent off-list or not yet received): [email protected]

author maintainer
#6 lore raw
Quoted 69 lines · www

Hi Amir,

Quoted 51 lines · Amir Goldstein
Quoted 38 lines · www

Hi Amir,

Quoted 25 lines · Amir Goldstein
Quoted 11 lines · www

Hi Amir,

Thank you very much for your help. I have another question to clarify.

Quoted 4 lines
Quoted 1 line

3. If we upgrade overlayfs separately, we are not very good at verifying that we have solved this problem, because the recurrence probability of this problem is very low. So I want to ask, how can we quickly reproduce this problem?

Re-creating a lower squashfs after files have been copied to upper should reproduce the problem quite often.

Quoted 8 lines

Does the re-creating lower squashfs here mean that remount squashfs?
I've tested dozens of times in the remount way, but I haven't found this problem again?
My test steps are:
1). umount lower squashfs;
2). modify the file in upper dir;
3). mount lower squanshfs;
4). restart service(it will re-parse the modified file) or reboot the system and the problem didn't happen.

No. That's not what I mean by re-creating lower fs.
What I mean is that overlayfs is the file is question is in the squashfs image and has been copied up.

Quoted 7 lines

I don't know where the squashfs image you are using comes from but I am guessing you have replaced it with a new squashfs image. In the new squashfs image, files have different inode numbers.

I reckon this behavior is common for OpenWRT where the system image is being upgraded.

Quoted 11 lines

https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-phosphor/initrdscripts/files/obmc-init.sh (init file)
...
line 376: mount /run/image-rofs /run/initramfs/ro -t squanshfs -o ro,loop (line 358, set copy-base-filesystem-to-ram=y)
...
line 416: mkdir -p $upper/var/log --- new add line, not in the file
line 417: mount -t overlayfs -o lowerdir=run/initramfs/ro,upperdir=run/initramfs/rw/cow,workdir=run/initramfs/rw/work cow /root
...
I would like to ask if the newly added line(line 416) or "set set copy-base-filesystem-to-ram=y" causes this problem?(This folder is created and the value is set every time the system starts, but the probability of problems is not high) There are no other changes.

Quoted 7 lines

I don't know.

I am not sure what wasn't clear in my answer.

The lowerfs image, namely /run/image-rofs in your script can be downloaded from tftp or from the web. I have no idea what actually happens in your setup.

the /run/image-rofs image is copied form flash.
line 359: test ! -e /run/image-rofs && ! cp $rodev /run/image-rofs -- the $rodev is mtd4

Quoted 7 lines
Quoted 4 lines

Every time you run this init script this lower fs image may change, while the upper f2fs rwfs does not change.
When that happens, it MAY cause the reported issue.

the /run/image-rofs is squashfs, readonly, cannot be modified.
Do you mean that when I upgrade the new firmware, rofs (squashfs) has been updated (rofs has been modified), is it easy to have this problem?

YES!

Thanks,
Amir.

maintainer
#7 lore raw
In reply to Amir Goldstein
Quoted 21 lines · Amir Goldstein

We get occasional reports of lookup errors due to mismatched origin ftype from users that re-format a lower squashfs image.

Commit 13c6ad0f45fd ("ovl: document lower modification caveats") tries to discourage the practice of re-formating lower layers and describes the expected behavior as undefined.

Commit b0e0f69731cd ("ovl: restrict lower null uuid for "xino=auto"") limits the configurations in which origin file handles are followed.

In addition to these measures, change the behavior in case of detecting a mismatch origin ftype in lookup to issue a warning, not follow origin, but not fail the lookup operation either.

That should make overall more users happy without any big consequences.

---

Quoted 7 lines

Miklos,

I am getting tired of dealing with lower squashfs related reports. How about this?

It passes the xfstests quick tests and no, I do not have a reproducer for origin mismatch, so will wait for Byron to test the patch.

Pushed a simplified variant that just changes the
ovl_check_origin_fh() return value from -EIO to -ESTALE. Do you see
a problem with this?

Thanks,
Miklos

Quoted 32 lines

Thanks,
Amir.

 fs/overlayfs/namei.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/overlayfs/namei.c +2 -3
index 71e264e2f16b..850c0a37f1f0 100644
@@ -392,7 +392,7 @@ int ovl_check_origin_fh(struct ovl_fs *ofs, struct ovl_fh *fh, bool connected,
                            upperdentry, d_inode(upperdentry)->i_mode & S_IFMT,
                            d_inode(origin)->i_mode & S_IFMT);
        dput(origin);
-       return -EIO;
+       return -EINVAL;
 }
 
 static int ovl_check_origin(struct ovl_fs *ofs, struct dentry *upperdentry,
@@ -408,7 +408,7 @@ static int ovl_check_origin(struct ovl_fs *ofs, struct dentry *upperdentry,
        kfree(fh);
 
        if (err) {
-               if (err == -ESTALE)
+               if (err == -ESTALE || err == -EINVAL)
                        return 0;
                return err;
        }
--

2.25.1

author maintainer
#8 lore raw
Quoted 28 lines · Miklos Szeredi
Quoted 26 lines · Amir Goldstein

We get occasional reports of lookup errors due to mismatched origin ftype from users that re-format a lower squashfs image.

Commit 13c6ad0f45fd ("ovl: document lower modification caveats") tries to discourage the practice of re-formating lower layers and describes the expected behavior as undefined.

Commit b0e0f69731cd ("ovl: restrict lower null uuid for "xino=auto"") limits the configurations in which origin file handles are followed.

In addition to these measures, change the behavior in case of detecting a mismatch origin ftype in lookup to issue a warning, not follow origin, but not fail the lookup operation either.

That should make overall more users happy without any big consequences.

---

Miklos,

I am getting tired of dealing with lower squashfs related reports. How about this?

Quoted 7 lines
Quoted 2 lines

It passes the xfstests quick tests and no, I do not have a reproducer for origin mismatch, so will wait for Byron to test the patch.

Pushed a simplified variant that just changes the
ovl_check_origin_fh() return value from -EIO to -ESTALE. Do you see
a problem with this?

The only difference is if ovl_fh_to_dentry() also emits a warning, but since ovl_check_origin_fh() already has a warning I think that your simplified version is fine.

Thanks,
Amir.

maintainer
#9 lore raw
Quoted 33 lines · Amir Goldstein
Quoted 31 lines · Miklos Szeredi
Quoted 28 lines · Amir Goldstein

We get occasional reports of lookup errors due to mismatched origin ftype from users that re-format a lower squashfs image.

Commit 13c6ad0f45fd ("ovl: document lower modification caveats") tries to discourage the practice of re-formating lower layers and describes the expected behavior as undefined.

Commit b0e0f69731cd ("ovl: restrict lower null uuid for "xino=auto"") limits the configurations in which origin file handles are followed.

In addition to these measures, change the behavior in case of detecting a mismatch origin ftype in lookup to issue a warning, not follow origin, but not fail the lookup operation either.

That should make overall more users happy without any big consequences.

---

Miklos,

I am getting tired of dealing with lower squashfs related reports. How about this?

It passes the xfstests quick tests and no, I do not have a reproducer for origin mismatch, so will wait for Byron to test the patch.

Quoted 8 lines
Quoted 4 lines

Pushed a simplified variant that just changes the
ovl_check_origin_fh() return value from -EIO to -ESTALE. Do you see
a problem with this?

The only difference is if ovl_fh_to_dentry() also emits a warning, but since ovl_check_origin_fh() already has a warning I think that your simplified version is fine.

Yeah, you are right, new version could have one less warning in the ->fh_to_dentry() case, but I agree that that should not be a problem.

Thanks,
Miklos

author maintainer
#10 lore raw
Quoted 40 lines · Miklos Szeredi
Quoted 38 lines · Amir Goldstein
Quoted 35 lines · Miklos Szeredi
Quoted 21 lines · Amir Goldstein

We get occasional reports of lookup errors due to mismatched origin ftype from users that re-format a lower squashfs image.

Commit 13c6ad0f45fd ("ovl: document lower modification caveats") tries to discourage the practice of re-formating lower layers and describes the expected behavior as undefined.

Commit b0e0f69731cd ("ovl: restrict lower null uuid for "xino=auto"") limits the configurations in which origin file handles are followed.

In addition to these measures, change the behavior in case of detecting a mismatch origin ftype in lookup to issue a warning, not follow origin, but not fail the lookup operation either.

That should make overall more users happy without any big consequences.

---

Quoted 7 lines

Miklos,

I am getting tired of dealing with lower squashfs related reports. How about this?

It passes the xfstests quick tests and no, I do not have a reproducer for origin mismatch, so will wait for Byron to test the patch.

Pushed a simplified variant that just changes the
ovl_check_origin_fh() return value from -EIO to -ESTALE. Do you see
a problem with this?

Quoted 7 lines
Quoted 3 lines

The only difference is if ovl_fh_to_dentry() also emits a warning, but since ovl_check_origin_fh() already has a warning I think that your simplified version is fine.

Yeah, you are right, new version could have one less warning in the ->fh_to_dentry() case, but I agree that that should not be a problem.

Technically, there is also a change in vfs api in the error returned to exportfs_decode_fh_raw(), but both callers seem to convert EINVAL to ESTALE anyway...

Thanks,
Amir.