stat inconsistency with overlayfs
I am using find with its printf "%D" option (provides the same information as stats device information "%d" - device number in decimal) to figure out whether a file system entry resides in the r/o lowerdir or the r/w upperdir of an overlayfs mounted filesystem. I distinguish between the two by getting the device number from a (plain) file know to be in the upperdir.
The use case behind that is to be able to backup only files from the upperdir for several systems sharing a common lowerdir filesystem. I have used that (scripted approach via rsync) now for quiet some time and a few kernels back and it seemed to have worked very well.
Currently I am using kernel 3.17.7 on gentoo and I seem to observe a strange behaviour (which I do not recall to have seen before on 3.13 and 3.11) with my approach as follows:
.) plain files still work and the device number is correct
.) directories, however, always seem to reside in the lowerdir - even thoguh they do not exist there; in fact there's not a single file in the whole filesystem hierarchy that, according to stat/find, seems to reside in the upperdir:
please see the stat output for a file and a directory, both residing in the same (parent) directory which is completely located in the upperdir (and does not at all exist in the lowerdir):
# stat serial File: ‘serial’ Size: 17 Blocks: 8 IO Block: 4096 regular file Device: ca03h/51715d Inode: 88 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2014-02-19 19:01:56.278161346 +0100 Modify: 2014-02-19 19:01:56.278161346 +0100 Change: 2014-02-19 19:01:56.278161346 +0100 Birth: - # # stat certs/ File: ‘certs/’ Size: 4096 Blocks: 8 IO Block: 4096 directory Device: dh/13d Inode: 331140 Links: 2 Access: (0700/drwx------) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2014-04-26 14:08:47.337968562 +0200 Modify: 2014-02-19 18:55:58.458161346 +0100 Change: 2014-02-19 18:55:58.458161346 +0100 Birth: -
For comparision, please see the stat of /bin which only resides in the lowerdir and does not exist in the upperdir:
# stat /bin File: ‘/bin’ Size: 4096 Blocks: 8 IO Block: 4096 directory Device: dh/13d Inode: 401777 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-02-20 17:50:24.066368607 +0100 Modify: 2015-02-09 17:51:18.000000000 +0100 Change: 2015-02-09 23:58:06.011825328 +0100 Birth: -
I do not think that this is the expected behaviour and I am pretty confident that this was different on older kernels - or am I missing anything/doing anything wrong here?
Thanks and regards Atom2
--
To unsubscribe from this list: send the line "unsubscribe linux-unionfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Atom2,
I didn't notice this before, but I had reproduced the situation as you described. I'm not sure if it is a real problem.
The performing of 'stat' in Overlayfs are different between files and directories. Files directly call upper/lower getattr function but directories will set @dev and @ino by Overlayfs superblock itself.
See line 135 in fs/overlayfs/dir.c
"""
static int ovl_dir_getattr(struct vfsmount *mnt, struct dentry *dentry,
struct kstat *stat)
{
int err;
enum ovl_path_type type;
struct path realpath;
type = ovl_path_real(dentry, &realpath);
err = vfs_getattr(&realpath, stat);
if (err)
return err;
stat->dev = dentry->d_sb->s_dev; // I think it's the cause.
stat->ino = dentry->d_inode->i_ino;
/*
* It's probably not worth it to count subdirs to get the
* correct link count. nlink=1 seems to pacify 'find' and
* other utilities.
*/
if (OVL_TYPE_MERGE(type))
stat->nlink = 1;
return 0;
}"""
I don't have the Overlayfs code on 3.11 or 3.13. I've lookup the v11 code in Miklos's git tree:
git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs.git
and found the code had been changed since the earliest code I could get. So I don't know the reason of this behavior. I guess it is used for filesystem consistence: directories in the same superblock have same device number?
Quoted 4 lines · On 2015/2/21 2:15, Atom2 wrote:
I am using find with its printf "%D" option (provides the same information as stats device information "%d" - device number in decimal) to figure out whether a file system entry resides in the r/o lowerdir or the r/w upperdir of an overlayfs mounted filesystem. I distinguish between the two by getting the device number from a (plain) file know to be in the upperdir.
The use case behind that is to be able to backup only files from the upperdir for several systems sharing a common lowerdir filesystem. I have used that (scripted approach via rsync) now for quiet some time and a few kernels back and it seemed to have worked very well.
I think your requirement need to be reconsidering. Maybe we could keep the @dev for a lower-only directory?
Add Cc Miklos.
Thanks,
Hu
Quoted 37 lines
Currently I am using kernel 3.17.7 on gentoo and I seem to observe a strange behaviour (which I do not recall to have seen before on 3.13 and 3.11) with my approach as follows:
.) plain files still work and the device number is correct
.) directories, however, always seem to reside in the lowerdir - even thoguh they do not exist there; in fact there's not a single file in the whole filesystem hierarchy that, according to stat/find, seems to reside in the upperdir:please see the stat output for a file and a directory, both residing in the same (parent) directory which is completely located in the upperdir (and does not at all exist in the lowerdir):
# stat serial File: ‘serial’ Size: 17 Blocks: 8 IO Block: 4096 regular file Device: ca03h/51715d Inode: 88 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2014-02-19 19:01:56.278161346 +0100 Modify: 2014-02-19 19:01:56.278161346 +0100 Change: 2014-02-19 19:01:56.278161346 +0100 Birth: - # # stat certs/ File: ‘certs/’ Size: 4096 Blocks: 8 IO Block: 4096 directory Device: dh/13d Inode: 331140 Links: 2 Access: (0700/drwx------) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2014-04-26 14:08:47.337968562 +0200 Modify: 2014-02-19 18:55:58.458161346 +0100 Change: 2014-02-19 18:55:58.458161346 +0100 Birth: -For comparision, please see the stat of /bin which only resides in the lowerdir and does not exist in the upperdir:
# stat /bin File: ‘/bin’ Size: 4096 Blocks: 8 IO Block: 4096 directory Device: dh/13d Inode: 401777 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-02-20 17:50:24.066368607 +0100 Modify: 2015-02-09 17:51:18.000000000 +0100 Change: 2015-02-09 23:58:06.011825328 +0100 Birth: -
Quoted 10 lines
I do not think that this is the expected behaviour and I am pretty confident that this was different on older kernels - or am I missing anything/doing anything wrong here?
Thanks and regards Atom2
Signature · 5 lines
To unsubscribe from this list: send the line "unsubscribe linux-unionfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html .
--
To unsubscribe from this list: send the line "unsubscribe linux-unionfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi, Hu, Atom2,
Quoted 2 lines
I don't have the Overlayfs code on 3.11 or 3.13. I've lookup the v11 code in Miklos's git tree:
Fortunately I got the overlay fs code of kernel V3.18, which is mostly same to v11 code in Miklos's git.
135 static int ovl_dir_getattr(struct vfsmount *mnt, struct dentry *dentry,
136 struct kstat *stat)
137 {
138 int err; 139 enum ovl_path_type type; 140 struct path realpath; 141 142 type = ovl_path_real(dentry, &realpath); 143 err = vfs_getattr(&realpath, stat); 144 if (err)
145 return err;
146
147 stat->dev = dentry->d_sb->s_dev;
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ here set dev number as Hu mentioned
148 stat->ino = dentry->d_inode->i_ino;
149
150 /*
151 * It's probably not worth it to count subdirs to get the 152 * correct link count. nlink=1 seems to pacify 'find' and 153 * other utilities. 154 */
155 if (type == OVL_PATH_MERGE)
156 stat->nlink = 1;
157
158 return 0;
159 }
And the d_sb->s_dev is coming from the super_block of overlayfs, which is fake. The super block s_dev is initialized by the call trace "mount_nodev->set_anon_super-->get_anon_bdev".
Either the dir exists in lower or exists in upper, the overlay fs fakes it exits in overlay, which holds the fake device number.
Thanks,
George
Signature · 12 lines
George Wang 王旭 Kernel Quantity Engineer Red Hat Software (Beijing) Co.,Ltd IRC:xuw Tel:+86-010-62608041 Phone:15901231579 9/F, Tower C, Raycom -- To unsubscribe from this list: send the line "unsubscribe linux-unionfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Xu,
thanks for your follow-up.
Am 26.02.15 um 07:45 schrieb Xu Wang:
Quoted 34 lines
Hi, Hu, Atom2,
Quoted 2 lines
I don't have the Overlayfs code on 3.11 or 3.13. I've lookup the v11 code in Miklos's git tree:
Fortunately I got the overlay fs code of kernel V3.18, which is mostly same to v11 code in Miklos's git.
135 static int ovl_dir_getattr(struct vfsmount *mnt, struct dentry *dentry,
136 struct kstat *stat)
137 {138 int err; 139 enum ovl_path_type type; 140 struct path realpath; 141 142 type = ovl_path_real(dentry, &realpath); 143 err = vfs_getattr(&realpath, stat); 144 if (err)145 return err;
146
147stat->dev=dentry->d_sb->s_dev;
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ here set dev number as Hu mentioned
148stat->ino=dentry->d_inode->i_ino;
149
150 /*151 * It's probably not worth it to count subdirs to get the 152 * correct link count. nlink=1 seems to pacify 'find' and 153 * other utilities. 154 */155 if (type ==
OVL_PATH_MERGE)
156stat->nlink= 1;
157
158 return 0;
159 }
Quoted 6 lines
And the
d_sb->s_devis coming from thesuper_blockof overlayfs, which is fake. The super blocks_devis initialized by the call trace "mount_nodev->set_anon_super-->get_anon_bdev".Either the dir exists in lower or exists in upper, the overlay fs fakes it exits in overlay, which holds the fake device number.
I see what you are saying and code-wise that seems to be the cause of the behaviour I am seeing. I am, however, still not convinced at all that this is the correct/to be expected behaviour.
Giving this some further thought, might this not create more issues further down the line: In my (granted: limited) understanding the device/i-number combination must be unique across all mounted file-systems. Now does this also mean that the i-number is faked as well? In other words what is going to happen if the i-number of the directory residing in the upperdir (but wrongly attributed to the lowerdir) is identical to an i-number of another existing entry in the lowerdir?
Finally, stat, as I understand it, amongst other pieces of information also provides information about the device an entry resides on and that information, IMHO, simply is plain wrong for directories only existing on the upperdir.
Thanks Atom2
Atom2,
Quoted 4 lines
The use case behind that is to be able to backup only files from the upperdir for several systems sharing a common lowerdir filesystem. I have used that (scripted approach via rsync) now for quiet some time and a few kernels back and it seemed to have worked very well.
Why don't you just back up the upper directory itself instead of messing around with device numbers?
Thanks,
Miklos
Mikolos,
thanks for joining the party.
Am 03.03.15 um 16:14 schrieb Miklos Szeredi:
Quoted 9 lines
Atom2,
Quoted 4 lines
The use case behind that is to be able to backup only files from the upperdir for several systems sharing a common lowerdir filesystem. I have used that (scripted approach via rsync) now for quiet some time and a few kernels back and it seemed to have worked very well.
Why don't you just back up the upper directory itself instead of messing around with device numbers?
I am not aware of any such option, but if you could explain that a little bit more in detail, I am happy to explore other options.
Just for everybody to be on the same page: In my use case the r/o lowerdir's root directory is also the root of my file system (which is shared across many systems) from which (all) the systems boot. Using an initramfs during the boot process, the upperdir file system (which is different per system) is then r/w overlay-mounted over the common r/o lowerdir's root directory in order to catch any r/w access to the file system.
So in essence and according to my understanding there's no direct access to (only) the r/w upperdir (or only the lowerdir) from within any of the running systems other than using the device field to see where a file system entry actually exists - at least that's my current understanding.
But again, I am happy to learn ...
Regards Atom2
Quoted 16 lines · On 2015/3/4 1:12, Atom2 wrote:
Mikolos,
thanks for joining the party.Am 03.03.15 um 16:14 schrieb Miklos Szeredi:
Quoted 9 lines
Atom2,
Quoted 4 lines
The use case behind that is to be able to backup only files from the upperdir for several systems sharing a common lowerdir filesystem. I have used that (scripted approach via rsync) now for quiet some time and a few kernels back and it seemed to have worked very well.
Why don't you just back up the upper directory itself instead of messing around with device numbers?
I am not aware of any such option, but if you could explain that a little bit more in detail, I am happy to explore other options.
Quoted 9 lines
Just for everybody to be on the same page: In my use case the r/o lowerdir's root directory is also the root of my file system (which is shared across many systems) from which (all) the systems boot. Using an initramfs during the boot process, the upperdir file system (which is different per system) is then r/w overlay-mounted over the common r/o lowerdir's root directory in order to catch any r/w access to the file system.
So in essence and according to my understanding there's no direct access to (only) the r/w upperdir (or only the lowerdir) from within any of the running systems other than using the device field to see where a file system entry actually exists - at least that's my current understanding.
But again, I am happy to learn ...
Regards Atom2
Seems not all upper directories are accessible.
Miklos, I didn't find the change log about device number changing, Maybe there are some reasons?
For Atom2's requirement, I have some thoughts:
The device number of Overlayfs are now different between files and directories, My plan is showing Overlayfs device number for upper and upper-merged(upper and several lower) file/dir, using
"stat->dev = dentry->d_sb->s_dev"
and showing lowerfs device number for a lower only file/dir.
For upper non-directory files, report Overlayfs device number. Do not change the stat of a lower non-directory file.
For upper and upper-merged directories, report Overlayfs device number as before. But for lower only directories, report lowerfs device number.
After this, Atom2 could backup the files and directories have a device number equal to Overlayfs mount device number. But as Xu mentioned, this device number is just fake, so I don't know if there are other problems.
And for multi-lower mount, an Overlayfs directory may mixed up by several lower directories, Atom2 seems don't want to backup this kind of directories, we couldn't set the device number to Overlayfs device number, maybe we could use the left most lowerdir device number, but that may confused with the lower only directories.
Thanks,
Hu
Quoted 27 lines · On Tue, Mar 3, 2015 at 6:12 PM, Atom2 <[email protected]> wrote:
Mikolos,
thanks for joining the party.Am 03.03.15 um 16:14 schrieb Miklos Szeredi:
Quoted 10 lines
Atom2,
Quoted 4 lines
The use case behind that is to be able to backup only files from the upperdir for several systems sharing a common lowerdir filesystem. I have used that (scripted approach via rsync) now for quiet some time and a few kernels back and it seemed to have worked very well.
Why don't you just back up the upper directory itself instead of messing around with device numbers?
I am not aware of any such option, but if you could explain that a little bit more in detail, I am happy to explore other options.
Just for everybody to be on the same page: In my use case the r/o lowerdir's root directory is also the root of my file system (which is shared across many systems) from which (all) the systems boot. Using an initramfs during the boot process, the upperdir file system (which is different per system) is then r/w overlay-mounted over the common r/o lowerdir's root directory in order to catch any r/w access to the file system.
Quoted 4 lines
So in essence and according to my understanding there's no direct access to (only) the r/w upperdir (or only the lowerdir) from within any of the running systems other than using the device field to see where a file system entry actually exists - at least that's my current understanding.
Having access to the upperdir is just your own choice.
Without knowing the details of how you set up the mounts I can't give you a concrete example, but see for example the pivot_root(8) man page on how to get access to your old mountpoint. You can also use bind mounts to clone the upperdir into the overlay namespace.
Thanks,
Miklos
Hi Hu,
thanks for looking into this.
Am 26.02.15 um 06:25 schrieb hujianyang:
Quoted 4 lines
Hi Atom2,
I didn't notice this before, but I had reproduced the situation as you described. I'm not sure if it is a real problem.
My take on this is that stat, under all circumstances, should reliably provide information about the device the queried entry resides on. In case the entry only exists on the upperdir-device, overlayfs should not fool the user that it instead resides in the r/o lowerdir when in fact no such entry exists there.
In my view that's the only viable option to back-up (empty) directories (from a running system) which only reside in the upperdir provided that no backup is requested for the r/o (lower) file layer.
Quoted 4 lines
The performing of 'stat' in Overlayfs are different between files and directories. Files directly call upper/lower getattr function but directories will set @dev and @ino by Overlayfs superblock itself.
In my view that then seems to defeat the purpose of the device field for the stat sys-calls for overlayfs.
Quoted 34 lines
See line 135 in
fs/overlayfs/dir.c"""
static int ovl_dir_getattr(struct vfsmount *mnt, struct dentry *dentry,
struct kstat *stat){ int err; enum ovl_path_type type; struct path realpath; type = ovl_path_real(dentry, &realpath); err = vfs_getattr(&realpath, stat); if (err) return err; stat->dev = dentry->d_sb->s_dev; // I think it's the cause. stat->ino = dentry->d_inode->i_ino; /* * It's probably not worth it to count subdirs to get the * correct link count. nlink=1 seems to pacify 'find' and * other utilities. */ if (OVL_TYPE_MERGE(type)) stat->nlink = 1; return 0; }"""
I don't have the Overlayfs code on 3.11 or 3.13. I've lookup the v11 code in Miklos's git tree:
Quoted 6 lines
git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs.gitand found the code had been changed since the earliest code I could get. So I don't know the reason of this behavior. I guess it is used for filesystem consistence: directories in the same superblock have same device number?
Following that argument, why would that then be different for plain files? In my understanding a directory in linux is not really different from a file (both are just an abstraction of blocks from the underlying block device) with the main difference being that the former's content is interpreted by the file system code to allow traversal of directories. So all operations that can be applied likewise to both should provide consistent results between files and directories.
Probably there's a reason behind that difference, but it currently is beyond me.
Thanks and regards Atom2
Quoted 7 lines · On 2015/2/21 2:15, Atom2 wrote:
Quoted 4 lines
I am using find with its printf "%D" option (provides the same information as stats device information "%d" - device number in decimal) to figure out whether a file system entry resides in the r/o lowerdir or the r/w upperdir of an overlayfs mounted filesystem. I distinguish between the two by getting the device number from a (plain) file know to be in the upperdir.
The use case behind that is to be able to backup only files from the upperdir for several systems sharing a common lowerdir filesystem. I have used that (scripted approach via rsync) now for quiet some time and a few kernels back and it seemed to have worked very well.
Quoted 46 lines
I think your requirement need to be reconsidering. Maybe we could keep the @dev for a lower-only directory?
Add Cc Miklos.
Thanks,
HuQuoted 38 lines
Currently I am using kernel 3.17.7 on gentoo and I seem to observe a strange behaviour (which I do not recall to have seen before on 3.13 and 3.11) with my approach as follows:
.) plain files still work and the device number is correct
.) directories, however, always seem to reside in the lowerdir - even thoguh they do not exist there; in fact there's not a single file in the whole filesystem hierarchy that, according to stat/find, seems to reside in the upperdir:please see the stat output for a file and a directory, both residing in the same (parent) directory which is completely located in the upperdir (and does not at all exist in the lowerdir):
# stat serial File: ‘serial’ Size: 17 Blocks: 8 IO Block: 4096 regular file Device: ca03h/51715d Inode: 88 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2014-02-19 19:01:56.278161346 +0100 Modify: 2014-02-19 19:01:56.278161346 +0100 Change: 2014-02-19 19:01:56.278161346 +0100 Birth: - # # stat certs/ File: ‘certs/’ Size: 4096 Blocks: 8 IO Block: 4096 directory Device: dh/13d Inode: 331140 Links: 2 Access: (0700/drwx------) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2014-04-26 14:08:47.337968562 +0200 Modify: 2014-02-19 18:55:58.458161346 +0100 Change: 2014-02-19 18:55:58.458161346 +0100 Birth: -For comparision, please see the stat of /bin which only resides in the lowerdir and does not exist in the upperdir:
# stat /bin File: ‘/bin’ Size: 4096 Blocks: 8 IO Block: 4096 directory Device: dh/13d Inode: 401777 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-02-20 17:50:24.066368607 +0100 Modify: 2015-02-09 17:51:18.000000000 +0100 Change: 2015-02-09 23:58:06.011825328 +0100 Birth: -I do not think that this is the expected behaviour and I am pretty confident that this was different on older kernels - or am I missing anything/doing anything wrong here?
--
To unsubscribe from this list: send the line "unsubscribe linux-unionfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html