FAILED: patch "[PATCH] ovl: hash directory inodes for fsnotify" failed to apply to 4.14-stable tree
Subsystems (2)
[email protected]
maintainerThe patch below does not apply to the 4.14-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.
thanks,
greg k-h
------------------ original commit in Linus's tree ------------------
Quoted 1 line
From
31747eda41ef3c30c09c5c096b380bf54013746aMon Sep 17 00:00:00 2001
From: Amir Goldstein <[email protected]>
Date: Sun, 14 Jan 2018 18:35:40 +0200
Subject: [PATCH] ovl: hash directory inodes for fsnotify
fsnotify pins a watched directory inode in cache, but if directory dentry is released, new lookup will allocate a new dentry and a new inode. Directory events will be notified on the new inode, while fsnotify listener is watching the old pinned inode.
Hash all directory inodes to reuse the pinned inode on lookup. Pure upper dirs are hashes by real upper inode, merge and lower dirs are hashed by real lower inode.
The reference to lower inode was being held by the lower dentry object in the overlay dentry (oe->lowerstack[0]). Releasing the overlay dentry may drop lower inode refcount to zero. Add a refcount on behalf of the overlay inode to prevent that.
As a by-product, hashing directory inodes also detects multiple redirected dirs to the same lower dir and uncovered redirected dir target on and returns -ESTALE on lookup.
The reported issue dates back to initial version of overlayfs, but this patch depends on ovl_inode code that was introduced in kernel v4.13.
fs/overlayfs/inode.c +28 -11
@@ -606,6 +606,16 @@ static int ovl_inode_set(struct inode *inode, void *data)static bool ovl_verify_inode (struct inode *inode,struct dentry *lowerdentry,struct dentry *upperdentry) { +if (S_ISDIR (inode->i_mode)) { +/* Real lower dir moved to upper layer under us? */ +if (!lowerdentry &&ovl_inode_lower (inode)) +return false; + +/* Lookup of an uncovered redirect origin? */ +if (!upperdentry &&ovl_inode_upper (inode)) +return false; + } +/* * Allow non-NULL lower inode in ovl_inode even if lowerdentry is NULL. * This happens when finding a copied up overlay inode for a renamed
@@ -633,6 +643,8 @@ struct inode *ovl_get_inode(struct dentry *dentry, struct dentry *upperdentry,struct inode *inode;/* Already indexed or could be indexed on copy up? */ bool indexed = (index || (ovl_indexdir (dentry->d_sb) && !upperdentry)); +struct dentry *origin = indexed ? lowerdentry :NULL ; +bool is_dir;if (WARN_ON (upperdentry && indexed && !lowerdentry))return ERR_PTR (-EIO );
@@ -641,15 +653,19 @@ struct inode *ovl_get_inode(struct dentry *dentry, struct dentry *upperdentry, realinode =d_inode (lowerdentry);/* -* Copy up origin (lower) may exist for non-indexed upper, but we must -* not use lower as hash key in that case. -* Hash inodes that are or could be indexed by origin inode and -* non-indexed upper inodes that could be hard linked by upper inode. +* Copy up origin (lower) may exist for non-indexed non-dir upper, but +* we must not use lower as hash key in that case. +* Hash non-dir that is or could be indexed by origin inode. +* Hash dir that is or could be merged by origin inode. +* Hash pure upper and non-indexed non-dir by upper inode. */ -if (!S_ISDIR (realinode->i_mode) && (upperdentry || indexed)) { -struct inode *key =d_inode (indexed ? lowerdentry : - upperdentry); -unsigned int nlink; + is_dir =S_ISDIR (realinode->i_mode); +if (is_dir) + origin = lowerdentry; + +if (upperdentry || origin) { +struct inode *key =d_inode (origin ?: upperdentry); +unsigned int nlink = is_dir ?1 : realinode->i_nlink; inode =iget5_locked (dentry->d_sb, (unsigned long ) key, ovl_inode_test, ovl_inode_set, key);
@@ -670,8 +686,9 @@ struct inode *ovl_get_inode(struct dentry *dentry, struct dentry *upperdentry, goto out; } - nlink =ovl_get_nlink (lowerdentry, upperdentry, - realinode->i_nlink); +/* Recalculate nlink for non-dir due to indexing */ +if (!is_dir) + nlink =ovl_get_nlink (lowerdentry, upperdentry, nlink);set_nlink (inode, nlink); }else { inode =new_inode (dentry->d_sb);
@@ -685,7 +702,7 @@ struct inode *ovl_get_inode(struct dentry *dentry, struct dentry *upperdentry,ovl_set_flag (OVL_IMPURE , inode);/* Check for non-merge dir that may have whiteouts */ -if (S_ISDIR (realinode->i_mode)) { +if (is_dir) {struct ovl_entry *oe = dentry->d_fsdata; if (((upperdentry && lowerdentry) || oe->numlower > 1) ||
fs/overlayfs/super.c +1 -0
@@ -211,6 +211,7 @@ static void ovl_destroy_inode(struct inode *inode)struct ovl_inode *oi =OVL_I (inode);dput (oi->__upperdentry); +iput (oi->lower);kfree (oi->redirect);ovl_dir_cache_free (inode);mutex_destroy (&oi->lock);
fs/overlayfs/util.c +2 -2
@@ -257,7 +257,7 @@ void ovl_inode_init(struct inode *inode, struct dentry *upperdentry,if (upperdentry)OVL_I (inode)->__upperdentry = upperdentry;if (lowerdentry) -OVL_I (inode)->lower =d_inode (lowerdentry); +OVL_I (inode)->lower =igrab (d_inode (lowerdentry));ovl_copyattr (d_inode (upperdentry ?: lowerdentry), inode); }
@@ -273,7 +273,7 @@ void ovl_inode_update(struct inode *inode, struct dentry *upperdentry)*/ smp_wmb ();OVL_I (inode)->__upperdentry = upperdentry; -if (!S_ISDIR (upperinode->i_mode) &&inode_unhashed (inode)) { +if (inode_unhashed (inode)) { inode->i_private = upperinode;__insert_inode_hash (inode, (unsigned long ) upperinode); }
Quoted 6 lines · On Thu, Feb 15, 2018 at 4:31 PM, <[email protected]> wrote:
The patch below does not apply to the 4.14-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.
Hi Niklas,
The conflict resolution of this patch for v4.14 is trivial (just dropping last hunk of inode.c patch), but I no means to test this right now.
Are you able and/or interested to test the attached backport path for stable kernel v4.14?
I have posted an LTP test (inotify07) to test this fix if anyone else is interested in testing:
https://github.com/linux-test-project/ltp/pull/246
Thanks,
Amir.
Quoted 141 lines
------------------ original commit in Linus's tree ------------------
From
31747eda41ef3c30c09c5c096b380bf54013746aMon Sep 17 00:00:00 2001
From: Amir Goldstein <[email protected]>
Date: Sun, 14 Jan 2018 18:35:40 +0200
Subject: [PATCH] ovl: hash directory inodes for fsnotifyfsnotify pins a watched directory inode in cache, but if directory dentry is released, new lookup will allocate a new dentry and a new inode. Directory events will be notified on the new inode, while fsnotify listener is watching the old pinned inode.
Hash all directory inodes to reuse the pinned inode on lookup. Pure upper dirs are hashes by real upper inode, merge and lower dirs are hashed by real lower inode.
The reference to lower inode was being held by the lower dentry object in the overlay dentry (oe->lowerstack[0]). Releasing the overlay dentry may drop lower inode refcount to zero. Add a refcount on behalf of the overlay inode to prevent that.
As a by-product, hashing directory inodes also detects multiple redirected dirs to the same lower dir and uncovered redirected dir target on and returns
-ESTALEon lookup.The reported issue dates back to initial version of overlayfs, but this patch depends on
ovl_inodecode that was introduced in kernel v4.13.Cc <[email protected]> #v4.13Reported-by Niklas Cassel <[email protected]>Signed-off-by Amir Goldstein <[email protected]>Signed-off-by Miklos Szeredi <[email protected]>Tested-by Niklas Cassel <[email protected]>fs/overlayfs/inode.c +28
-11@@ -606,6 +606,16 @@ static int ovl_inode_set(struct inode *inode, void *data)static bool ovl_verify_inode (struct inode *inode,struct dentry *lowerdentry,struct dentry *upperdentry) { +if (S_ISDIR (inode->i_mode)) { +/* Real lower dir moved to upper layer under us? */ +if (!lowerdentry &&ovl_inode_lower (inode)) +return false; + +/* Lookup of an uncovered redirect origin? */ +if (!upperdentry &&ovl_inode_upper (inode)) +return false; + } +/* * Allow non-NULL lower inode in ovl_inode even if lowerdentry is NULL. * This happens when finding a copied up overlay inode for a renamed @@ -633,6 +643,8 @@ struct inode *ovl_get_inode(struct dentry *dentry, struct dentry *upperdentry,struct inode *inode;/* Already indexed or could be indexed on copy up? */ bool indexed = (index || (ovl_indexdir (dentry->d_sb) && !upperdentry)); +struct dentry *origin = indexed ? lowerdentry :NULL ; +bool is_dir;if (WARN_ON (upperdentry && indexed && !lowerdentry))return ERR_PTR (-EIO );@@ -641,15 +653,19 @@ struct inode *ovl_get_inode(struct dentry *dentry, struct dentry *upperdentry, realinode =d_inode (lowerdentry);/* -* Copy up origin (lower) may exist for non-indexed upper, but we must -* not use lower as hash key in that case. -* Hash inodes that are or could be indexed by origin inode and -* non-indexed upper inodes that could be hard linked by upper inode. +* Copy up origin (lower) may exist for non-indexed non-dir upper, but +* we must not use lower as hash key in that case. +* Hash non-dir that is or could be indexed by origin inode. +* Hash dir that is or could be merged by origin inode. +* Hash pure upper and non-indexed non-dir by upper inode. */ -if (!S_ISDIR (realinode->i_mode) && (upperdentry || indexed)) { -struct inode *key =d_inode (indexed ? lowerdentry : - upperdentry); -unsigned int nlink; + is_dir =S_ISDIR (realinode->i_mode); +if (is_dir) + origin = lowerdentry; + +if (upperdentry || origin) { +struct inode *key =d_inode (origin ?: upperdentry); +unsigned int nlink = is_dir ?1 : realinode->i_nlink; inode =iget5_locked (dentry->d_sb, (unsigned long ) key, ovl_inode_test, ovl_inode_set, key);@@ -670,8 +686,9 @@ struct inode *ovl_get_inode(struct dentry *dentry, struct dentry *upperdentry, goto out; } - nlink =ovl_get_nlink (lowerdentry, upperdentry, - realinode->i_nlink); +/* Recalculate nlink for non-dir due to indexing */ +if (!is_dir) + nlink =ovl_get_nlink (lowerdentry, upperdentry, nlink);set_nlink (inode, nlink); }else { inode =new_inode (dentry->d_sb);@@ -685,7 +702,7 @@ struct inode *ovl_get_inode(struct dentry *dentry, struct dentry *upperdentry,ovl_set_flag (OVL_IMPURE , inode);/* Check for non-merge dir that may have whiteouts */ -if (S_ISDIR (realinode->i_mode)) { +if (is_dir) {struct ovl_entry *oe = dentry->d_fsdata; if (((upperdentry && lowerdentry) || oe->numlower > 1) ||fs/overlayfs/super.c +1
-0@@ -211,6 +211,7 @@ static void ovl_destroy_inode(struct inode *inode)struct ovl_inode *oi =OVL_I (inode);dput (oi->__upperdentry); +iput (oi->lower);kfree (oi->redirect);ovl_dir_cache_free (inode);mutex_destroy (&oi->lock);fs/overlayfs/util.c +1
-1@@ -257,7 +257,7 @@ void ovl_inode_init(struct inode *inode, struct dentry *upperdentry,if (upperdentry)OVL_I (inode)->__upperdentry = upperdentry;if (lowerdentry) -OVL_I (inode)->lower =d_inode (lowerdentry); +OVL_I (inode)->lower =igrab (d_inode (lowerdentry));ovl_copyattr (d_inode (upperdentry ?: lowerdentry), inode); }
fs/overlayfs/util.c void ovl_inode_update(struct inode *inode, struct dentry *upperdentry) · line 279
Quoted 8 lines
fs/overlayfs/util.c +1
-1@@ -273,7 +273,7 @@ void ovl_inode_update(struct inode *inode, struct dentry *upperdentry)*/ smp_wmb ();OVL_I (inode)->__upperdentry = upperdentry; -if (!S_ISDIR (upperinode->i_mode) &&inode_unhashed (inode)) { +if (inode_unhashed (inode)) { inode->i_private = upperinode;__insert_inode_hash (inode, (unsigned long ) upperinode); }
From 0b7e49bedb28e1d3a9653be2c57545f074481faf Mon Sep 17 00:00:00 2001
From: Amir Goldstein <[email protected]>
Date: Sun, 14 Jan 2018 18:35:40 +0200
Subject: [PATCH] ovl: hash directory inodes for fsnotify
fsnotify pins a watched directory inode in cache, but if directory dentry is released, new lookup will allocate a new dentry and a new inode. Directory events will be notified on the new inode, while fsnotify listener is watching the old pinned inode.
Hash all directory inodes to reuse the pinned inode on lookup. Pure upper dirs are hashes by real upper inode, merge and lower dirs are hashed by real lower inode.
The reference to lower inode was being held by the lower dentry object in the overlay dentry (oe->lowerstack[0]). Releasing the overlay dentry may drop lower inode refcount to zero. Add a refcount on behalf of the overlay inode to prevent that.
As a by-product, hashing directory inodes also detects multiple redirected dirs to the same lower dir and uncovered redirected dir target on and returns -ESTALE on lookup.
The reported issue dates back to initial version of overlayfs, but this patch depends on ovl_inode code that was introduced in kernel v4.13.
--- fs/overlayfs/inode.c | 37 +++++++++++++++++++++++++++---------- fs/overlayfs/super.c | 1 + fs/overlayfs/util.c | 4 ++-- 3 files changed, 30 insertions(+), 12 deletions(-)
fs/overlayfs/inode.c +27 -10
@@ -579,6 +579,16 @@ static int ovl_inode_set(struct inode *inode, void *data)static bool ovl_verify_inode (struct inode *inode,struct dentry *lowerdentry,struct dentry *upperdentry) { +if (S_ISDIR (inode->i_mode)) { +/* Real lower dir moved to upper layer under us? */ +if (!lowerdentry &&ovl_inode_lower (inode)) +return false; + +/* Lookup of an uncovered redirect origin? */ +if (!upperdentry &&ovl_inode_upper (inode)) +return false; + } +/* * Allow non-NULL lower inode in ovl_inode even if lowerdentry is NULL. * This happens when finding a copied up overlay inode for a renamed
@@ -606,6 +616,8 @@ struct inode *ovl_get_inode(struct dentry *dentry, struct dentry *upperdentry,struct inode *inode;/* Already indexed or could be indexed on copy up? */ bool indexed = (index || (ovl_indexdir (dentry->d_sb) && !upperdentry)); +struct dentry *origin = indexed ? lowerdentry :NULL ; +bool is_dir;if (WARN_ON (upperdentry && indexed && !lowerdentry))return ERR_PTR (-EIO );
@@ -614,15 +626,19 @@ struct inode *ovl_get_inode(struct dentry *dentry, struct dentry *upperdentry, realinode =d_inode (lowerdentry);/* -* Copy up origin (lower) may exist for non-indexed upper, but we must -* not use lower as hash key in that case. -* Hash inodes that are or could be indexed by origin inode and -* non-indexed upper inodes that could be hard linked by upper inode. +* Copy up origin (lower) may exist for non-indexed non-dir upper, but +* we must not use lower as hash key in that case. +* Hash non-dir that is or could be indexed by origin inode. +* Hash dir that is or could be merged by origin inode. +* Hash pure upper and non-indexed non-dir by upper inode. */ -if (!S_ISDIR (realinode->i_mode) && (upperdentry || indexed)) { -struct inode *key =d_inode (indexed ? lowerdentry : - upperdentry); -unsigned int nlink; + is_dir =S_ISDIR (realinode->i_mode); +if (is_dir) + origin = lowerdentry; + +if (upperdentry || origin) { +struct inode *key =d_inode (origin ?: upperdentry); +unsigned int nlink = is_dir ?1 : realinode->i_nlink; inode =iget5_locked (dentry->d_sb, (unsigned long ) key, ovl_inode_test, ovl_inode_set, key);
@@ -643,8 +659,9 @@ struct inode *ovl_get_inode(struct dentry *dentry, struct dentry *upperdentry, goto out; } - nlink =ovl_get_nlink (lowerdentry, upperdentry, - realinode->i_nlink); +/* Recalculate nlink for non-dir due to indexing */ +if (!is_dir) + nlink =ovl_get_nlink (lowerdentry, upperdentry, nlink);set_nlink (inode, nlink); }else { inode =new_inode (dentry->d_sb);
fs/overlayfs/super.c +1 -0
@@ -200,6 +200,7 @@ static void ovl_destroy_inode(struct inode *inode)struct ovl_inode *oi =OVL_I (inode);dput (oi->__upperdentry); +iput (oi->lower);kfree (oi->redirect);ovl_dir_cache_free (inode);mutex_destroy (&oi->lock);
fs/overlayfs/util.c +2 -2
@@ -253,7 +253,7 @@ void ovl_inode_init(struct inode *inode, struct dentry *upperdentry,if (upperdentry)OVL_I (inode)->__upperdentry = upperdentry;if (lowerdentry) -OVL_I (inode)->lower =d_inode (lowerdentry); +OVL_I (inode)->lower =igrab (d_inode (lowerdentry));ovl_copyattr (d_inode (upperdentry ?: lowerdentry), inode); }
@@ -269,7 +269,7 @@ void ovl_inode_update(struct inode *inode, struct dentry *upperdentry)*/ smp_wmb ();OVL_I (inode)->__upperdentry = upperdentry; -if (!S_ISDIR (upperinode->i_mode) &&inode_unhashed (inode)) { +if (inode_unhashed (inode)) { inode->i_private = upperinode;__insert_inode_hash (inode, (unsigned long ) upperinode); }
Signature · 1 line
2.7.4
Quoted 16 lines · Amir Goldstein
Quoted 6 lines · On Thu, Feb 15, 2018 at 4:31 PM, <[email protected]> wrote:
The patch below does not apply to the 4.14-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.Hi Niklas,
The conflict resolution of this patch for v4.14 is trivial (just dropping last hunk of
inode.cpatch), but I no means to test this right now.Are you able and/or interested to test the attached backport path for stable kernel v4.14?
Quoted 6 lines
I have posted an LTP test (inotify07) to test this fix if anyone else is interested in testing:
https://github.com/linux-test-project/ltp/pull/246Thanks,
Amir.
Hello Amir,
I have already backported+verified the fix to v4.13 in one of our internal trees.
Like you said, I simply dropped the last hunk of the patch that modfied inode.c, i.e. I simply ignored this hunk of the patch:
@@ -685,7 +702,7 @@ struct inode *ovl_get_inode(struct dentry *dentry, struct dentry *upperdentry,
ovl_set_flag(OVL_IMPURE, inode);
/* Check for non-merge dir that may have whiteouts */
- if (S_ISDIR(realinode->i_mode)) {
+ if (is_dir) {
struct ovl_entry *oe = dentry->d_fsdata;
if (((upperdentry && lowerdentry) || oe->numlower > 1) ||
Regards,
Niklas
Greg KH
maintainerQuoted 34 lines · Niklas Cassel
Quoted 16 lines · Amir Goldstein
Quoted 6 lines · On Thu, Feb 15, 2018 at 4:31 PM, <[email protected]> wrote:
The patch below does not apply to the 4.14-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.Hi Niklas,
The conflict resolution of this patch for v4.14 is trivial (just dropping last hunk of
inode.cpatch), but I no means to test this right now.Are you able and/or interested to test the attached backport path for stable kernel v4.14?
Quoted 6 lines
I have posted an LTP test (inotify07) to test this fix if anyone else is interested in testing:
https://github.com/linux-test-project/ltp/pull/246Thanks,
Amir.Hello Amir,
I have already backported+verified the fix to v4.13 in one of our internal trees.
Like you said, I simply dropped the last hunk of the patch that modfied
inode.c, i.e. I simply ignored this hunk of the patch:
Review struct inode *ovl_get_inode(struct dentry *dentry, struct dentry *upperdentry, · line 708
Quoted 8 lines
(quoted diff) +1
-1@@ -685,7 +702,7 @@ struct inode *ovl_get_inode(struct dentry *dentry, struct dentry *upperdentry, ovl_set_flag(OVL_IMPURE, inode); /* Check for non-merge dir that may have whiteouts */ - if (S_ISDIR(realinode->i_mode)) { + if (is_dir) { struct ovl_entry *oe = dentry->d_fsdata; if (((upperdentry && lowerdentry) || oe->numlower > 1) ||
Ok, last hunk dropped, thanks.
greg k-h