ocfs2: Fix quota file corruption
PATCH 1/1 ocfs2: Fix quota file corruption
Global quota files are accessed from different nodes and reading and writing of these files happens via block device page cache. Thus even though the access between nodes is properly serialized by quota file's inode lock we cannot rely on consistency of block device page cache between nodes. Indeed currently it is possible to corrupt quota files by creating and deleting quota structures from two nodes in parallel. Fix the problem by using OCFS2_BH_IGNORE_CACHE mount option when reading from quota file.
--- fs/ocfs2/quota_global.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
This is a quick fix for quota file corruption I have found during my testing
(to be used for 3.13 and -stable kernels). Longer term we likely want to do something more clever. Ideally we would invalidate relevant blocks from buffer cache when node releases quota file's inode lock (similarly as we do it for page cache). I wanted to do it similarly as for e.g. extent tree blocks but I failed to find how consistency of those between nodes is achieved. Can anyone point me in the right direction? Thanks in advance.
fs/ocfs2/quota_global.c +2 -1
@@ -151,7 +151,8 @@ int ocfs2_read_quota_phys_block(struct inode *inode, u64 p_block,int rc; *bhp =NULL ; - rc =ocfs2_read_blocks (INODE_CACHE (inode), p_block,1 , bhp,0 , + rc =ocfs2_read_blocks (INODE_CACHE (inode), p_block,1 , bhp, +OCFS2_BH_IGNORE_CACHE , ocfs2_validate_quota_block);if (rc)mlog_errno (rc);
Signature · 1 line
1.8.1.4
PATCH 1/1 ocfs2: Fix quota file corruption
Quoted 16 lines · Jan Kara
Global quota files are accessed from different nodes and reading and writing of these files happens via block device page cache. Thus even though the access between nodes is properly serialized by quota file's inode lock we cannot rely on consistency of block device page cache between nodes. Indeed currently it is possible to corrupt quota files by creating and deleting quota structures from two nodes in parallel. Fix the problem by using
OCFS2_BH_IGNORE_CACHEmount option when reading from quota file.CC Goldwyn Rodrigues <[email protected]>CC Mark Fasheh <[email protected]>Signed-off-by Jan Kara <[email protected]>--- fs/ocfs2/quota_global.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Quoted 7 lines
This is a quick fix for quota file corruption I have found during my testing
(to be used for 3.13 and -stable kernels). Longer term we likely want to do something more clever. Ideally we would invalidate relevant blocks from buffer cache when node releases quota file's inode lock (similarly as we do it for page cache). I wanted to do it similarly as for e.g. extent tree blocks but I failed to find how consistency of those between nodes is achieved. Can anyone point me in the right direction? Thanks in advance.
Hum, so this likely isn't the whole story because I can still see the
corruption. At least I've now found the metadata cache and basically understand how it works so I'm now reading code around that and trying to understand what screws up quota code...
Honza
Quoted 18 lines
fs/ocfs2/quota_global.c +2
-1@@ -151,7 +151,8 @@ int ocfs2_read_quota_phys_block(struct inode *inode, u64 p_block,int rc; *bhp =NULL ; - rc =ocfs2_read_blocks (INODE_CACHE (inode), p_block,1 , bhp,0 , + rc =ocfs2_read_blocks (INODE_CACHE (inode), p_block,1 , bhp, +OCFS2_BH_IGNORE_CACHE , ocfs2_validate_quota_block);if (rc)mlog_errno (rc);Signature · 1 line
1.8.1.4
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
PATCH 1/1 ocfs2: Fix quota file corruption
Hi Jan,
What errors do you see regarding corrupted quotas, something like below?
kernel: [119890.305935] __quota_error: 3 callbacks suppressed
kernel: [119890.305941] Quota error (device dm-8): find_free_dqentry:
Data block full but it shouldn't
kernel: [119890.306071] Quota error (device dm-8): qtree_write_dquot:
Error -5 occurred while creating quota
kernel: [119890.306203] (smbd,25964,2):ocfs2_acquire_dquot:817 ERROR:
status = -5
There seems to be a lack of documentation regarding this issue, let me know if you would like me to test anything for you.
Marty
Quoted 57 lines · Jan Kara
Quoted 16 lines · Jan Kara
Global quota files are accessed from different nodes and reading and writing of these files happens via block device page cache. Thus even though the access between nodes is properly serialized by quota file's inode lock we cannot rely on consistency of block device page cache between nodes. Indeed currently it is possible to corrupt quota files by creating and deleting quota structures from two nodes in parallel. Fix the problem by using
OCFS2_BH_IGNORE_CACHEmount option when reading from quota file.CC Goldwyn Rodrigues <[email protected]>CC Mark Fasheh <[email protected]>Signed-off-by Jan Kara <[email protected]>--- fs/ocfs2/quota_global.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)Quoted 7 lines
This is a quick fix for quota file corruption I have found during my testing
(to be used for 3.13 and -stable kernels). Longer term we likely want to do something more clever. Ideally we would invalidate relevant blocks from buffer cache when node releases quota file's inode lock (similarly as we do it for page cache). I wanted to do it similarly as for e.g. extent tree blocks but I failed to find how consistency of those between nodes is achieved. Can anyone point me in the right direction? Thanks in advance.Hum, so this likely isn't the whole story because I can still see the
corruption. At least I've now found the metadata cache and basically understand how it works so I'm now reading code around that and trying to understand what screws up quota code...Honza
Quoted 16 lines
fs/ocfs2/quota_global.c +2
-2@@ -151,7 +151,8 @@ int ocfs2_read_quota_phys_block(struct inode *inode, u64 p_block,int rc; *bhp =NULL ; - rc =ocfs2_read_blocks (INODE_CACHE (inode), p_block,1 , bhp,0 , + rc =ocfs2_read_blocks (INODE_CACHE (inode), p_block,1 , bhp, +OCFS2_BH_IGNORE_CACHE , ocfs2_validate_quota_block);if (rc)mlog_errno (rc); --Quoted 2 lines
1.8.1.4
--
Jan Kara <[email protected]>
SUSE Labs, CR_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel at oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel
Signature · 1 line
Marty
PATCH 1/1 ocfs2: Fix quota file corruption
Hello,
Quoted 8 lines · Marty Sweet
What errors do you see regarding corrupted quotas, something like below?
kernel: [119890.305935]__quota_error: 3 callbacks suppressed
kernel: [119890.305941] Quota error (device dm-8):find_free_dqentry:
Data block full but it shouldn't
kernel: [119890.306071] Quota error (device dm-8):qtree_write_dquot:
Error -5 occurred while creating quota
kernel: [119890.306203] (smbd,25964,2):ocfs2_acquire_dquot:817 ERROR:
status = -5
Yes, exactly that. I have a simple reproducer and I'm debugging the
issue so I should find the culprit soon.
Honza
Quoted 63 lines · Jan Kara
Quoted 57 lines · Jan Kara
Quoted 16 lines
Global quota files are accessed from different nodes and reading and writing of these files happens via block device page cache. Thus even though the access between nodes is properly serialized by quota file's inode lock we cannot rely on consistency of block device page cache between nodes. Indeed currently it is possible to corrupt quota files by creating and deleting quota structures from two nodes in parallel. Fix the problem by using
OCFS2_BH_IGNORE_CACHEmount option when reading from quota file.CC Goldwyn Rodrigues <[email protected]>CC Mark Fasheh <[email protected]>Signed-off-by Jan Kara <[email protected]>--- fs/ocfs2/quota_global.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)Quoted 7 lines
This is a quick fix for quota file corruption I have found during my testing
(to be used for 3.13 and -stable kernels). Longer term we likely want to do something more clever. Ideally we would invalidate relevant blocks from buffer cache when node releases quota file's inode lock (similarly as we do it for page cache). I wanted to do it similarly as for e.g. extent tree blocks but I failed to find how consistency of those between nodes is achieved. Can anyone point me in the right direction? Thanks in advance.Hum, so this likely isn't the whole story because I can still see the
corruption. At least I've now found the metadata cache and basically understand how it works so I'm now reading code around that and trying to understand what screws up quota code...Honza
Quoted 16 lines
fs/ocfs2/quota_global.c +2
-2@@ -151,7 +151,8 @@ int ocfs2_read_quota_phys_block(struct inode *inode, u64 p_block,int rc; *bhp =NULL ; - rc =ocfs2_read_blocks (INODE_CACHE (inode), p_block,1 , bhp,0 , + rc =ocfs2_read_blocks (INODE_CACHE (inode), p_block,1 , bhp, +OCFS2_BH_IGNORE_CACHE , ocfs2_validate_quota_block);if (rc)mlog_errno (rc); --Quoted 2 lines
1.8.1.4
--
Jan Kara <[email protected]>
SUSE Labs, CR_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel at oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-develSignature · 1 line
Marty
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR