gfs2: Get rid of I_MUTEX_QUOTA usage
Subsystems (2)
Jan Kara
reviewerPATCH 1/1 gfs2: Get rid of I_MUTEX_QUOTA usage
GFS2 uses i_mutex on its system quota inode to synchronize writes to quota file. Since this is an internal inode to GFS2 (not part of directory hiearchy or visible by user) we are safe to define locking rules for it. So let's just get it its own locking class to make it clear.
--- fs/gfs2/ops_fstype.c | 8 ++++++++ fs/gfs2/quota.c | 2 +- 2 files changed, 9 insertions(+), 1 deletions(-)
So this is probably the simplest what can be done for GFS2 (actually OCFS2
does the same for internal system files). Compile tested only. Steven?
fs/gfs2/ops_fstype.c +8 -0
@@ -19,6 +19,7 @@#include <linux/mount.h> #include <linux/gfs2_ondisk.h> #include <linux/quotaops.h> +#include <linux/lockdep.h> #include "gfs2.h" #include "incore.h"
@@ -767,6 +768,7 @@ fail:return error; } +static struct lock_class_key gfs2_quota_imutex_key;static int init_inodes (struct gfs2_sbd *sdp,int undo) {
@@ -804,6 +806,12 @@ static int init_inodes(struct gfs2_sbd *sdp, int undo)fs_err (sdp,"can't get quota file inode: %d\n" , error); goto fail_rindex; } +/* +* i_mutex on quota files is special. Since this inode is hidden system +* file, we are safe to define locking ourselves. +*/ +lockdep_set_class (&sdp->sd_quota_inode->i_mutex, + &gfs2_quota_imutex_key); error =gfs2_rindex_update (sdp);if (error)
fs/gfs2/quota.c +1 -1
@@ -772,7 +772,7 @@ static int do_sync(unsigned int num_qd, struct gfs2_quota_data **qda)return -ENOMEM ;sort (qda, num_qd,sizeof (struct gfs2_quota_data *), sort_qd,NULL ); -mutex_lock_nested (&ip->i_inode.i_mutex,I_MUTEX_QUOTA ); +mutex_lock (&ip->i_inode.i_mutex);for (qx =0 ; qx < num_qd; qx++) { error =gfs2_glock_nq_init (qda[qx]->qd_gl,LM_ST_EXCLUSIVE ,GL_NOCACHE , &ghs[qx]);
Signature · 1 line
1.7.1
Hi,
Quoted 5 lines · Jan Kara
GFS2 uses
i_mutexon its system quota inode to synchronize writes to quota file. Since this is an internal inode to GFS2 (not part of directory hiearchy or visible by user) we are safe to define locking rules for it. So let's just get it its own locking class to make it clear.
It is visible to the user if the gfs2 metadata filesystem is mounted. That used to be the way in which the quotas were set by gfs2_quota. However that is really considered obsolete now, so maybe we don't have to support that interface any more... the gfs2_quota package is no longer in distros as the generic quota package does the job for us these days.
I'd just been looking at using sd_quota_mutex instead of the inode mutex, but run into that same issue,
Steve.
Quoted 56 lines
CC Steven Whitehouse <[email protected]>Signed-off-by Jan Kara <[email protected]>--- fs/gfs2/ops_fstype.c | 8 ++++++++ fs/gfs2/quota.c | 2 +- 2 files changed, 9 insertions(+), 1 deletions(-)So this is probably the simplest what can be done for GFS2 (actually OCFS2
does the same for internal system files). Compile tested only. Steven?fs/gfs2/ops_fstype.c +8
-0@@ -19,6 +19,7 @@#include <linux/mount.h> #include <linux/gfs2_ondisk.h> #include <linux/quotaops.h> +#include <linux/lockdep.h> #include "gfs2.h" #include "incore.h" @@ -767,6 +768,7 @@ fail:return error; } +static struct lock_class_key gfs2_quota_imutex_key;static int init_inodes (struct gfs2_sbd *sdp,int undo) {@@ -804,6 +806,12 @@ static int init_inodes(struct gfs2_sbd *sdp, int undo)fs_err (sdp,"can't get quota file inode: %d\n" , error); goto fail_rindex; } +/* +* i_mutex on quota files is special. Since this inode is hidden system +* file, we are safe to define locking ourselves. +*/ +lockdep_set_class (&sdp->sd_quota_inode->i_mutex, + &gfs2_quota_imutex_key); error =gfs2_rindex_update (sdp);if (error)fs/gfs2/quota.c +1
-1@@ -772,7 +772,7 @@ static int do_sync(unsigned int num_qd, struct gfs2_quota_data **qda)return -ENOMEM ;sort (qda, num_qd,sizeof (struct gfs2_quota_data *), sort_qd,NULL ); -mutex_lock_nested (&ip->i_inode.i_mutex,I_MUTEX_QUOTA ); +mutex_lock (&ip->i_inode.i_mutex);for (qx =0 ; qx < num_qd; qx++) { error =gfs2_glock_nq_init (qda[qx]->qd_gl,LM_ST_EXCLUSIVE ,GL_NOCACHE , &ghs[qx]);
Jan Kara
reviewerHello,
Quoted 6 lines · Steven Whitehouse
Quoted 5 lines · Jan Kara
GFS2 uses
i_mutexon its system quota inode to synchronize writes to quota file. Since this is an internal inode to GFS2 (not part of directory hiearchy or visible by user) we are safe to define locking rules for it. So let's just get it its own locking class to make it clear.
Quoted 6 lines
It is visible to the user if the gfs2 metadata filesystem is mounted. That used to be the way in which the quotas were set by
gfs2_quota. However that is really considered obsolete now, so maybe we don't have to support that interface any more... thegfs2_quotapackage is no longer in distros as the generic quota package does the job for us these days.
Ah, OK. But then if users could write to (or even truncate?) the quota
file, wasn't it really deadlockable? mutex_lock_nested would silence lockdep but the deadlock won't change. Another advantage of my change is that if there is some problem, lockdep will warn about it because all places which end up taking i_mutex on quota file will use the same locking class. Using of lockdep subclasses (i.e. the _nested variant of mutex_lock) essentially tells lockdep - I promise that uses of the lock tagged as one locking subclass cannot interact with uses tagged as another locking class.
Honza
Quoted 58 lines
Quoted 56 lines
CC Steven Whitehouse <[email protected]>Signed-off-by Jan Kara <[email protected]>--- fs/gfs2/ops_fstype.c | 8 ++++++++ fs/gfs2/quota.c | 2 +- 2 files changed, 9 insertions(+), 1 deletions(-)So this is probably the simplest what can be done for GFS2 (actually OCFS2
does the same for internal system files). Compile tested only. Steven?fs/gfs2/ops_fstype.c +8
-0@@ -19,6 +19,7 @@#include <linux/mount.h> #include <linux/gfs2_ondisk.h> #include <linux/quotaops.h> +#include <linux/lockdep.h> #include "gfs2.h" #include "incore.h" @@ -767,6 +768,7 @@ fail:return error; } +static struct lock_class_key gfs2_quota_imutex_key;static int init_inodes (struct gfs2_sbd *sdp,int undo) {@@ -804,6 +806,12 @@ static int init_inodes(struct gfs2_sbd *sdp, int undo)fs_err (sdp,"can't get quota file inode: %d\n" , error); goto fail_rindex; } +/* +* i_mutex on quota files is special. Since this inode is hidden system +* file, we are safe to define locking ourselves. +*/ +lockdep_set_class (&sdp->sd_quota_inode->i_mutex, + &gfs2_quota_imutex_key); error =gfs2_rindex_update (sdp);if (error)fs/gfs2/quota.c +1
-1@@ -772,7 +772,7 @@ static int do_sync(unsigned int num_qd, struct gfs2_quota_data **qda)return -ENOMEM ;sort (qda, num_qd,sizeof (struct gfs2_quota_data *), sort_qd,NULL ); -mutex_lock_nested (&ip->i_inode.i_mutex,I_MUTEX_QUOTA ); +mutex_lock (&ip->i_inode.i_mutex);for (qx =0 ; qx < num_qd; qx++) { error =gfs2_glock_nq_init (qda[qx]->qd_gl,LM_ST_EXCLUSIVE ,GL_NOCACHE , &ghs[qx]);
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Hi,
Quoted 25 lines · Jan Kara
Hello,
Quoted 6 lines · Steven Whitehouse
Quoted 5 lines · Jan Kara
GFS2 uses
i_mutexon its system quota inode to synchronize writes to quota file. Since this is an internal inode to GFS2 (not part of directory hiearchy or visible by user) we are safe to define locking rules for it. So let's just get it its own locking class to make it clear.Quoted 6 lines
It is visible to the user if the gfs2 metadata filesystem is mounted. That used to be the way in which the quotas were set by
gfs2_quota. However that is really considered obsolete now, so maybe we don't have to support that interface any more... thegfs2_quotapackage is no longer in distros as the generic quota package does the job for us these days.Ah, OK. But then if users could write to (or even truncate?) the quota
file, wasn't it really deadlockable?mutex_lock_nestedwould silence lockdep but the deadlock won't change. Another advantage of my change is that if there is some problem, lockdep will warn about it because all places which end up takingi_mutexon quota file will use the same locking class. Using of lockdep subclasses (i.e. the_nestedvariant ofmutex_lock) essentially tells lockdep - I promise that uses of the lock tagged as one locking subclass cannot interact with uses tagged as another locking class.
Quoted 2 lines
Honza
It has been fairly well tested over a period of time, so I'm pretty confident that it does work at the moment. The locking is not that obvious though and could certainly do with a clean up. I'd prefer to clean up the locking than just silence the messages.
I am taking a look at it currently to see what we might do. It may take me a little while to get something together though, bearing in mind all the possible cases that need checking. If we can use sd_quota_mutex everywhere, then that should at least reduce the complexity and remove the need for I_MUTEX_QUOTA, so thats my preferred solution if possible. Otherwise I'm open to suggestions,
Steve.
Quoted 58 lines
Quoted 58 lines
Quoted 56 lines
CC Steven Whitehouse <[email protected]>Signed-off-by Jan Kara <[email protected]>--- fs/gfs2/ops_fstype.c | 8 ++++++++ fs/gfs2/quota.c | 2 +- 2 files changed, 9 insertions(+), 1 deletions(-)So this is probably the simplest what can be done for GFS2 (actually OCFS2
does the same for internal system files). Compile tested only. Steven?fs/gfs2/ops_fstype.c +8
-0@@ -19,6 +19,7 @@#include <linux/mount.h> #include <linux/gfs2_ondisk.h> #include <linux/quotaops.h> +#include <linux/lockdep.h> #include "gfs2.h" #include "incore.h" @@ -767,6 +768,7 @@ fail:return error; } +static struct lock_class_key gfs2_quota_imutex_key;static int init_inodes (struct gfs2_sbd *sdp,int undo) {@@ -804,6 +806,12 @@ static int init_inodes(struct gfs2_sbd *sdp, int undo)fs_err (sdp,"can't get quota file inode: %d\n" , error); goto fail_rindex; } +/* +* i_mutex on quota files is special. Since this inode is hidden system +* file, we are safe to define locking ourselves. +*/ +lockdep_set_class (&sdp->sd_quota_inode->i_mutex, + &gfs2_quota_imutex_key); error =gfs2_rindex_update (sdp);if (error)fs/gfs2/quota.c +1
-1@@ -772,7 +772,7 @@ static int do_sync(unsigned int num_qd, struct gfs2_quota_data **qda)return -ENOMEM ;sort (qda, num_qd,sizeof (struct gfs2_quota_data *), sort_qd,NULL ); -mutex_lock_nested (&ip->i_inode.i_mutex,I_MUTEX_QUOTA ); +mutex_lock (&ip->i_inode.i_mutex);for (qx =0 ; qx < num_qd; qx++) { error =gfs2_glock_nq_init (qda[qx]->qd_gl,LM_ST_EXCLUSIVE ,GL_NOCACHE , &ghs[qx]);
Jan Kara
reviewerHi,
Quoted 33 lines · Steven Whitehouse
Quoted 25 lines · Jan Kara
Hello,
Quoted 6 lines · Steven Whitehouse
Quoted 5 lines · Jan Kara
GFS2 uses
i_mutexon its system quota inode to synchronize writes to quota file. Since this is an internal inode to GFS2 (not part of directory hiearchy or visible by user) we are safe to define locking rules for it. So let's just get it its own locking class to make it clear.Quoted 6 lines
It is visible to the user if the gfs2 metadata filesystem is mounted. That used to be the way in which the quotas were set by
gfs2_quota. However that is really considered obsolete now, so maybe we don't have to support that interface any more... thegfs2_quotapackage is no longer in distros as the generic quota package does the job for us these days.Ah, OK. But then if users could write to (or even truncate?) the quota
file, wasn't it really deadlockable?mutex_lock_nestedwould silence lockdep but the deadlock won't change. Another advantage of my change is that if there is some problem, lockdep will warn about it because all places which end up takingi_mutexon quota file will use the same locking class. Using of lockdep subclasses (i.e. the_nestedvariant ofmutex_lock) essentially tells lockdep - I promise that uses of the lock tagged as one locking subclass cannot interact with uses tagged as another locking class.Quoted 2 lines
Honza
It has been fairly well tested over a period of time, so I'm pretty confident that it does work at the moment. The locking is not that obvious though and could certainly do with a clean up. I'd prefer to clean up the locking than just silence the messages.
Quoted 6 lines
I am taking a look at it currently to see what we might do. It may take me a little while to get something together though, bearing in mind all the possible cases that need checking. If we can use
sd_quota_mutexeverywhere, then that should at least reduce the complexity and remove the need forI_MUTEX_QUOTA, so thats my preferred solution if possible. Otherwise I'm open to suggestions,
Ok, sounds good.
Honza
Quoted 60 lines
Quoted 58 lines
Quoted 58 lines
Quoted 56 lines
CC Steven Whitehouse <[email protected]>Signed-off-by Jan Kara <[email protected]>--- fs/gfs2/ops_fstype.c | 8 ++++++++ fs/gfs2/quota.c | 2 +- 2 files changed, 9 insertions(+), 1 deletions(-)So this is probably the simplest what can be done for GFS2 (actually OCFS2
does the same for internal system files). Compile tested only. Steven?fs/gfs2/ops_fstype.c +8
-0@@ -19,6 +19,7 @@#include <linux/mount.h> #include <linux/gfs2_ondisk.h> #include <linux/quotaops.h> +#include <linux/lockdep.h> #include "gfs2.h" #include "incore.h" @@ -767,6 +768,7 @@ fail:return error; } +static struct lock_class_key gfs2_quota_imutex_key;static int init_inodes (struct gfs2_sbd *sdp,int undo) {@@ -804,6 +806,12 @@ static int init_inodes(struct gfs2_sbd *sdp, int undo)fs_err (sdp,"can't get quota file inode: %d\n" , error); goto fail_rindex; } +/* +* i_mutex on quota files is special. Since this inode is hidden system +* file, we are safe to define locking ourselves. +*/ +lockdep_set_class (&sdp->sd_quota_inode->i_mutex, + &gfs2_quota_imutex_key); error =gfs2_rindex_update (sdp);if (error)fs/gfs2/quota.c +1
-1@@ -772,7 +772,7 @@ static int do_sync(unsigned int num_qd, struct gfs2_quota_data **qda)return -ENOMEM ;sort (qda, num_qd,sizeof (struct gfs2_quota_data *), sort_qd,NULL ); -mutex_lock_nested (&ip->i_inode.i_mutex,I_MUTEX_QUOTA ); +mutex_lock (&ip->i_inode.i_mutex);for (qx =0 ; qx < num_qd; qx++) { error =gfs2_glock_nq_init (qda[qx]->qd_gl,LM_ST_EXCLUSIVE ,GL_NOCACHE , &ghs[qx]);
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR