Fix some deadlocks in quota code and implement lockdep for cluster locks
Subsystems (3)
Jan Kara
reviewercover Fix some deadlocks in quota code and implement lockdep for cluster locks
Hi,
I'm resending this patch series. It's rediffed against linux-next branch of
Joel's git tree. The first four patches are obvious fixes of deadlocks in quota code and should go in as soon as possible. The other three patches implement lockdep support for OCFS2 cluster locks. So you can have a look whether the code make sence to you and possibly merge them. They should be NOP when lockdep is disabled and help a lot when debugging various locking problems (with these patches I found the problems fixed in the beginning of the series).
BTW: Currently lockdep complains about some configfs issue and turns itself
off so that's why I didn't get to finding more deadlocks ;)
Honza
Jan Kara
reviewerPATCH 1/1 ocfs2: Fix possible deadlock in ocfs2_global_read_dquot()
It is not possible to get a read lock and then try to get the same write lock in one thread as that can block on downconvert being requested by other node leading to deadlock. So first drop the quota lock for reading and only after that get it for writing.
--- fs/ocfs2/quota_global.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)
fs/ocfs2/quota_global.c +3 -1
@@ -421,6 +421,7 @@ int ocfs2_global_read_dquot(struct dquot *dquot)OCFS2_DQUOT (dquot)->dq_originodes = dquot->dq_dqb.dqb_curinodes;if (!dquot->dq_off) {/* No real quota entry? */ /* Upgrade to exclusive lock for allocation */ +ocfs2_qinfo_unlock (info,0 ); err =ocfs2_qinfo_lock (info,1 );if (err <0 ) goto out_qlock;
@@ -435,7 +436,8 @@ int ocfs2_global_read_dquot(struct dquot *dquot) out_qlock:if (ex)ocfs2_qinfo_unlock (info,1 ); -ocfs2_qinfo_unlock (info,0 ); +else +ocfs2_qinfo_unlock (info,0 ); out:if (err <0 )mlog_errno (err);
Signature · 1 line
1.6.0.2
PATCH 1/1 ocfs2: Fix possible deadlock in ocfs2_global_read_dquot()
Quoted 15 lines · Jan Kara
It is not possible to get a read lock and then try to get the same write lock in one thread as that can block on downconvert being requested by other node leading to deadlock. So first drop the quota lock for reading and only after that get it for writing.
Signed-off-by Jan Kara <[email protected]>--- fs/ocfs2/quota_global.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)fs/ocfs2/quota_global.c +0
-0
fs/ocfs2/quota_global.c int ocfs2_global_read_dquot(struct dquot *dquot) · line 427
Quoted 7 lines
fs/ocfs2/quota_global.c +1
-0@@ -421,6 +421,7 @@ int ocfs2_global_read_dquot(struct dquot *dquot)OCFS2_DQUOT (dquot)->dq_originodes = dquot->dq_dqb.dqb_curinodes;if (!dquot->dq_off) {/* No real quota entry? */ /* Upgrade to exclusive lock for allocation */ +ocfs2_qinfo_unlock (info,0 ); err =ocfs2_qinfo_lock (info,1 );if (err <0 ) goto out_qlock;
Can other nodes change the state of dq_off while you're
getting the exclusive? I'm wondering if you have to retry the qtree_read_dquot(), and I don't know enough about the code to say yes or no.
Joel
Signature · 7 lines
f/8 and be there. Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127
Jan Kara
reviewerPATCH 1/1 ocfs2: Fix possible deadlock in ocfs2_global_read_dquot()
Quoted 20 lines · Joel Becker
Quoted 19 lines · Jan Kara
It is not possible to get a read lock and then try to get the same write lock in one thread as that can block on downconvert being requested by other node leading to deadlock. So first drop the quota lock for reading and only after that get it for writing.
Signed-off-by Jan Kara <[email protected]>--- fs/ocfs2/quota_global.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)fs/ocfs2/quota_global.c +1
-0@@ -421,6 +421,7 @@ int ocfs2_global_read_dquot(struct dquot *dquot)OCFS2_DQUOT (dquot)->dq_originodes = dquot->dq_dqb.dqb_curinodes;if (!dquot->dq_off) {/* No real quota entry? */ /* Upgrade to exclusive lock for allocation */ +ocfs2_qinfo_unlock (info,0 );
Quoted 8 lines
fs/ocfs2/quota_global.c int ocfs2_global_read_dquot(struct dquot *dquot) · line 427
Quoted 3 lines
fs/ocfs2/quota_global.c +0
-0err =ocfs2_qinfo_lock (info,1 );if (err <0 ) goto out_qlock;Can other nodes change the state of
dq_offwhile you're
getting the exclusive? I'm wondering if you have to retry theqtree_read_dquot(), and I don't know enough about the code to say yes or no.
Good question. No, we don't have to restart it since we also hold a
global quotafile lock which protects us from other nodes adding the quota entry. There's a comment about it just above ocfs2_global_read_dquot()...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Jan Kara
reviewerPATCH 1/1 ocfs2: Fix lock inversion in ocfs2_local_read_info()
In reply to Jan KaraThis function is called with dqio_mutex held but it has to acquire lock from global quota file which ranks above this lock. This is not deadlockable lock inversion since this code path is take only during mount when noone else can race with us but let's clean this up to silence lockdep.
We just drop the dqio_mutex in the beginning of the function and reacquire it in the end since we don't need it - noone can race with us at this moment.
--- fs/ocfs2/quota_local.c | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-)
fs/ocfs2/quota_local.c +5 -0
@@ -655,6 +655,9 @@ static int ocfs2_local_read_info(struct super_block *sb, int type)struct ocfs2_quota_recovery *rec;int locked =0 ; +/* We don't need the lock and we have to acquire quota file locks +* which will later depend on this lock */ +mutex_unlock (&sb_dqopt (sb)->dqio_mutex); info->dqi_maxblimit =0x7fffffffffffffffLL ; info->dqi_maxilimit =0x7fffffffffffffffLL ; oinfo =kmalloc (sizeof (struct ocfs2_mem_dqinfo ),GFP_NOFS );
@@ -733,6 +736,7 @@ static int ocfs2_local_read_info(struct super_block *sb, int type) goto out_err; } +mutex_lock (&sb_dqopt (sb)->dqio_mutex);return 0 ; out_err:if (oinfo) {
@@ -746,6 +750,7 @@ out_err:kfree (oinfo); }brelse (bh); +mutex_lock (&sb_dqopt (sb)->dqio_mutex);return -1 ; }
Signature · 1 line
1.6.0.2
Jan Kara
reviewerPATCH 1/1 ocfs2: Fix possible deadlock with quotas in ocfs2_setattr()
In reply to Jan KaraWe called vfs_dq_transfer() with global quota file lock held. This can lead to deadlocks as if vfs_dq_transfer() has to allocate new quota structure, it calls ocfs2_dquot_acquire() which tries to get quota file lock again and this can block if another node requested the lock in the mean time.
Since we have to call vfs_dq_transfer() with transaction already started and quota file lock ranks above the transaction start, we cannot just rely on ocfs2_dquot_acquire() or ocfs2_dquot_release() on getting the lock if they need it. We fix the problem by acquiring pointers to all quota structures needed by vfs_dq_transfer() already before calling the function. By this we are sure that all quota structures are properly allocated and they can be freed only after we drop references to them. Thus we don't need quota file lock anywhere inside vfs_dq_transfer().
--- fs/ocfs2/file.c | 53 ++++++++++++++++++++++++++++++----------------------- 1 files changed, 30 insertions(+), 23 deletions(-)
fs/ocfs2/file.c +30 -23
@@ -894,9 +894,9 @@ int ocfs2_setattr(struct dentry *dentry, struct iattr *attr)struct ocfs2_super *osb =OCFS2_SB (sb);struct buffer_head *bh =NULL ;handle_t *handle =NULL ; -int locked[MAXQUOTAS ] = {0 ,0 }; -int credits, qtype; -struct ocfs2_mem_dqinfo *oinfo; +int qtype; +struct dquot *transfer_from[MAXQUOTAS ] = { }; +struct dquot *transfer_to[MAXQUOTAS ] = { };mlog_entry ("(0x%p, '%.*s')\n" , dentry, dentry->d_name.len, dentry->d_name.name);
@@ -969,30 +969,37 @@ int ocfs2_setattr(struct dentry *dentry, struct iattr *attr)if ((attr->ia_valid &ATTR_UID && attr->ia_uid != inode->i_uid) || (attr->ia_valid &ATTR_GID && attr->ia_gid != inode->i_gid)) { - credits =OCFS2_INODE_UPDATE_CREDITS ; +/* +* Gather pointers to quota structures so that allocation / +* freeing of quota structures happens here and not inside +* vfs_dq_transfer() where we have problems with lock ordering +*/ if (attr->ia_valid &ATTR_UID && attr->ia_uid != inode->i_uid &&OCFS2_HAS_RO_COMPAT_FEATURE (sb,OCFS2_FEATURE_RO_COMPAT_USRQUOTA )) { - oinfo =sb_dqinfo (sb,USRQUOTA )->dqi_priv; - status =ocfs2_lock_global_qf (oinfo,1 ); -if (status <0 ) + transfer_to[USRQUOTA ] =dqget (sb, attr->ia_uid, +USRQUOTA ); + transfer_from[USRQUOTA ] =dqget (sb, inode->i_uid, +USRQUOTA ); +if (!transfer_to[USRQUOTA ] || !transfer_from[USRQUOTA ]) { + status = -ESRCH ; goto bail_unlock; - credits +=ocfs2_calc_qinit_credits (sb,USRQUOTA ) + -ocfs2_calc_qdel_credits (sb,USRQUOTA ); - locked[USRQUOTA ] =1 ; + } }if (attr->ia_valid &ATTR_GID && attr->ia_gid != inode->i_gid &&OCFS2_HAS_RO_COMPAT_FEATURE (sb,OCFS2_FEATURE_RO_COMPAT_GRPQUOTA )) { - oinfo =sb_dqinfo (sb,GRPQUOTA )->dqi_priv; - status =ocfs2_lock_global_qf (oinfo,1 ); -if (status <0 ) + transfer_to[GRPQUOTA ] =dqget (sb, attr->ia_gid, +GRPQUOTA ); + transfer_from[GRPQUOTA ] =dqget (sb, inode->i_gid, +GRPQUOTA ); +if (!transfer_to[GRPQUOTA ] || !transfer_from[GRPQUOTA ]) { + status = -ESRCH ; goto bail_unlock; - credits +=ocfs2_calc_qinit_credits (sb,GRPQUOTA ) + -ocfs2_calc_qdel_credits (sb,GRPQUOTA ); - locked[GRPQUOTA ] =1 ; + } } - handle =ocfs2_start_trans (osb, credits); + handle =ocfs2_start_trans (osb,OCFS2_INODE_UPDATE_CREDITS + +2 *ocfs2_quota_trans_credits (sb));if (IS_ERR (handle)) { status =PTR_ERR (handle);mlog_errno (status);
@@ -1030,12 +1037,6 @@ int ocfs2_setattr(struct dentry *dentry, struct iattr *attr) bail_commit:ocfs2_commit_trans (osb, handle); bail_unlock: -for (qtype =0 ; qtype <MAXQUOTAS ; qtype++) { -if (!locked[qtype]) -continue ; - oinfo =sb_dqinfo (sb, qtype)->dqi_priv; -ocfs2_unlock_global_qf (oinfo,1 ); - }ocfs2_inode_unlock (inode,1 ); bail_unlock_rw:if (size_change)
@@ -1043,6 +1044,12 @@ bail_unlock_rw: bail:brelse (bh); +/* Release quota pointers in case we acquired them */ +for (qtype =0 ; qtype <MAXQUOTAS ; qtype++) { +dqput (transfer_to[qtype]); +dqput (transfer_from[qtype]); + } +if (!status && attr->ia_valid &ATTR_MODE ) { status =ocfs2_acl_chmod (inode);if (status <0 )
Signature · 1 line
1.6.0.2
Jan Kara
reviewerPATCH 1/1 ocfs2: Fix possible deadlock in quota recovery
In reply to Jan KaraIn ocfs2_finish_quota_recovery() we acquired global quota file lock and started recovering local quota file. During this process we need to get quota structures, which calls ocfs2_dquot_acquire() which gets global quota file lock again. This second lock can block in case some other node has requested the quota file lock in the mean time. Fix the problem by moving quota file locking down into the function where it is really needed. Then dqget() or dqput() won't be called with the lock held.
--- fs/ocfs2/quota_local.c | 16 +++++++++------- 1 files changed, 9 insertions(+), 7 deletions(-)
fs/ocfs2/quota_local.c +9 -7
@@ -444,10 +444,6 @@ static int ocfs2_recover_local_quota_file(struct inode *lqinode,mlog_entry ("ino=%lu type=%u" , (unsigned long )lqinode->i_ino, type); - status =ocfs2_lock_global_qf (oinfo,1 ); -if (status <0 ) - goto out; -list_for_each_entry_safe (rchunk, next, &(rec->r_list[type]), rc_list) { chunk = rchunk->rc_chunk; hbh =NULL ;
@@ -480,12 +476,18 @@ static int ocfs2_recover_local_quota_file(struct inode *lqinode, type); goto out_put_bh; } + status =ocfs2_lock_global_qf (oinfo,1 ); +if (status <0 ) { +mlog_errno (status); + goto out_put_dquot; + } + handle =ocfs2_start_trans (OCFS2_SB (sb),OCFS2_QSYNC_CREDITS );if (IS_ERR (handle)) { status =PTR_ERR (handle);mlog_errno (status); - goto out_put_dquot; + goto out_drop_lock; }mutex_lock (&sb_dqopt (sb)->dqio_mutex);spin_lock (&dq_data_lock);
@@ -523,6 +525,8 @@ static int ocfs2_recover_local_quota_file(struct inode *lqinode, out_commit:mutex_unlock (&sb_dqopt (sb)->dqio_mutex);ocfs2_commit_trans (OCFS2_SB (sb), handle); +out_drop_lock: +ocfs2_unlock_global_qf (oinfo,1 ); out_put_dquot:dqput (dquot); out_put_bh:
@@ -537,8 +541,6 @@ out_put_bh:if (status <0 )break ; } -ocfs2_unlock_global_qf (oinfo,1 ); -out:if (status <0 )free_recovery_list (&(rec->r_list[type]));mlog_exit (status);
Signature · 1 line
1.6.0.2
Jan Kara
reviewerPATCH 1/1 ocfs2: Correct ordering of ip_alloc_sem and localloc locks for directories
In reply to Jan KaraWe use ordering ip_alloc_sem -> local alloc locks in ocfs2_write_begin(). So change lock ordering in ocfs2_extend_dir() and ocfs2_expand_inline_dir() to also use this lock ordering.
--- fs/ocfs2/dir.c | 21 ++++++++++----------- 1 files changed, 10 insertions(+), 11 deletions(-)
fs/ocfs2/dir.c +10 -11
@@ -2900,6 +2900,8 @@ static int ocfs2_expand_inline_dir(struct inode *dir, struct buffer_head *di_bh, alloc =ocfs2_clusters_for_bytes (sb, bytes); dx_alloc =0 ; +down_write (&oi->ip_alloc_sem); +if (ocfs2_supports_indexed_dirs (osb)) { credits +=ocfs2_add_dir_index_credits (sb);
@@ -2940,8 +2942,6 @@ static int ocfs2_expand_inline_dir(struct inode *dir, struct buffer_head *di_bh, goto out; } -down_write (&oi->ip_alloc_sem); -/* * Prepare for worst case allocation scenario of two separate * extents in the unindexed tree.
@@ -2953,7 +2953,7 @@ static int ocfs2_expand_inline_dir(struct inode *dir, struct buffer_head *di_bh,if (IS_ERR (handle)) { ret =PTR_ERR (handle);mlog_errno (ret); - goto out_sem; + goto out; } if (vfs_dq_alloc_space_nodirty(dir,
@@ -3172,10 +3172,8 @@ out_commit:ocfs2_commit_trans (osb, handle); -out_sem: -up_write (&oi->ip_alloc_sem); - out: +up_write (&oi->ip_alloc_sem);if (data_ac)ocfs2_free_alloc_context (data_ac);if (meta_ac)
@@ -3322,11 +3320,15 @@ static int ocfs2_extend_dir(struct ocfs2_super *osb,brelse (new_bh); new_bh =NULL ; +down_write (&OCFS2_I (dir)->ip_alloc_sem); + drop_alloc_sem =1 ; dir_i_size =i_size_read (dir); credits =OCFS2_SIMPLE_DIR_EXTEND_CREDITS ; goto do_extend; } +down_write (&OCFS2_I (dir)->ip_alloc_sem); + drop_alloc_sem =1 ; dir_i_size =i_size_read (dir);mlog (0 ,"extending dir %llu (i_size = %lld)\n" , (unsigned long long )OCFS2_I (dir)->ip_blkno, dir_i_size);
@@ -3370,9 +3372,6 @@ do_extend: credits++;/* For attaching the new dirent block to the * dx_root */ -down_write (&OCFS2_I (dir)->ip_alloc_sem); - drop_alloc_sem =1 ; - handle =ocfs2_start_trans (osb, credits);if (IS_ERR (handle)) { status =PTR_ERR (handle);
@@ -3435,10 +3434,10 @@ bail_bh: *new_de_bh = new_bh;get_bh (*new_de_bh); bail: -if (drop_alloc_sem) -up_write (&OCFS2_I (dir)->ip_alloc_sem);if (handle)ocfs2_commit_trans (osb, handle); +if (drop_alloc_sem) +up_write (&OCFS2_I (dir)->ip_alloc_sem);if (data_ac)ocfs2_free_alloc_context (data_ac);
Signature · 1 line
1.6.0.2
PATCH 1/1 ocfs2: Correct ordering of ip_alloc_sem and localloc locks for directories
Quoted 5 lines · Jan Kara
We use ordering
ip_alloc_sem-> local alloc locks inocfs2_write_begin(). So change lock ordering inocfs2_extend_dir()andocfs2_expand_inline_dir()to also use this lock ordering.Signed-off-by Jan Kara <[email protected]>
--
Mark Fasheh
Jan Kara
reviewerPATCH 1/1 vfs: Set special lockdep map for dirs only if not set by fs
In reply to Jan KaraSome filesystems need to set lockdep map for i_mutex differently for different directories. For example OCFS2 has system directories (for orphan inode tracking and for gathering all system files like journal or quota files into a single place) which have different locking locking rules than standard directories. For a filesystem setting lockdep map is naturaly done when the inode is read but we have to modify unlock_new_inode() not to overwrite the lockdep map the filesystem has set.
--- fs/inode.c | 17 +++++++++++------ include/linux/lockdep.h | 15 +++++++++++++++ 2 files changed, 26 insertions(+), 6 deletions(-)
fs/inode.c +11 -6
@@ -654,12 +654,17 @@ void unlock_new_inode(struct inode *inode)if (inode->i_mode &S_IFDIR ) {struct file_system_type *type = inode->i_sb->s_type; -/* -* ensure nobody is actually holding i_mutex -*/ -mutex_destroy (&inode->i_mutex); -mutex_init (&inode->i_mutex); -lockdep_set_class (&inode->i_mutex, &type->i_mutex_dir_key); +/* Set new key only if filesystem hasn't already changed it */ +if (!lockdep_match_class (&inode->i_mutex, + &type->i_mutex_key)) { +/* +* ensure nobody is actually holding i_mutex +*/ +mutex_destroy (&inode->i_mutex); +mutex_init (&inode->i_mutex); +lockdep_set_class (&inode->i_mutex, + &type->i_mutex_dir_key); + } }#endif /*
include/linux/lockdep.h +15 -0
@@ -258,6 +258,16 @@ extern void lockdep_init_map(struct lockdep_map *lock, const char *name,#define lockdep_set_subclass (lock, sub) \ lockdep_init_map(&(lock)->dep_map, #lock, \ (lock)->dep_map.key, sub) +/* +* Compare locking classes +*/ +#define lockdep_match_class (lock, key) lockdep_match_key(&(lock)->dep_map, key) + +static inline int lockdep_match_key (struct lockdep_map *lock, +struct lock_class_key *key) +{ +return lock->key == key; +}/* * Acquire a lock.
@@ -326,6 +336,11 @@ static inline void lockdep_on(void)#define lockdep_set_class_and_subclass (lock, key, sub) \ do { (void)(key); } while (0)#define lockdep_set_subclass (lock, sub) do { } while (0) +/* +* We don't define lockdep_match_class() and lockdep_match_key() for !LOCKDEP +* case since the result is not well defined and the caller should rather +* #ifdef the call himself. +*/ # define INIT_LOCKDEP # define lockdep_reset () do { debug_locks = 1; } while (0)
Signature · 1 line
1.6.0.2
Peter Zijlstra
maintainerPATCH 1/1 vfs: Set special lockdep map for dirs only if not set by fs
Quoted 8 lines · Jan Kara
Some filesystems need to set lockdep map for
i_mutexdifferently for different directories. For example OCFS2 has system directories (for orphan inode tracking and for gathering all system files like journal or quota files into a single place) which have different locking locking rules than standard directories. For a filesystem setting lockdep map is naturaly done when the inode is read but we have to modifyunlock_new_inode()not to overwrite the lockdep map the filesystem has set.
Still looks good ;-)
Quoted 68 lines
CC mingo at redhat.comSigned-off-by Jan Kara <[email protected]>--- fs/inode.c | 17 +++++++++++------ include/linux/lockdep.h | 15 +++++++++++++++ 2 files changed, 26 insertions(+), 6 deletions(-)fs/inode.c +11
-6@@ -654,12 +654,17 @@ void unlock_new_inode(struct inode *inode)if (inode->i_mode &S_IFDIR ) {struct file_system_type *type = inode->i_sb->s_type; -/* -* ensure nobody is actually holding i_mutex -*/ -mutex_destroy (&inode->i_mutex); -mutex_init (&inode->i_mutex); -lockdep_set_class (&inode->i_mutex, &type->i_mutex_dir_key); +/* Set new key only if filesystem hasn't already changed it */ +if (!lockdep_match_class (&inode->i_mutex, + &type->i_mutex_key)) { +/* +* ensure nobody is actually holding i_mutex +*/ +mutex_destroy (&inode->i_mutex); +mutex_init (&inode->i_mutex); +lockdep_set_class (&inode->i_mutex, + &type->i_mutex_dir_key); + } }#endif /* include/linux/lockdep.h +15
-0@@ -258,6 +258,16 @@ extern void lockdep_init_map(struct lockdep_map *lock, const char *name,#define lockdep_set_subclass (lock, sub) \ lockdep_init_map(&(lock)->dep_map, #lock, \ (lock)->dep_map.key, sub) +/* +* Compare locking classes +*/ +#define lockdep_match_class (lock, key) lockdep_match_key(&(lock)->dep_map, key) + +static inline int lockdep_match_key (struct lockdep_map *lock, +struct lock_class_key *key) +{ +return lock->key == key; +}/* * Acquire a lock. @@ -326,6 +336,11 @@ static inline void lockdep_on(void)#define lockdep_set_class_and_subclass (lock, key, sub) \ do { (void)(key); } while (0)#define lockdep_set_subclass (lock, sub) do { } while (0) +/* +* We don't define lockdep_match_class() and lockdep_match_key() for !LOCKDEP +* case since the result is not well defined and the caller should rather +* #ifdef the call himself. +*/ # define INIT_LOCKDEP # define lockdep_reset () do { debug_locks = 1; } while (0)
Jan Kara
reviewerPATCH 1/1 ocfs2: Add lockdep annotations
In reply to Jan KaraAdd lockdep support to OCFS2. The support also covers all of the cluster locks except for open locks, journal locks, and local quotafile locks. These are special because they are acquired for a node, not for a particular process and lockdep cannot deal with such type of locking.
--- fs/ocfs2/dlmglue.c | 99 ++++++++++++++++++++++++++++++++++++++++++++-------- fs/ocfs2/dlmglue.h | 12 +++++-- fs/ocfs2/inode.c | 11 ++++++ fs/ocfs2/inode.h | 8 ++++ fs/ocfs2/namei.c | 15 +++++--- fs/ocfs2/ocfs2.h | 4 ++ fs/ocfs2/sysfile.c | 17 +++++++++ 7 files changed, 142 insertions(+), 24 deletions(-)
fs/ocfs2/dlmglue.c +84 -15
@@ -92,6 +92,9 @@ struct ocfs2_unblock_ctl {enum ocfs2_unblock_action unblock_action; }; +/* Lockdep class keys */ +struct lock_class_key lockdep_keys[OCFS2_NUM_LOCK_TYPES ]; +static int ocfs2_check_meta_downconvert (struct ocfs2_lock_res *lockres,int new_level);static void ocfs2_set_meta_lvb (struct ocfs2_lock_res *lockres);
@@ -313,9 +316,16 @@ static int ocfs2_lock_create(struct ocfs2_super *osb, u32 dlm_flags);static inline int ocfs2_may_continue_on_blocked_lock (struct ocfs2_lock_res *lockres,int wanted); -static void ocfs2_cluster_unlock (struct ocfs2_super *osb, -struct ocfs2_lock_res *lockres, -int level); +static void __ocfs2_cluster_unlock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level,unsigned long ip); +static inline void ocfs2_cluster_unlock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level) +{ +__ocfs2_cluster_unlock (osb, lockres, level, _RET_IP_); +} +static inline void ocfs2_generic_handle_downconvert_action (struct ocfs2_lock_res *lockres);static inline void ocfs2_generic_handle_convert_action (struct ocfs2_lock_res *lockres);static inline void ocfs2_generic_handle_attach_action (struct ocfs2_lock_res *lockres);
@@ -485,6 +495,13 @@ static void ocfs2_lock_res_init_common(struct ocfs2_super *osb,ocfs2_add_lockres_tracking (res, osb->osb_dlm_debug);ocfs2_init_lock_stats (res); +#ifdef CONFIG_DEBUG_LOCK_ALLOC +if (type !=OCFS2_LOCK_TYPE_OPEN ) +lockdep_init_map (&res->l_lockdep_map, ocfs2_lock_type_strings[type], + &lockdep_keys[type],0 ); +else + res->l_lockdep_map.key =NULL ; +#endif }void ocfs2_lock_res_init_once (struct ocfs2_lock_res *res)
@@ -1239,11 +1256,21 @@ static int ocfs2_wait_for_mask_interruptible(struct ocfs2_mask_waiter *mw,return ret; } -static int ocfs2_cluster_lock (struct ocfs2_super *osb, -struct ocfs2_lock_res *lockres, -int level, -u32 lkm_flags, -int arg_flags) +#ifdef CONFIG_DEBUG_LOCK_ALLOC +static int __ocfs2_cluster_lock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level, +u32 lkm_flags, +int arg_flags, +int l_subclass, + unsigned long ip) +#else +static int __ocfs2_cluster_lock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level, +u32 lkm_flags, + int arg_flags) +#endif {struct ocfs2_mask_waiter mw;int wait, catch_signals = !(osb->s_mount_opt &OCFS2_MOUNT_NOINTR );
@@ -1386,13 +1413,45 @@ out: }ocfs2_update_lock_stats (lockres, level, &mw, ret); +#ifdef CONFIG_DEBUG_LOCK_ALLOC +if (!ret && lockres->l_lockdep_map.key !=NULL ) { +if (level ==DLM_LOCK_PR ) +rwsem_acquire_read (&lockres->l_lockdep_map, l_subclass, + !!(arg_flags &OCFS2_META_LOCK_NOQUEUE ), ip); +else +rwsem_acquire (&lockres->l_lockdep_map, l_subclass, + !!(arg_flags &OCFS2_META_LOCK_NOQUEUE ), ip); + } +#endif mlog_exit (ret);return ret; } -static void ocfs2_cluster_unlock (struct ocfs2_super *osb, -struct ocfs2_lock_res *lockres, -int level) +static inline int ocfs2_cluster_lock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level, +u32 lkm_flags, +int arg_flags) +{ +#ifdef CONFIG_DEBUG_LOCK_ALLOC +return __ocfs2_cluster_lock (osb, lockres, level, lkm_flags, arg_flags, +0 , _RET_IP_); +#else +return __ocfs2_cluster_lock (osb, lockres, level, lkm_flags, arg_flags); +#endif +} + + +#ifdef CONFIG_DEBUG_LOCK_ALLOC +static void __ocfs2_cluster_unlock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level, + unsigned long ip) +#else +static void __ocfs2_cluster_unlock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, + int level) +#endif {unsigned long flags;
@@ -1401,6 +1460,10 @@ static void ocfs2_cluster_unlock(struct ocfs2_super *osb,ocfs2_dec_holders (lockres, level);ocfs2_downconvert_on_unlock (osb, lockres);spin_unlock_irqrestore (&lockres->l_lock, flags); +#ifdef CONFIG_DEBUG_LOCK_ALLOC +if (lockres->l_lockdep_map.key !=NULL ) +rwsem_release (&lockres->l_lockdep_map,1 , ip); +#endif mlog_exit_void (); }
@@ -2145,10 +2208,11 @@ static int ocfs2_assign_bh(struct inode *inode,* returns < 0 error if the callback will never be called, otherwise * the result of the lock will be communicated via the callback. */ -int ocfs2_inode_lock_full (struct inode *inode, -struct buffer_head **ret_bh, -int ex, -int arg_flags) +int ocfs2_inode_lock_full_nested (struct inode *inode, +struct buffer_head **ret_bh, +int ex, +int arg_flags, +int subclass) {int status, level, acquired;u32 dlm_flags;
@@ -2186,7 +2250,12 @@ int ocfs2_inode_lock_full(struct inode *inode,if (arg_flags &OCFS2_META_LOCK_NOQUEUE ) dlm_flags |=DLM_LKF_NOQUEUE ; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + status =__ocfs2_cluster_lock (osb, lockres, level, dlm_flags, + arg_flags, subclass, _RET_IP_); +#else status =ocfs2_cluster_lock (osb, lockres, level, dlm_flags, arg_flags); +#endif if (status <0 ) {if (status != -EAGAIN && status != -EIOCBRETRY )mlog_errno (status);
fs/ocfs2/dlmglue.h +9 -3
@@ -96,17 +96,23 @@ void ocfs2_open_unlock(struct inode *inode);int ocfs2_inode_lock_atime (struct inode *inode,struct vfsmount *vfsmnt,int *level); -int ocfs2_inode_lock_full (struct inode *inode, +int ocfs2_inode_lock_full_nested (struct inode *inode,struct buffer_head **ret_bh,int ex, -int arg_flags); +int arg_flags, +int subclass);int ocfs2_inode_lock_with_page (struct inode *inode,struct buffer_head **ret_bh,int ex,struct page *page); +/* Variants without special locking class or flags */ +#define ocfs2_inode_lock_full (i, r, e, f)\ + ocfs2_inode_lock_full_nested(i, r, e, f, 0) +#define ocfs2_inode_lock_nested (i, b, e, s)\ + ocfs2_inode_lock_full_nested(i, b, e, 0, s)/* 99% of the time we don't want to supply any additional flags -- * those are for very specific cases only. */ -#define ocfs2_inode_lock (i, b, e) ocfs2_inode_lock_full(i, b, e, 0) +#define ocfs2_inode_lock (i, b, e) ocfs2_inode_lock_full_nested(i, b, e, 0, 0)void ocfs2_inode_unlock (struct inode *inode,int ex); int ocfs2_super_lock(struct ocfs2_super *osb,
fs/ocfs2/inode.c +11 -0
@@ -215,6 +215,8 @@ bail:static int ocfs2_init_locked_inode (struct inode *inode,void *opaque) {struct ocfs2_find_inode_args *args = opaque; +static struct lock_class_key ocfs2_quota_ip_alloc_sem_key, + ocfs2_file_ip_alloc_sem_key;mlog_entry ("inode = %p, opaque = %p\n" , inode, opaque);
@@ -223,6 +225,15 @@ static int ocfs2_init_locked_inode(struct inode *inode, void *opaque)if (args->fi_sysfile_type !=0 )lockdep_set_class (&inode->i_mutex, &ocfs2_sysfile_lock_key[args->fi_sysfile_type]); +if (args->fi_sysfile_type ==USER_QUOTA_SYSTEM_INODE || + args->fi_sysfile_type ==GROUP_QUOTA_SYSTEM_INODE || + args->fi_sysfile_type ==LOCAL_USER_QUOTA_SYSTEM_INODE || + args->fi_sysfile_type ==LOCAL_GROUP_QUOTA_SYSTEM_INODE ) +lockdep_set_class (&OCFS2_I (inode)->ip_alloc_sem, + &ocfs2_quota_ip_alloc_sem_key); +else +lockdep_set_class (&OCFS2_I (inode)->ip_alloc_sem, + &ocfs2_file_ip_alloc_sem_key);mlog_exit (0 );return 0 ;
fs/ocfs2/inode.h +8 -0
@@ -109,6 +109,14 @@ struct ocfs2_inode_info/* Indicates that the metadata cache should be used as an array. */ #define OCFS2_INODE_CACHE_INLINE 0x00000080 +/* Locking subclasses of inode cluster lock */ +enum { +OI_LS_NORMAL , +OI_LS_PARENT , +OI_LS_RENAME1 , +OI_LS_RENAME2 , +}; +static inline struct ocfs2_inode_info *OCFS2_I (struct inode *inode) { return container_of(inode, struct ocfs2_inode_info, vfs_inode);
fs/ocfs2/namei.c +9 -6
@@ -118,7 +118,7 @@ static struct dentry *ocfs2_lookup(struct inode *dir, struct dentry *dentry,mlog (0 ,"find name %.*s in directory %llu\n" , dentry->d_name.len, dentry->d_name.name, (unsigned long long )OCFS2_I (dir)->ip_blkno); - status =ocfs2_inode_lock (dir,NULL ,0 ); + status =ocfs2_inode_lock_nested (dir,NULL ,0 ,OI_LS_PARENT );if (status <0 ) {if (status != -ENOENT )mlog_errno (status);
@@ -636,7 +636,7 @@ static int ocfs2_link(struct dentry *old_dentry,if (S_ISDIR (inode->i_mode))return -EPERM ; - err =ocfs2_inode_lock (dir, &parent_fe_bh,1 ); + err =ocfs2_inode_lock_nested (dir, &parent_fe_bh,1 ,OI_LS_PARENT );if (err <0 ) {if (err != -ENOENT )mlog_errno (err);
@@ -800,7 +800,8 @@ static int ocfs2_unlink(struct inode *dir,return -EPERM ; } - status =ocfs2_inode_lock (dir, &parent_node_bh,1 ); + status =ocfs2_inode_lock_nested (dir, &parent_node_bh,1 , +OI_LS_PARENT );if (status <0 ) {if (status != -ENOENT )mlog_errno (status);
@@ -978,7 +979,8 @@ static int ocfs2_double_lock(struct ocfs2_super *osb, inode1 = tmpinode; }/* lock id2 */ - status =ocfs2_inode_lock (inode2, bh2,1 ); + status =ocfs2_inode_lock_nested (inode2, bh2,1 , +OI_LS_RENAME1 );if (status <0 ) {if (status != -ENOENT )mlog_errno (status);
@@ -987,7 +989,7 @@ static int ocfs2_double_lock(struct ocfs2_super *osb, }/* lock id1 */ - status =ocfs2_inode_lock (inode1, bh1,1 ); + status =ocfs2_inode_lock_nested (inode1, bh1,1 ,OI_LS_RENAME2 );if (status <0 ) {/* * An error return must mean that no cluster locks
@@ -1103,7 +1105,8 @@ static int ocfs2_rename(struct inode *old_dir,* won't have to concurrently downconvert the inode and the * dentry locks. */ - status =ocfs2_inode_lock (old_inode, &old_inode_bh,1 ); + status =ocfs2_inode_lock_nested (old_inode, &old_inode_bh,1 , +OI_LS_PARENT );if (status <0 ) {if (status != -ENOENT )mlog_errno (status);
fs/ocfs2/ocfs2.h +4 -0
@@ -34,6 +34,7 @@#include <linux/workqueue.h> #include <linux/kref.h> #include <linux/mutex.h> +#include <linux/lockdep.h> #ifndef CONFIG_OCFS2_COMPAT_JBD # include <linux/jbd2.h> #else
@@ -149,6 +150,9 @@ struct ocfs2_lock_res {unsigned int l_lock_max_exmode;/* Max wait for EX */ unsigned int l_lock_refresh;/* Disk refreshes */ #endif +#ifdef CONFIG_DEBUG_LOCK_ALLOC +struct lockdep_map l_lockdep_map; +#endif };struct ocfs2_dlm_debug {
fs/ocfs2/sysfile.c +17 -0
@@ -50,6 +50,8 @@ static inline int is_in_system_inode_array(struct ocfs2_super *osb,int type, u32 slot); +static struct lock_class_key ocfs2_sysfile_cluster_lock_key[NUM_SYSTEM_INODES ]; +static inline int is_global_system_inode (int type) { return type >= OCFS2_FIRST_ONLINE_SYSTEM_INODE &&
@@ -118,6 +120,21 @@ static struct inode * _ocfs2_get_system_file_inode(struct ocfs2_super *osb, inode =NULL ; goto bail; } +#ifdef CONFIG_DEBUG_LOCK_ALLOC +if (type ==LOCAL_USER_QUOTA_SYSTEM_INODE || + type ==LOCAL_GROUP_QUOTA_SYSTEM_INODE || + type ==JOURNAL_SYSTEM_INODE ) { +/* Ignore inode lock on these inodes as the lock does not +* really belong to any process and lockdep cannot handle +* that */ +OCFS2_I (inode)->ip_inode_lockres.l_lockdep_map.key =NULL ; + }else { +lockdep_init_map (&OCFS2_I (inode)->ip_inode_lockres. + l_lockdep_map, + ocfs2_system_inodes[type].si_name, + &ocfs2_sysfile_cluster_lock_key[type],0 ); + } +#endif bail:return inode;
Signature · 1 line
1.6.0.2
PATCH 1/1 ocfs2: Add lockdep annotations
Quoted 4 lines · Jan Kara
Add lockdep support to OCFS2. The support also covers all of the cluster locks except for open locks, journal locks, and local quotafile locks. These are special because they are acquired for a node, not for a particular process and lockdep cannot deal with such type of locking.
I like the idea of ocfs2 having lockdep on everything.
fs/ocfs2/dlmglue.c static int ocfs2_lock_create(struct ocfs2_super *osb, · line 327
Quoted 12 lines
fs/ocfs2/dlmglue.c +9
-3-static void ocfs2_cluster_unlock (struct ocfs2_super *osb, -struct ocfs2_lock_res *lockres, -int level); +static void __ocfs2_cluster_unlock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level,unsigned long ip); +static inline void ocfs2_cluster_unlock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level) +{ +__ocfs2_cluster_unlock (osb, lockres, level, _RET_IP_); +}
I don't know why 'ip' needs to be part of the function prototype
- every place you use it, you just pass _RET_IP_.
Quoted 34 lines
fs/ocfs2/dlmglue.c +18
-5@@ -1239,11 +1256,21 @@ static int ocfs2_wait_for_mask_interruptible(struct ocfs2_mask_waiter *mw,return ret; } -static int ocfs2_cluster_lock (struct ocfs2_super *osb, -struct ocfs2_lock_res *lockres, -int level, -u32 lkm_flags, -int arg_flags) +#ifdef CONFIG_DEBUG_LOCK_ALLOC +static int __ocfs2_cluster_lock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level, +u32 lkm_flags, +int arg_flags, +int l_subclass, + unsigned long ip) +#else +static int __ocfs2_cluster_lock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level, +u32 lkm_flags, + int arg_flags) +#endif {struct ocfs2_mask_waiter mw;int wait, catch_signals = !(osb->s_mount_opt &OCFS2_MOUNT_NOINTR );@@ -1386,13 +1413,45 @@ out: }ocfs2_update_lock_stats (lockres, level, &mw, ret); +#ifdef CONFIG_DEBUG_LOCK_ALLOC +if (!ret && lockres->l_lockdep_map.key !=NULL ) { +if (level ==DLM_LOCK_PR )
fs/ocfs2/dlmglue.c out: · line 1428
Quoted 10 lines
fs/ocfs2/dlmglue.c +7
-0+rwsem_acquire_read (&lockres->l_lockdep_map, l_subclass, + !!(arg_flags &OCFS2_META_LOCK_NOQUEUE ), ip); + else + rwsem_acquire(&lockres->l_lockdep_map, l_subclass, + !!(arg_flags & OCFS2_META_LOCK_NOQUEUE), ip); + } +#endif mlog_exit (ret);return ret; }
Here again you've wrapped an ifdef around whether 'ip' is in the
argument list or not. Since it's always _RET_IP_, why not just leave it out of the argument list and just pas _RET_IP_ to the rwsem_acquire calls? That also eliminates the ugly ifdefs and need for __ocfs2_cluster_lock().
Quoted 14 lines
fs/ocfs2/dlmglue.c +10
-0+#ifdef CONFIG_DEBUG_LOCK_ALLOC +static void __ocfs2_cluster_unlock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level, + unsigned long ip) +#else +static void __ocfs2_cluster_unlock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, + int level) +#endif {unsigned long flags;
fs/ocfs2/dlmglue.c static void ocfs2_cluster_unlock(struct ocfs2_super *osb, · line 1468
Quoted 9 lines
fs/ocfs2/dlmglue.c +4
-0@@ -1401,6 +1460,10 @@ static void ocfs2_cluster_unlock(struct ocfs2_super *osb,ocfs2_dec_holders (lockres, level);ocfs2_downconvert_on_unlock (osb, lockres);spin_unlock_irqrestore (&lockres->l_lock, flags); +#ifdef CONFIG_DEBUG_LOCK_ALLOC +if (lockres->l_lockdep_map.key !=NULL ) +rwsem_release (&lockres->l_lockdep_map,1 , ip); +#endif mlog_exit_void (); }
Same here.
Quoted 17 lines
fs/ocfs2/dlmglue.c +5
-4@@ -2145,10 +2208,11 @@ static int ocfs2_assign_bh(struct inode *inode,* returns < 0 error if the callback will never be called, otherwise * the result of the lock will be communicated via the callback. */ -int ocfs2_inode_lock_full (struct inode *inode, -struct buffer_head **ret_bh, -int ex, -int arg_flags) +int ocfs2_inode_lock_full_nested (struct inode *inode, +struct buffer_head **ret_bh, +int ex, +int arg_flags, +int subclass) {int status, level, acquired;u32 dlm_flags;
fs/ocfs2/dlmglue.c int ocfs2_inode_lock_full(struct inode *inode, · line 2258
Quoted 9 lines
fs/ocfs2/dlmglue.c +5
-0@@ -2186,7 +2250,12 @@ int ocfs2_inode_lock_full(struct inode *inode,if (arg_flags &OCFS2_META_LOCK_NOQUEUE ) dlm_flags |=DLM_LKF_NOQUEUE ; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + status =__ocfs2_cluster_lock (osb, lockres, level, dlm_flags, + arg_flags, subclass, _RET_IP_); +#else status =ocfs2_cluster_lock (osb, lockres, level, dlm_flags, arg_flags); +#endif
This hunk wouldn't be necessary either.
Quoted 14 lines
fs/ocfs2/inode.c +2
-0@@ -215,6 +215,8 @@ bail:static int ocfs2_init_locked_inode (struct inode *inode,void *opaque) {struct ocfs2_find_inode_args *args = opaque; +static struct lock_class_key ocfs2_quota_ip_alloc_sem_key, + ocfs2_file_ip_alloc_sem_key;mlog_entry ("inode = %p, opaque = %p\n" , inode, opaque);@@ -223,6 +225,15 @@ static int ocfs2_init_locked_inode(struct inode *inode, void *opaque)if (args->fi_sysfile_type !=0 ) lockdep_set_class(&inode->i_mutex,
fs/ocfs2/inode.c static int ocfs2_init_locked_inode(struct inode *inode, void *opaque) · line 236
Quoted 10 lines
fs/ocfs2/inode.c +9
-0&ocfs2_sysfile_lock_key[args->fi_sysfile_type]); +if (args->fi_sysfile_type ==USER_QUOTA_SYSTEM_INODE || + args->fi_sysfile_type ==GROUP_QUOTA_SYSTEM_INODE || + args->fi_sysfile_type ==LOCAL_USER_QUOTA_SYSTEM_INODE || + args->fi_sysfile_type ==LOCAL_GROUP_QUOTA_SYSTEM_INODE ) +lockdep_set_class (&OCFS2_I (inode)->ip_alloc_sem, + &ocfs2_quota_ip_alloc_sem_key); +else +lockdep_set_class (&OCFS2_I (inode)->ip_alloc_sem, + &ocfs2_file_ip_alloc_sem_key);
These keys don't need to be intialized?
Quoted 6 lines
fs/ocfs2/inode.h +0
-0@@ -109,6 +109,14 @@ struct ocfs2_inode_info/* Indicates that the metadata cache should be used as an array. */
fs/ocfs2/inode.h struct ocfs2_inode_info · line 119
Quoted 10 lines
fs/ocfs2/inode.h +8
-0#define OCFS2_INODE_CACHE_INLINE 0x00000080 +/* Locking subclasses of inode cluster lock */ +enum { +OI_LS_NORMAL , +OI_LS_PARENT , +OI_LS_RENAME1 , +OI_LS_RENAME2 , +}; +
I couldn't see where OI_LS_NORMAL was ever used, until I
realized that your #define of ocfs2_inode_lock_full hardcodes OI_LS_NORMAL as '0'. It should really use the subclass name, right?
Joel
Signature · 8 lines
"Senator let's be sincere, As much as you can." Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127
Jan Kara
reviewerPATCH 1/1 ocfs2: Add lockdep annotations
Hi Joel,
thanks for review.
Quoted 8 lines · Joel Becker
Quoted 4 lines · Jan Kara
Add lockdep support to OCFS2. The support also covers all of the cluster locks except for open locks, journal locks, and local quotafile locks. These are special because they are acquired for a node, not for a particular process and lockdep cannot deal with such type of locking.
I like the idea of ocfs2 having lockdep on everything.
Quoted 15 lines
fs/ocfs2/dlmglue.c static int ocfs2_lock_create(struct ocfs2_super *osb, · line 327
Quoted 12 lines
fs/ocfs2/dlmglue.c +9
-3-static void ocfs2_cluster_unlock (struct ocfs2_super *osb, -struct ocfs2_lock_res *lockres, -int level); +static void __ocfs2_cluster_unlock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level,unsigned long ip); +static inline void ocfs2_cluster_unlock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level) +{ +__ocfs2_cluster_unlock (osb, lockres, level, _RET_IP_); +}I don't know why 'ip' needs to be part of the function prototype
- every place you use it, you just pass _RET_IP_.
The problem is following: I need to give to lockdep code an IP of a
function which acquires a lock. For this to be useful I don't want to show "ocfs2_cluster_lock" or "ocfs2_rw_lock" all the time. So I have to get IP of the function calling e.g. ocfs2_rw_lock and that's why there's this ip argument because it's hard to find what the IP of the caller of our caller is...
Quoted 42 lines
Quoted 42 lines
fs/ocfs2/dlmglue.c +25
-5@@ -1239,11 +1256,21 @@ static int ocfs2_wait_for_mask_interruptible(struct ocfs2_mask_waiter *mw,return ret; } -static int ocfs2_cluster_lock (struct ocfs2_super *osb, -struct ocfs2_lock_res *lockres, -int level, -u32 lkm_flags, -int arg_flags) +#ifdef CONFIG_DEBUG_LOCK_ALLOC +static int __ocfs2_cluster_lock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level, +u32 lkm_flags, +int arg_flags, +int l_subclass, + unsigned long ip) +#else +static int __ocfs2_cluster_lock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level, +u32 lkm_flags, + int arg_flags) +#endif {struct ocfs2_mask_waiter mw;int wait, catch_signals = !(osb->s_mount_opt &OCFS2_MOUNT_NOINTR );@@ -1386,13 +1413,45 @@ out: }ocfs2_update_lock_stats (lockres, level, &mw, ret); +#ifdef CONFIG_DEBUG_LOCK_ALLOC +if (!ret && lockres->l_lockdep_map.key !=NULL ) { +if (level ==DLM_LOCK_PR ) +rwsem_acquire_read (&lockres->l_lockdep_map, l_subclass, + !!(arg_flags &OCFS2_META_LOCK_NOQUEUE ), ip); +else +rwsem_acquire (&lockres->l_lockdep_map, l_subclass, + !!(arg_flags &OCFS2_META_LOCK_NOQUEUE ), ip); + } +#endif mlog_exit (ret);
Quoted 8 lines
Review out: · line 1428
Quoted 2 lines
(quoted diff) +0
-0return ret; }Here again you've wrapped an ifdef around whether 'ip' is in the
argument list or not. Since it's always _RET_IP_, why not just leave it out of the argument list and just pas _RET_IP_ to therwsem_acquirecalls? That also eliminates the ugly ifdefs and need for__ocfs2_cluster_lock().
I agree ifdefs aren't nice but I wanted to avoid the unnecessary argument
in !LOCKDEP case. I could also just always pass _RET_IP_ and hope that compiler optimizes this in case we don't use the value of ip argument in !LOCKDEP case.
I can do that if you think it's better.
Quoted 66 lines
Quoted 14 lines
fs/ocfs2/dlmglue.c +10
-0+#ifdef CONFIG_DEBUG_LOCK_ALLOC +static void __ocfs2_cluster_unlock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level, + unsigned long ip) +#else +static void __ocfs2_cluster_unlock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, + int level) +#endif {unsigned long flags;fs/ocfs2/dlmglue.c static void ocfs2_cluster_unlock(struct ocfs2_super *osb, · line 1468
Quoted 9 lines
fs/ocfs2/dlmglue.c +4
-0@@ -1401,6 +1460,10 @@ static void ocfs2_cluster_unlock(struct ocfs2_super *osb,ocfs2_dec_holders (lockres, level);ocfs2_downconvert_on_unlock (osb, lockres);spin_unlock_irqrestore (&lockres->l_lock, flags); +#ifdef CONFIG_DEBUG_LOCK_ALLOC +if (lockres->l_lockdep_map.key !=NULL ) +rwsem_release (&lockres->l_lockdep_map,1 , ip); +#endif mlog_exit_void (); }Same here.
Quoted 17 lines
fs/ocfs2/dlmglue.c +5
-4@@ -2145,10 +2208,11 @@ static int ocfs2_assign_bh(struct inode *inode,* returns < 0 error if the callback will never be called, otherwise * the result of the lock will be communicated via the callback. */ -int ocfs2_inode_lock_full (struct inode *inode, -struct buffer_head **ret_bh, -int ex, -int arg_flags) +int ocfs2_inode_lock_full_nested (struct inode *inode, +struct buffer_head **ret_bh, +int ex, +int arg_flags, +int subclass) {int status, level, acquired;u32 dlm_flags;fs/ocfs2/dlmglue.c int ocfs2_inode_lock_full(struct inode *inode, · line 2258
Quoted 9 lines
fs/ocfs2/dlmglue.c +5
-0@@ -2186,7 +2250,12 @@ int ocfs2_inode_lock_full(struct inode *inode,if (arg_flags &OCFS2_META_LOCK_NOQUEUE ) dlm_flags |=DLM_LKF_NOQUEUE ; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + status =__ocfs2_cluster_lock (osb, lockres, level, dlm_flags, + arg_flags, subclass, _RET_IP_); +#else status =ocfs2_cluster_lock (osb, lockres, level, dlm_flags, arg_flags); +#endif This hunk wouldn't be necessary either.
Quoted 11 lines
fs/ocfs2/inode.c +2
-0@@ -215,6 +215,8 @@ bail:static int ocfs2_init_locked_inode (struct inode *inode,void *opaque) {struct ocfs2_find_inode_args *args = opaque; +static struct lock_class_key ocfs2_quota_ip_alloc_sem_key, + ocfs2_file_ip_alloc_sem_key;mlog_entry ("inode = %p, opaque = %p\n" , inode, opaque);
Quoted 15 lines
fs/ocfs2/inode.c static int ocfs2_init_locked_inode(struct inode *inode, void *opaque) · line 236
Quoted 13 lines
fs/ocfs2/inode.c +9
-0@@ -223,6 +225,15 @@ static int ocfs2_init_locked_inode(struct inode *inode, void *opaque)if (args->fi_sysfile_type !=0 )lockdep_set_class (&inode->i_mutex, &ocfs2_sysfile_lock_key[args->fi_sysfile_type]); +if (args->fi_sysfile_type ==USER_QUOTA_SYSTEM_INODE || + args->fi_sysfile_type ==GROUP_QUOTA_SYSTEM_INODE || + args->fi_sysfile_type ==LOCAL_USER_QUOTA_SYSTEM_INODE || + args->fi_sysfile_type ==LOCAL_GROUP_QUOTA_SYSTEM_INODE ) +lockdep_set_class (&OCFS2_I (inode)->ip_alloc_sem, + &ocfs2_quota_ip_alloc_sem_key); +else +lockdep_set_class (&OCFS2_I (inode)->ip_alloc_sem, + &ocfs2_file_ip_alloc_sem_key);These keys don't need to be intialized?
No. The key here is the fixed adress of that variable - seeinclude/linux/lockdep.h:
/*
* Lock-classes are keyed via unique addresses, by embedding the
* lockclass-key into the kernel (or module) .data section. (For
* static locks we use the lock address itself as the key.) */
struct lockdep_subclass_key {
char __one_byte;
} __attribute__ ((__packed__));struct lock_class_key {
struct lockdep_subclass_key subkeys[MAX_LOCKDEP_SUBCLASSES];
};Quoted 8 lines
Quoted 8 lines
fs/ocfs2/inode.h +0
-0@@ -109,6 +109,14 @@ struct ocfs2_inode_info/* Indicates that the metadata cache should be used as an array. */ #define OCFS2_INODE_CACHE_INLINE 0x00000080
Quoted 12 lines
fs/ocfs2/inode.h struct ocfs2_inode_info · line 119
Quoted 8 lines
fs/ocfs2/inode.h +8
-0+/* Locking subclasses of inode cluster lock */ +enum { +OI_LS_NORMAL , +OI_LS_PARENT , +OI_LS_RENAME1 , +OI_LS_RENAME2 , +}; +I couldn't see where
OI_LS_NORMALwas ever used, until I
realized that your #define ofocfs2_inode_lock_fullhardcodesOI_LS_NORMALas '0'. It should really use the subclass name, right?
Ah, right. Thanks for spotting this. Fixed.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
PATCH 1/1 ocfs2: Add lockdep annotations
Quoted 24 lines · Jan Kara
Quoted 8 lines · Joel Becker
Quoted 4 lines · Jan Kara
Add lockdep support to OCFS2. The support also covers all of the cluster locks except for open locks, journal locks, and local quotafile locks. These are special because they are acquired for a node, not for a particular process and lockdep cannot deal with such type of locking.
I like the idea of ocfs2 having lockdep on everything.
Quoted 15 lines
fs/ocfs2/dlmglue.c static int ocfs2_lock_create(struct ocfs2_super *osb, · line 327
Quoted 12 lines
fs/ocfs2/dlmglue.c +9
-3-static void ocfs2_cluster_unlock (struct ocfs2_super *osb, -struct ocfs2_lock_res *lockres, -int level); +static void __ocfs2_cluster_unlock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level,unsigned long ip); +static inline void ocfs2_cluster_unlock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level) +{ +__ocfs2_cluster_unlock (osb, lockres, level, _RET_IP_); +}I don't know why 'ip' needs to be part of the function prototype
- every place you use it, you just pass _RET_IP_.
Quoted 6 lines
The problem is following: I need to give to lockdep code an IP of a
function which acquires a lock. For this to be useful I don't want to show "ocfs2_cluster_lock" or "ocfs2_rw_lock" all the time. So I have to get IP of the function calling e.g.ocfs2_rw_lockand that's why there's this ip argument because it's hard to find what the IP of the caller of our caller is...
Doesn't lockdep provide backtraces when things go wrong? Or is
this used to differentiate between callers internally in lockdep?
Quoted 50 lines
Quoted 42 lines
Quoted 42 lines
fs/ocfs2/dlmglue.c +25
-5@@ -1239,11 +1256,21 @@ static int ocfs2_wait_for_mask_interruptible(struct ocfs2_mask_waiter *mw,return ret; } -static int ocfs2_cluster_lock (struct ocfs2_super *osb, -struct ocfs2_lock_res *lockres, -int level, -u32 lkm_flags, -int arg_flags) +#ifdef CONFIG_DEBUG_LOCK_ALLOC +static int __ocfs2_cluster_lock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level, +u32 lkm_flags, +int arg_flags, +int l_subclass, + unsigned long ip) +#else +static int __ocfs2_cluster_lock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level, +u32 lkm_flags, + int arg_flags) +#endif {struct ocfs2_mask_waiter mw;int wait, catch_signals = !(osb->s_mount_opt &OCFS2_MOUNT_NOINTR );@@ -1386,13 +1413,45 @@ out: }ocfs2_update_lock_stats (lockres, level, &mw, ret); +#ifdef CONFIG_DEBUG_LOCK_ALLOC +if (!ret && lockres->l_lockdep_map.key !=NULL ) { +if (level ==DLM_LOCK_PR ) +rwsem_acquire_read (&lockres->l_lockdep_map, l_subclass, + !!(arg_flags &OCFS2_META_LOCK_NOQUEUE ), ip); +else +rwsem_acquire (&lockres->l_lockdep_map, l_subclass, + !!(arg_flags &OCFS2_META_LOCK_NOQUEUE ), ip); + } +#endif mlog_exit (ret);Quoted 8 lines
Review out: · line 1428
Quoted 2 lines
(quoted diff) +0
-0return ret; }Here again you've wrapped an ifdef around whether 'ip' is in the
argument list or not. Since it's always _RET_IP_, why not just leave it out of the argument list and just pas _RET_IP_ to therwsem_acquirecalls? That also eliminates the ugly ifdefs and need for__ocfs2_cluster_lock().
Quoted 5 lines
I agree ifdefs aren't nice but I wanted to avoid the unnecessary argument
in !LOCKDEP case. I could also just always pass _RET_IP_ and hope that compiler optimizes this in case we don't use the value of ip argument in !LOCKDEP case.
I can do that if you think it's better.
I would much rather you just rename to __ocfs2.., add the
argument, call it "caller_ip", and then wrap it with a static inline. This keeps the ifdefs in one place and much more readable. You can pass 0 when lockdep isn't enabled:
#ifdef CONFIG_DEBUG_LOCK_ALLOC
static inline int ocfs2_cluster_lock(struct ocfs2_super *osb,
struct ocfs2_lock_res *lockres,
int level,
u32 lkm_flags,
int arg_flags)
{
return __ocfs2_cluster_lock(osb, lockres, level, lkm_flags,
arg_flags, _RET_IP_);
}static inline void ocfs2_cluster_unlock(struct ocfs2_super *osb,
struct ocfs2_lock_res *lockres, int level)
{
return __ocfs2_cluster_unlock(osb, lockres, level, _RET_IP_);
}
#else
static inline int ocfs2_cluster_lock(struct ocfs2_super *osb,
struct ocfs2_lock_res *lockres,
int level,
u32 lkm_flags,
int arg_flags)
{
return __ocfs2_cluster_lock(osb, lockres, level, lkm_flags,
arg_flags, 0);
}static inline void ocfs2_cluster_unlock(struct ocfs2_super *osb,
struct ocfs2_lock_res *lockres, int level)
{
return __ocfs2_cluster_unlock(osb, lockres, level, 0);
}#endif
Don't worry about the efficiency of passing a sixth argument.
That generally fits on x86, and ocfs2 is one of the worst abusers of 12-argument functions anyway. Locking code really, really wants readability first.
Quoted 22 lines
Quoted 7 lines
Quoted 7 lines
fs/ocfs2/inode.c +3
-0@@ -223,6 +225,15 @@ static int ocfs2_init_locked_inode(struct inode *inode, void *opaque)if (args->fi_sysfile_type !=0 )lockdep_set_class (&inode->i_mutex, &ocfs2_sysfile_lock_key[args->fi_sysfile_type]); + if (args->fi_sysfile_type == USER_QUOTA_SYSTEM_INODE || + args->fi_sysfile_type == GROUP_QUOTA_SYSTEM_INODE || + args->fi_sysfile_type == LOCAL_USER_QUOTA_SYSTEM_INODE ||Quoted 8 lines
fs/ocfs2/inode.c static int ocfs2_init_locked_inode(struct inode *inode, void *opaque) · line 236
Quoted 6 lines
fs/ocfs2/inode.c +6
-0+ args->fi_sysfile_type == LOCAL_GROUP_QUOTA_SYSTEM_INODE) +lockdep_set_class (&OCFS2_I (inode)->ip_alloc_sem, + &ocfs2_quota_ip_alloc_sem_key); + else + lockdep_set_class(&OCFS2_I(inode)->ip_alloc_sem, + &ocfs2_file_ip_alloc_sem_key);These keys don't need to be intialized?
No. The key here is the fixed adress of that variable - see
include/linux/lockdep.h:
/*
* Lock-classes are keyed via unique addresses, by embedding the
* lockclass-key into the kernel (or module) .data section. (For
* static locks we use the lock address itself as the key.) */
Quoted 7 lines
struct lockdep_subclass_key { char __one_byte; } __attribute__ ((__packed__));struct lock_class_key { struct lockdep_subclass_key subkeys[MAX_LOCKDEP_SUBCLASSES]; };
Ok, got it.
Joel
Signature · 9 lines
"Heav'n hath no rage like love to hatred turn'd, nor Hell a fury,
like a woman scorn'd."
- William Congreve
Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127Jan Kara
reviewerPATCH 1/1 ocfs2: Add lockdep annotations
Quoted 32 lines · Joel Becker
Quoted 24 lines · Jan Kara
Quoted 8 lines · Joel Becker
Quoted 4 lines · Jan Kara
Add lockdep support to OCFS2. The support also covers all of the cluster locks except for open locks, journal locks, and local quotafile locks. These are special because they are acquired for a node, not for a particular process and lockdep cannot deal with such type of locking.
I like the idea of ocfs2 having lockdep on everything.
Quoted 15 lines
fs/ocfs2/dlmglue.c static int ocfs2_lock_create(struct ocfs2_super *osb, · line 327
Quoted 12 lines
fs/ocfs2/dlmglue.c +9
-3-static void ocfs2_cluster_unlock (struct ocfs2_super *osb, -struct ocfs2_lock_res *lockres, -int level); +static void __ocfs2_cluster_unlock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level,unsigned long ip); +static inline void ocfs2_cluster_unlock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level) +{ +__ocfs2_cluster_unlock (osb, lockres, level, _RET_IP_); +}I don't know why 'ip' needs to be part of the function prototype
- every place you use it, you just pass _RET_IP_.Quoted 6 lines
The problem is following: I need to give to lockdep code an IP of a
function which acquires a lock. For this to be useful I don't want to show "ocfs2_cluster_lock" or "ocfs2_rw_lock" all the time. So I have to get IP of the function calling e.g.ocfs2_rw_lockand that's why there's this ip argument because it's hard to find what the IP of the caller of our caller is...
Quoted 2 lines
Doesn't lockdep provide backtraces when things go wrong? Or is
this used to differentiate between callers internally in lockdep?
Yes, it does. But this IP is for the message like:
=======================================================
[ INFO: possible circular locking dependency detected ] 2.6.25-0.161.rc7.fc9.i686 #1
-------------------------------------------------------
NetworkManager/2308 is trying to acquire lock:
(events){--..}, at: [flush_workqueue+0/133] flush_workqueue+0x0/0x85
^^^^^^^^^^^^^^^^^^^^^^ HERE
but task is already holding lock:
(rtnl_mutex){--..}, at: [rtnetlink_rcv+18/38] rtnetlink_rcv+0x12/0x26
^^^^^^^^^^^^^^^^^^^^ HERE
which lock already depends on the new lock.
=======================================================
So it's usually possible to find all the needed information in the backtraces but it's much more convenient to find it here at the beginning.
BTW: What I did just mimics which IP is passed for ordinary mutexes so it's also more consistent.
Quoted 95 lines
Quoted 50 lines
Quoted 42 lines
Quoted 42 lines
fs/ocfs2/dlmglue.c +25
-5@@ -1239,11 +1256,21 @@ static int ocfs2_wait_for_mask_interruptible(struct ocfs2_mask_waiter *mw,return ret; } -static int ocfs2_cluster_lock (struct ocfs2_super *osb, -struct ocfs2_lock_res *lockres, -int level, -u32 lkm_flags, -int arg_flags) +#ifdef CONFIG_DEBUG_LOCK_ALLOC +static int __ocfs2_cluster_lock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level, +u32 lkm_flags, +int arg_flags, +int l_subclass, + unsigned long ip) +#else +static int __ocfs2_cluster_lock (struct ocfs2_super *osb, +struct ocfs2_lock_res *lockres, +int level, +u32 lkm_flags, + int arg_flags) +#endif {struct ocfs2_mask_waiter mw;int wait, catch_signals = !(osb->s_mount_opt &OCFS2_MOUNT_NOINTR );@@ -1386,13 +1413,45 @@ out: }ocfs2_update_lock_stats (lockres, level, &mw, ret); +#ifdef CONFIG_DEBUG_LOCK_ALLOC +if (!ret && lockres->l_lockdep_map.key !=NULL ) { +if (level ==DLM_LOCK_PR ) +rwsem_acquire_read (&lockres->l_lockdep_map, l_subclass, + !!(arg_flags &OCFS2_META_LOCK_NOQUEUE ), ip); +else +rwsem_acquire (&lockres->l_lockdep_map, l_subclass, + !!(arg_flags &OCFS2_META_LOCK_NOQUEUE ), ip); + } +#endif mlog_exit (ret);Quoted 8 lines
Review out: · line 1428
Quoted 2 lines
(quoted diff) +0
-0return ret; }Here again you've wrapped an ifdef around whether 'ip' is in the
argument list or not. Since it's always _RET_IP_, why not just leave it out of the argument list and just pas _RET_IP_ to therwsem_acquirecalls? That also eliminates the ugly ifdefs and need for__ocfs2_cluster_lock().Quoted 5 lines
I agree ifdefs aren't nice but I wanted to avoid the unnecessary argument
in !LOCKDEP case. I could also just always pass _RET_IP_ and hope that compiler optimizes this in case we don't use the value of ip argument in !LOCKDEP case.
I can do that if you think it's better.I would much rather you just rename to
__ocfs2.., add the
argument, call it "caller_ip", and then wrap it with a static inline. This keeps the ifdefs in one place and much more readable. You can pass 0 when lockdep isn't enabled:#ifdef
CONFIG_DEBUG_LOCK_ALLOC
static inline int ocfs2_cluster_lock(structocfs2_super*osb,struct ocfs2_lock_res *lockres, int level, u32 lkm_flags, int arg_flags) { return __ocfs2_cluster_lock(osb, lockres, level, lkm_flags, arg_flags, _RET_IP_); }static inline void ocfs2_cluster_unlock(struct
ocfs2_super*osb,
structocfs2_lock_res*lockres, int level){ return __ocfs2_cluster_unlock(osb, lockres, level, _RET_IP_); } #else static inline int ocfs2_cluster_lock(struct ocfs2_super *osb, struct ocfs2_lock_res *lockres, int level, u32 lkm_flags, int arg_flags) { return __ocfs2_cluster_lock(osb, lockres, level, lkm_flags, arg_flags, 0); }static inline void ocfs2_cluster_unlock(struct
ocfs2_super*osb,
structocfs2_lock_res*lockres, int level){ return __ocfs2_cluster_unlock(osb, lockres, level, 0); }
Quoted 6 lines
#endif
Don't worry about the efficiency of passing a sixth argument.
That generally fits on x86, and ocfs2 is one of the worst abusers of 12-argument functions anyway. Locking code really, really wants readability first.
OK, I like what you suggest. Will do.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
cover Fix some deadlocks in quota code and implement lockdep for cluster locks
In reply to Jan KaraQuoted 2 lines · Jan Kara
BTW: Currently lockdep complains about some configfs issue and turns itself
off so that's why I didn't get to finding more deadlocks ;)
That's in linux-next of the configfs tree :-)
Joel
Signature · 8 lines
"Hell is oneself, hell is alone, the other figures in it, merely projections."
- T. S. Eliot
Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127cover Fix some deadlocks in quota code and implement lockdep for cluster locks
In reply to Jan KaraQuoted 9 lines · Jan Kara
I'm resending this patch series. It's rediffed against linux-next branch of
Joel's git tree. The first four patches are obvious fixes of deadlocks in quota code and should go in as soon as possible. The other three patches implement lockdep support for OCFS2 cluster locks. So you can have a look whether the code make sence to you and possibly merge them. They should be NOP when lockdep is disabled and help a lot when debugging various locking problems (with these patches I found the problems fixed in the beginning of the series).
BTW: Currently lockdep complains about some configfs issue and turns itself
off so that's why I didn't get to finding more deadlocks ;)
The fixes are now in the merge-window branch of ocfs2.git. They
also live on the quota-fixes topic branch. The lockdep changes still need some marinating.
Joel
Signature · 9 lines
Life's Little Instruction Book #274 "Leave everything a little better than you found it." Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127
Jan Kara
reviewercover Fix some deadlocks in quota code and implement lockdep for cluster locks
Quoted 11 lines · Joel Becker
Quoted 9 lines · Jan Kara
I'm resending this patch series. It's rediffed against linux-next branch of
Joel's git tree. The first four patches are obvious fixes of deadlocks in quota code and should go in as soon as possible. The other three patches implement lockdep support for OCFS2 cluster locks. So you can have a look whether the code make sence to you and possibly merge them. They should be NOP when lockdep is disabled and help a lot when debugging various locking problems (with these patches I found the problems fixed in the beginning of the series).
BTW: Currently lockdep complains about some configfs issue and turns itself
off so that's why I didn't get to finding more deadlocks ;)
Quoted 3 lines
The fixes are now in the merge-window branch of ocfs2.git. They
also live on the quota-fixes topic branch. The lockdep changes still need some marinating.
Thanks. With lockdep patch, there's this 'ip' parameter thing causing
ifdefs but after we agree how to resolve this, it can go into linux-next branch, right?
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
cover Fix some deadlocks in quota code and implement lockdep for cluster locks
Quoted 7 lines · Jan Kara
Quoted 3 lines · Joel Becker
The fixes are now in the merge-window branch of ocfs2.git. They
also live on the quota-fixes topic branch. The lockdep changes still need some marinating.Thanks. With lockdep patch, there's this 'ip' parameter thing causing
ifdefs but after we agree how to resolve this, it can go into linux-next branch, right?
Yeah. I'm going to hold off on merging it until the 2.6.32
window so that it can get some testing, as it's a significant change to our locking code. The rest of the fixes are going for 2.6.31.
Joel
Signature · 9 lines
"Sometimes when reading Goethe I have the paralyzing suspicion
that he is trying to be funny."
- Guy Davenport
Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127