Skip to contentSkip to conversation

zcache: avoid AB-BA deadlock condition

PATCH In review Andrea Righi 2 participants 3 messages Feb 20, 2012 – Feb 27, 2012 Staging

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 zcache: avoid AB-BA deadlock condition

Commit 9256a47 fixed a deadlock condition, being sure that the buddy list spinlock is always taken before the page spinlock.

However in zbud_free_and_delist() locking order is the opposite (page lock -> list lock).

Possible unsafe locking scenario (reported by lockdep):
        CPU0                    CPU1
        ----                    ----
   lock(&(&zbpg->lock)->rlock);
                                lock(zbud_budlists_spinlock);
                                lock(&(&zbpg->lock)->rlock);

lock(zbud_budlists_spinlock);

Fix by grabbing the locks in opposite order in zbud_free_and_delist().

---
 drivers/staging/zcache/zcache-main.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
drivers/staging/zcache/zcache-main.c +2 -2
index ef7c52b..dce04be 100644
@@ -299,10 +299,12 @@ static void zbud_free_and_delist(struct zbud_hdr *zh)
 	struct zbud_page *zbpg =
 		container_of(zh, struct zbud_page, buddy[budnum]);
 
+	spin_lock(&zbud_budlists_spinlock);
 	spin_lock(&zbpg->lock);
 	if (list_empty(&zbpg->bud_list)) {
 		/* ignore zombie page... see zbud_evict_pages() */
 		spin_unlock(&zbpg->lock);
+		spin_unlock(&zbud_budlists_spinlock);
 		return;
 	}
 	size = zbud_free(zh);
@@ -310,7 +312,6 @@ static void zbud_free_and_delist(struct zbud_hdr *zh)
 	zh_other = &zbpg->buddy[(budnum == 0) ? 1 : 0];
 	if (zh_other->size == 0) { /* was unbuddied: unlist and free */
 		chunks = zbud_size_to_chunks(size) ;
-		spin_lock(&zbud_budlists_spinlock);
 		BUG_ON(list_empty(&zbud_unbuddied[chunks].list));
 		list_del_init(&zbpg->bud_list);
 		zbud_unbuddied[chunks].count--;
@@ -318,7 +319,6 @@ static void zbud_free_and_delist(struct zbud_hdr *zh)
 		zbud_free_raw_page(zbpg);
 	} else { /* was buddied: move remaining buddy to unbuddied list */
 		chunks = zbud_size_to_chunks(zh_other->size) ;
-		spin_lock(&zbud_budlists_spinlock);
 		list_del_init(&zbpg->bud_list);
 		zcache_zbud_buddied_count--;
 		list_add_tail(&zbpg->bud_list, &zbud_unbuddied[chunks].list);
Signature · 1 line
1.7.5.4
Acked-by on v1 1/1
#1 lore raw
Quoted 20 lines

From: Andrea Righi [mailto:[email protected]]
Sent: Monday, February 20, 2012 5:12 AM
To: Greg Kroah-Hartman
Cc: Dan Magenheimer; Seth Jennings; [email protected]; [email protected]; [email protected]
Subject: [PATCH] zcache: avoid AB-BA deadlock condition

Commit 9256a47 fixed a deadlock condition, being sure that the buddy list spinlock is always taken before the page spinlock.

However in zbud_free_and_delist() locking order is the opposite (page lock -> list lock).

Possible unsafe locking scenario (reported by lockdep):
        CPU0                    CPU1
        ----                    ----
   lock(&(&zbpg->lock)->rlock);
                                lock(zbud_budlists_spinlock);
                                lock(&(&zbpg->lock)->rlock);
Quoted 5 lines

lock(zbud_budlists_spinlock);

Fix by grabbing the locks in opposite order in zbud_free_and_delist().

Thanks for catching this Andrea! (And thanks also to Alex Vallacis-Lasso for independently reporting and testing:
http://permalink.gmane.org/gmane.linux.kernel/1257214 )

Greg, this patch could be targeted for 3.3-rc6 and 3.2-stable. AFAIK, nobody has actually experienced a deadlock from this so if Linus has the screws down tight for -rc6, it could wait until the 3.4 window.

Quoted 39 lines
---
 drivers/staging/zcache/zcache-main.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
drivers/staging/zcache/zcache-main.c +2 -3
index ef7c52b..dce04be 100644
@@ -299,10 +299,12 @@ static void zbud_free_and_delist(struct zbud_hdr *zh)
 	struct zbud_page *zbpg =
 		container_of(zh, struct zbud_page, buddy[budnum]);
 
+	spin_lock(&zbud_budlists_spinlock);
 	spin_lock(&zbpg->lock);
 	if (list_empty(&zbpg->bud_list)) {
 		/* ignore zombie page... see zbud_evict_pages() */
 		spin_unlock(&zbpg->lock);
+		spin_unlock(&zbud_budlists_spinlock);
 		return;
 	}
 	size = zbud_free(zh);
@@ -310,7 +312,6 @@ static void zbud_free_and_delist(struct zbud_hdr *zh)
 	zh_other = &zbpg->buddy[(budnum == 0) ? 1 : 0];
 	if (zh_other->size == 0) { /* was unbuddied: unlist and free */
 		chunks = zbud_size_to_chunks(size) ;
-		spin_lock(&zbud_budlists_spinlock);
 		BUG_ON(list_empty(&zbud_unbuddied[chunks].list));
 		list_del_init(&zbpg->bud_list);
 		zbud_unbuddied[chunks].count--;
@@ -318,7 +319,6 @@ static void zbud_free_and_delist(struct zbud_hdr *zh)
 		zbud_free_raw_page(zbpg);
 	} else { /* was buddied: move remaining buddy to unbuddied list */
 		chunks = zbud_size_to_chunks(zh_other->size) ;
-		spin_lock(&zbud_budlists_spinlock);
 		list_del_init(&zbpg->bud_list);
 		zcache_zbud_buddied_count--;
 		list_add_tail(&zbpg->bud_list, &zbud_unbuddied[chunks].list);
--

1.7.5.4

#2 lore raw
Quoted 28 lines · Dan Magenheimer
Quoted 20 lines

From: Andrea Righi [mailto:[email protected]]
Sent: Monday, February 20, 2012 5:12 AM
To: Greg Kroah-Hartman
Cc: Dan Magenheimer; Seth Jennings; [email protected]; [email protected]; [email protected]
Subject: [PATCH] zcache: avoid AB-BA deadlock condition

Commit 9256a47 fixed a deadlock condition, being sure that the buddy list spinlock is always taken before the page spinlock.

However in zbud_free_and_delist() locking order is the opposite (page lock -> list lock).

Possible unsafe locking scenario (reported by lockdep):
        CPU0                    CPU1
        ----                    ----
   lock(&(&zbpg->lock)->rlock);
                                lock(zbud_budlists_spinlock);
                                lock(&(&zbpg->lock)->rlock);
Quoted 5 lines

lock(zbud_budlists_spinlock);

Fix by grabbing the locks in opposite order in zbud_free_and_delist().

Quoted 8 lines

Thanks for catching this Andrea! (And thanks also to Alex Vallacis-Lasso for independently reporting and testing:
http://permalink.gmane.org/gmane.linux.kernel/1257214 )

Greg, this patch could be targeted for 3.3-rc6 and 3.2-stable. AFAIK, nobody has actually experienced a deadlock from this so if Linus has the screws down tight for -rc6, it could wait until the 3.4 window.

If it helps, without the fix I can easily trigger the lockdep splat running a echo 3 > /proc/sys/vm/drop_caches.

-Andrea