raid1: Add a filed array_frozen to indicate whether raid in freeze state.

Because the following patch will rewrite the contend between nornal IO and resync IO. So we used a parameter to indicate whether raid is in freeze array.

---
 drivers/md/raid1.c | 15 +++++++--------
 drivers/md/raid1.h |  1 +
 2 files changed, 8 insertions(+), 8 deletions(-)

index d60412c..92a6d29 100644
@@ -829,6 +829,7 @@ static void raise_barrier(struct r1conf *conf)
 
 	/* Now wait for all pending IO to complete */
 	wait_event_lock_irq(conf->wait_barrier,
+			    !conf->array_frozen &&
 			    !conf->nr_pending && conf->barrier < RESYNC_DEPTH,
 			    conf->resync_lock);
 

@@ -860,10 +861,11 @@ static void wait_barrier(struct r1conf *conf)
 		 * count down.
 		 */
 		wait_event_lock_irq(conf->wait_barrier,
-				    !conf->barrier ||
+				    !conf->array_frozen &&
+				    (!conf->barrier ||
 				    (conf->nr_pending &&
 				     current->bio_list &&
-				     !bio_list_empty(current->bio_list)),
+				     !bio_list_empty(current->bio_list))),
 				    conf->resync_lock);
 		conf->nr_waiting--;
 	}

@@ -884,8 +886,7 @@ static void freeze_array(struct r1conf *conf, int extra)
 {
 	/* stop syncio and normal IO and wait for everything to
 	 * go quite.
-	 * We increment barrier and nr_waiting, and then
-	 * wait until nr_pending match nr_queued+extra
+	 * We wait until nr_pending match nr_queued+extra
 	 * This is called in the context of one normal IO request
 	 * that has failed. Thus any sync request that might be pending
 	 * will be blocked by nr_pending, and we need to wait for

@@ -895,8 +896,7 @@ static void freeze_array(struct r1conf *conf, int extra)
 	 * we continue.
 	 */
 	spin_lock_irq(&conf->resync_lock);
-	conf->barrier++;
-	conf->nr_waiting++;
+	conf->array_frozen = 1;
 	wait_event_lock_irq_cmd(conf->wait_barrier,
 				conf->nr_pending == conf->nr_queued+extra,
 				conf->resync_lock,

@@ -907,8 +907,7 @@ static void unfreeze_array(struct r1conf *conf)
 {
 	/* reverse the effect of the freeze */
 	spin_lock_irq(&conf->resync_lock);
-	conf->barrier--;
-	conf->nr_waiting--;
+	conf->array_frozen = 0;
 	wake_up(&conf->wait_barrier);
 	spin_unlock_irq(&conf->resync_lock);
 }

index 0ff3715..331a98a 100644
@@ -65,6 +65,7 @@ struct r1conf {
 	int			nr_waiting;
 	int			nr_queued;
 	int			barrier;
+	int			array_frozen;
 
 	/* Set to 1 if a full sync is needed, (fresh device added).
 	 * Cleared when a sync completes.

1.8.1.2

Hello Jianpeng Ma, Neil,

This commit seems to introduce a severe regression, leading to IO getting stuck on the whole array.

Previously, the condition of wait_barrier() was:

                wait_event_lock_irq(conf->wait_barrier,
                                    !conf->barrier ||
                                    (conf->nr_pending &&
                                     current->bio_list &&
                                     !bio_list_empty(current->bio_list)),
                                    conf->resync_lock);

This means that if current->bio_list is not empty, we will not wait for the barrier to drop.

But right now the condition is:

                wait_event_lock_irq(conf->wait_barrier,
                                    !conf->array_frozen &&
                                    (!conf->barrier ||
                                    (conf->nr_pending &&
                                     current->bio_list &&
                                     !bio_list_empty(current->bio_list))),
                                    conf->resync_lock);

This means that if somebody calls freeze_array (for example, md_do_sync), then freeze_array unconditionally sets:
conf->array_frozen = 1;

And then if somebody calls wait_barrier and its current->bio_list is not empty, it will still wait, because array_frozen=1. But freeze_array is waiting for the bio complete, but this bio sits in current->bio_list and will never get submitted, because we are waiting in wait_barrier. DEADLOCK.

For me it happens on kernel 3.18, when raid1.c::make_request() submits a READ bio, which has raid1_end_read_request completion routine. But this completion routine never gets called, and I also confirmed that no IO is stuck on lower levels.

Additional notes:
# Please see my email titled "RAID1: deadlock between freeze_array and blk plug?" in http://permalink.gmane.org/gmane.linux.raid/52693. The problem described there also stems from the fact that now freeze_array() sets array_frozen=1 unconditionally. And wait_barrier() will always wait if array_frozen!=0.

# Also now freeze_array() is non-reentrant. Meaning that if two threads call freeze_array in parallel, it will not work, because array_frozen can only be 1 or 0. Example, when freeze_array can be called by two threads:
- We have 3-way RAID1 with one drive missing. We want to recover this drive.
- we have an in-flight READ request
- md_do_sync starts recovery and calls mddev->pers->quiesce(mddev, 1), which waits for the in-flight READ to complete
- meanwhile the READ bio fails
- so raid1d calls handle_read_error, which calls freeze_array
Result: we have two freeze_array calls in parallel. But we don't know to account for them properly, because array_frozen can only be 1 or 0.

Please advise how to fix this problem. Please kindly note that the fix should apply to kernel 3.18; which is a longterm kernel that we use.

Thanks,
Alex.