Skip to contentSkip to conversation

Bug fixes for --write-zeros option

PATCH Applied Logan Gunthorpe 3 participants 6 messages Jun 4, 2024 – Jun 13, 2024 MD RAID

lore
Share & tools
lore
#0 lore raw

cover Bug fixes for --write-zeros option

Hi,

Xiao noticed that the write-zeros tests failed randomly, especially with small disks. We tracked this down to an issue with signalfd which coallesced SIGCHLD signals into one. This is fixed by checking the status of all children after every SIGCHLD.

While we were at it, we noticed a potential reace with SIGCHLD coming in before the signal was blocked in wait_for_zero_forks() and fix this by moving the blocking before the child creation.

Thanks,

Logan

--

Logan Gunthorpe (2):
mdadm: Fix hang race condition in wait_for_zero_forks()
mdadm: Block SIGCHLD processes before starting children

 Create.c | 29 ++++++++++++++++-------------
 1 file changed, 16 insertions(+), 13 deletions(-)

base-commit: 46f19270265fe54cda1c728cb156b755273b4ab6 --
2.39.2

#1 lore raw

PATCH 1/2 mdadm: Fix hang race condition in wait_for_zero_forks()

Running a create operation with --write-zeros can randomly hang forever waiting for child processes. This happens roughly on in ten runs with when running with small (20MB) loop devices.

The bug is caused by the fact that signals can be coallesced into one if they are not read by signalfd quick enough. So if two children finish at exactly the same time, only one SIGCHLD will be received by the parent.

To fix this, wait on all processes with WNOHANG every time a SIGCHLD is recieved and exit when all processes have been waited on.

---
 Create.c | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)
Create.c +15 -13
index d033eb68f30c..4f992a22b7c9 100644
@@ -178,6 +178,7 @@ static int wait_for_zero_forks(int *zero_pids, int count)
 	bool interrupted = false;
 	sigset_t sigset;
 	ssize_t s;
+	pid_t pid;
 
 	for (i = 0; i < count; i++)
 		if (zero_pids[i])
@@ -196,7 +197,7 @@ static int wait_for_zero_forks(int *zero_pids, int count)
 		return 1;
 	}
 
-	while (1) {
+	while (wait_count) {
 		s = read(sfd, &fdsi, sizeof(fdsi));
 		if (s != sizeof(fdsi)) {
 			pr_err("Invalid signalfd read: %s\n", strerror(errno));
@@ -209,23 +210,24 @@ static int wait_for_zero_forks(int *zero_pids, int count)
 			pr_info("Interrupting zeroing processes, please wait...\n");
 			interrupted = true;
 		} else if (fdsi.ssi_signo == SIGCHLD) {
-			if (!--wait_count)
-				break;
+			for (i = 0; i < count; i++) {
+				if (!zero_pids[i])
+					continue;
+
+				pid = waitpid(zero_pids[i], &wstatus, WNOHANG);
+				if (pid <= 0)
+					continue;
+
+				zero_pids[i] = 0;
+				if (!WIFEXITED(wstatus) || WEXITSTATUS(wstatus))
+					ret = 1;
+				wait_count--;
+			}
 		}
 	}
 
 	close(sfd);
 
-	for (i = 0; i < count; i++) {
-		if (!zero_pids[i])
-			continue;
-
-		waitpid(zero_pids[i], &wstatus, 0);
-		zero_pids[i] = 0;
-		if (!WIFEXITED(wstatus) || WEXITSTATUS(wstatus))
-			ret = 1;
-	}
-
 	if (interrupted) {
 		pr_err("zeroing interrupted!\n");
 		return 1;
Signature · 1 line
2.39.2
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH mdadm 1/2] mdadm: Fix hang race condition in wait_for_zero_forks()

Dear Logan,

Thank you for the patch.

Am 04.06.24 um 18:38 schrieb Logan Gunthorpe:

Quoted 11 lines

Running a create operation with --write-zeros can randomly hang forever waiting for child processes. This happens roughly on in ten runs with when running with small (20MB) loop devices.

The bug is caused by the fact that signals can be coallesced into one if they are not read by signalfd quick enough. So if two children finish at exactly the same time, only one SIGCHLD will be received by the parent.

To fix this, wait on all processes with WNOHANG every time a SIGCHLD is recieved and exit when all processes have been waited on.

received

Quoted 54 lines
---
  Create.c | 28 +++++++++++++++-------------
  1 file changed, 15 insertions(+), 13 deletions(-)
Create.c +15 -6
index d033eb68f30c..4f992a22b7c9 100644
@@ -178,6 +178,7 @@ static int wait_for_zero_forks(int *zero_pids, int count)
  	bool interrupted = false;
  	sigset_t sigset;
  	ssize_t s;
+	pid_t pid;
 
  	for (i = 0; i < count; i++)
  		if (zero_pids[i])
@@ -196,7 +197,7 @@ static int wait_for_zero_forks(int *zero_pids, int count)
  		return 1;
  	}
 
-	while (1) {
+	while (wait_count) {
  		s = read(sfd, &fdsi, sizeof(fdsi));
  		if (s != sizeof(fdsi)) {
  			pr_err("Invalid signalfd read: %s\n", strerror(errno));
@@ -209,23 +210,24 @@ static int wait_for_zero_forks(int *zero_pids, int count)
  			pr_info("Interrupting zeroing processes, please wait...\n");
  			interrupted = true;
  		} else if (fdsi.ssi_signo == SIGCHLD) {
-			if (!--wait_count)
-				break;
+			for (i = 0; i < count; i++) {
+				if (!zero_pids[i])
+					continue;
+
+				pid = waitpid(zero_pids[i], &wstatus, WNOHANG);
+				if (pid <= 0)
+					continue;
+
+				zero_pids[i] = 0;
+				if (!WIFEXITED(wstatus) || WEXITSTATUS(wstatus))
+					ret = 1;
+				wait_count--;
+			}
  		}
  	}
 
  	close(sfd);
 
-	for (i = 0; i < count; i++) {
-		if (!zero_pids[i])
-			continue;

Create.c static int wait_for_zero_forks(int *zero_pids, int count) · line 233

Quoted 10 lines
Create.c +0 -7
-
-		waitpid(zero_pids[i], &wstatus, 0);
-		zero_pids[i] = 0;
-		if (!WIFEXITED(wstatus) || WEXITSTATUS(wstatus))
-			ret = 1;
-	}
-
  	if (interrupted) {
  		pr_err("zeroing interrupted!\n");
  		return 1;

Kind regards,

Paul

#3 lore raw

PATCH 2/2 mdadm: Block SIGCHLD processes before starting children

In reply to Logan Gunthorpe

There is a small race condition noticed during code review, but never actully hit in practice, with the write_zero feature.

If a write zeros fork finishes quickly before wait_for_zero_forks() gets called, then the SIGCHLD will be delivered before the signalfd is setup.

While this is only theoretical, fix this by blocking the SIGCHLD signal before forking any children.

---
 Create.c | 1 +
 1 file changed, 1 insertion(+)
Create.c +1 -0
index 4f992a22b7c9..bd4875e4a408 100644
@@ -401,6 +401,7 @@ static int add_disks(int mdfd, struct mdinfo *info, struct shape *s,
 	 */
 	sigemptyset(&sigset);
 	sigaddset(&sigset, SIGINT);
+	sigaddset(&sigset, SIGCHLD);
 	sigprocmask(SIG_BLOCK, &sigset, &orig_sigset);
 	memset(zero_pids, 0, sizeof(zero_pids));
 
Signature · 1 line
2.39.2
#4 lore raw
In reply to Logan Gunthorpe
Quoted 11 lines · On Tue, 4 Jun 2024 10:38:35 -0600 Logan Gunthorpe <[email protected]> wrote:

Hi,

Xiao noticed that the write-zeros tests failed randomly, especially with small disks. We tracked this down to an issue with signalfd which coallesced SIGCHLD signals into one. This is fixed by checking the status of all children after every SIGCHLD.

While we were at it, we noticed a potential reace with SIGCHLD coming in before the signal was blocked in wait_for_zero_forks() and fix this by moving the blocking before the child creation.

Quoted 5 lines

Thanks,

Logan

--

Hello Logan,
Thanks for fixes. LGTM.

I will fix typo when merging, no need to sent v2.
I have proxy issue, I have to solve it first.

Thanks,
Mariusz

#5 lore raw
In reply to Logan Gunthorpe
Quoted 11 lines · On Tue, 4 Jun 2024 10:38:35 -0600 Logan Gunthorpe <[email protected]> wrote:

Hi,

Xiao noticed that the write-zeros tests failed randomly, especially with small disks. We tracked this down to an issue with signalfd which coallesced SIGCHLD signals into one. This is fixed by checking the status of all children after every SIGCHLD.

While we were at it, we noticed a potential reace with SIGCHLD coming in before the signal was blocked in wait_for_zero_forks() and fix this by moving the blocking before the child creation.

Quoted 5 lines

Thanks,

Logan

--

Applied!

Thanks,
Mariusz