Skip to contentSkip to conversation

xfs_scrub_all process execution results in a dead lock condition

DISCUSSION Patrick Fischer 2 participants 2 messages Apr 30

lore
Share & tools
lore
#0 lore raw

Hello,
I've encountered a bug within the xfsprogs-dev utilities, particular within the Python script xfs_scrub_all.
I researched the master branch and saw, that the type of sub process call is like within kernel version 6.13.0-2 I stumbled across this issue.

Overview:
xfs_scrub_all.service systemd unit (or manual execution) is hanging due to a pipe buffer exhaustion after sub process call of lsblk.

Steps to reproduce:
Create a bunch of fake block devices to enlarge the output of lsblk to more than 65520 bytes:

Quoted 1 line

modprobe scsi_debug max_lunx=3 num_tgts=7 add_hosts=100

Run the command of xfs_scrub_all.service manually:

Quoted 1 line

/usr/sbin/xfs_scrub_all --auto-media-scan-interval 1mo

xfs_scrub_all shows a wait4 for the sub process lsblk:

Quoted 1 line

wait4(2148527,

Within sub process lsblk there is a write to FD1 / stdout:

Quoted 1 line

write(1, " {\n "..., 4096

Affected Code in /usr/sbin/xfs_scrub_all[1]:

Quoted 5 lines
54     cmd=['lsblk', '-o', 'NAME,KNAME,TYPE,FSTYPE,MOUNTPOINT', '-J']
55     result = subprocess.Popen(cmd, stdout=subprocess.PIPE)
56     result.wait()
57     if result.returncode != 0:

58 return fs

Actual Results:
The execution of the command above launches a sub process of lsblk and returns more than 65520 bytes, resulting in an endless wait for return.

Expected Results:
The unit / process should not enter a dead lock.

[1] https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git/tree/scrub/xfs_scrub_all.py.in

Regards,
Patrick Fischer

Die E-Mail wurde von IKARUS mail.security geprüft.

#1 lore raw
Quoted 5 lines · Patrick Fischer

Hello,
I've encountered a bug within the xfsprogs-dev utilities, particular within the Python script xfs_scrub_all. I researched the master branch and saw, that the type of sub process call is like within kernel version 6.13.0-2 I stumbled across this issue.

"6.13.0-2" ... Debian Trixie?

Quoted 8 lines

Overview:
xfs_scrub_all.service systemd unit (or manual execution) is hanging due to a pipe buffer exhaustion after sub process call of lsblk.

Steps to reproduce:
Create a bunch of fake block devices to enlarge the output of lsblk to more than 65520 bytes:

Quoted 1 line

modprobe scsi_debug max_lunx=3 num_tgts=7 add_hosts=100

/me notes that this wasn't enough to generate more than 60k of lsblk output. Creating a fake 130k json file and changing cmd to ['cat', '/tmp/garbage'] was sufficient to reproduce the problem, however.

Quoted 20 lines

Run the command of xfs_scrub_all.service manually:

Quoted 1 line

/usr/sbin/xfs_scrub_all --auto-media-scan-interval 1mo

xfs_scrub_all shows a wait4 for the sub process lsblk:

Quoted 1 line

wait4(2148527,

Within sub process lsblk there is a write to FD1 / stdout:

Quoted 1 line

write(1, " {\n "..., 4096

Affected Code in /usr/sbin/xfs_scrub_all[1]:

Quoted 5 lines
54     cmd=['lsblk', '-o', 'NAME,KNAME,TYPE,FSTYPE,MOUNTPOINT', '-J']
55     result = subprocess.Popen(cmd, stdout=subprocess.PIPE)
56     result.wait()
57     if result.returncode != 0:

58 return fs

Quoted 4 lines

Actual Results:
The execution of the command above launches a sub process of lsblk and returns more than 65520 bytes, resulting in an endless wait for return.

Yep, that's a bug.

Now that we can assume (demand?) python >= 3.5, I think we can replace all the string iteration and collection mess in that function with a simpler call to subprocess.run:

cmd=['lsblk', '-o', 'NAME,KNAME,TYPE,FSTYPE,MOUNTPOINT', '-J']
try:
proc = subprocess.run(cmd, capture_output = True, text = True, check = True)
except Exception as e:
print(e)
return fs
if proc.returncode != 0:
return fs

# The lsblk output had better be in disks-then-partitions order bdevdata = json.loads(proc.stdout)
for bdev in bdevdata['blockdevices']:

Quoted 4 lines

Expected Results:
The unit / process should not enter a dead lock.

[1] https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git/tree/scrub/xfs_scrub_all.py.in

Thanks for the report, sorry there was a bug. I'll post a fixpatch soon.

--D

Quoted 6 lines

Regards,
Patrick Fischer

Die E-Mail wurde von IKARUS mail.security geprüft.