a deadloop(?) when mkfs.xfs -o rtdev
Hi,
There seems a deadloop(?) when mkfs.xfs -o rtdev.
# mkfs.xfs -r rtdev=/dev/disk/by-id/ata-MZ7KM1T6HAJM00D3_S2CXNAAH600026 /dev/d
isk/by-id/scsi-SHITACHI_HUSMH842_CLAR100_0LX0JWVA -f
meta-data=/dev/disk/by-id/scsi-SHITACHI_HUSMH842_CLAR100_0LX0JWVA isize=512 agcount=20, agsize=1221072 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=0 bigtime=1 inobtcount=1 nrext64=1
= exchange=0
data = bsize=4096 blocks=24421440, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1, parent=0
log =internal log bsize=4096 blocks=41799, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =/dev/disk/by-id/ata-MZ7KM1T6HAJM00D3_S2CXNAAH600026 extsz=4096 blocks=390703446, rtextents=390703446
Discarding blocks...Done.
Discarding blocks...Done.It did not fishish after 10 mins.
# pstack 5785 #0 0x00007f8df5efc01a in pread64 () from /lib64/libc.so.6 #1 0x0000557b7bbfe9c3 in __read_buf.constprop.0 () #2 0x0000557b7bbc3030 in libxfs_buf_read_map () #3 0x0000557b7bbfe1ae in libxfs_trans_read_buf_map.constprop () #4 0x0000557b7bbee7ef in xfs_rtbuf_get () #5 0x0000557b7bbf03d8 in libxfs_rtfree_extent () #6 0x0000557b7bbbb322 in parseproto.lto_priv () #7 0x0000557b7bbb7643 in main ()
And,
1) more chance happen when kernel 6.12.36, but yet not happen when kernel 6.6.93.
2) it happen on both xfsprogs-6.11.0 and xfsprogs-6.4.
Best Regards
Wang Yugui ([email protected])
2025/07/17
Hi,
Quoted 30 lines
There seems a deadloop(?) when mkfs.xfs -o rtdev.
# mkfs.xfs -r rtdev=/dev/disk/by-id/ata-MZ7KM1T6HAJM00D3_S2CXNAAH600026 /dev/d isk/by-id/scsi-SHITACHI_HUSMH842_CLAR100_0LX0JWVA -f meta-data=/dev/disk/by-id/scsi-SHITACHI_HUSMH842_CLAR100_0LX0JWVA isize=512 agcount=20, agsize=1221072 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=0 bigtime=1 inobtcount=1 nrext64=1 = exchange=0 data = bsize=4096 blocks=24421440, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1, parent=0 log =internal log bsize=4096 blocks=41799, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =/dev/disk/by-id/ata-MZ7KM1T6HAJM00D3_S2CXNAAH600026 extsz=4096 blocks=390703446, rtextents=390703446 Discarding blocks...Done. Discarding blocks...Done.It did not fishish after 10 mins.
# pstack 5785 #0 0x00007f8df5efc01a in pread64 () from /lib64/libc.so.6 #1 0x0000557b7bbfe9c3 in __read_buf.constprop.0 () #2 0x0000557b7bbc3030 in libxfs_buf_read_map () #3 0x0000557b7bbfe1ae in libxfs_trans_read_buf_map.constprop () #4 0x0000557b7bbee7ef in xfs_rtbuf_get () #5 0x0000557b7bbf03d8 in libxfs_rtfree_extent () #6 0x0000557b7bbbb322 in parseproto.lto_priv () #7 0x0000557b7bbb7643 in main ()
Quoted 3 lines
And,
1) more chance happen when kernel 6.12.36, but yet not happen when kernel 6.6.93.
2) it happen on both xfsprogs-6.11.0 and xfsprogs-6.4.
it happen on xfsprogs-6.15.0 too.
Best Regards
Wang Yugui ([email protected])
2025/07/17
Quoted 10 lines · Wang Yugui
# pstack 5785 #0 0x00007f8df5efc01a in pread64 () from /lib64/libc.so.6 #1 0x0000557b7bbfe9c3 in __read_buf.constprop.0 () #2 0x0000557b7bbc3030 in libxfs_buf_read_map () #3 0x0000557b7bbfe1ae in libxfs_trans_read_buf_map.constprop () #4 0x0000557b7bbee7ef in xfs_rtbuf_get () #5 0x0000557b7bbf03d8 in libxfs_rtfree_extent () #6 0x0000557b7bbbb322 in parseproto.lto_priv () #7 0x0000557b7bbb7643 in main ()
This is reading the data from the main device for updating the bits on the rt bitmap.
If you have a very larger rt device this can take a long time and might not be stuck. And easy way to test that hypothesis would be to try formatting with a very large RT exent size.
Quoted 32 lines · Wang Yugui
Hi,
There seems a deadloop(?) when mkfs.xfs -o rtdev.
# mkfs.xfs -r rtdev=/dev/disk/by-id/ata-MZ7KM1T6HAJM00D3_S2CXNAAH600026 /dev/d isk/by-id/scsi-SHITACHI_HUSMH842_CLAR100_0LX0JWVA -f meta-data=/dev/disk/by-id/scsi-SHITACHI_HUSMH842_CLAR100_0LX0JWVA isize=512 agcount=20, agsize=1221072 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=0 bigtime=1 inobtcount=1 nrext64=1 = exchange=0 data = bsize=4096 blocks=24421440, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1, parent=0 log =internal log bsize=4096 blocks=41799, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =/dev/disk/by-id/ata-MZ7KM1T6HAJM00D3_S2CXNAAH600026 extsz=4096 blocks=390703446, rtextents=390703446 Discarding blocks...Done. Discarding blocks...Done.It did not fishish after 10 mins.
# pstack 5785 #0 0x00007f8df5efc01a in pread64 () from /lib64/libc.so.6 #1 0x0000557b7bbfe9c3 in __read_buf.constprop.0 () #2 0x0000557b7bbc3030 in libxfs_buf_read_map () #3 0x0000557b7bbfe1ae in libxfs_trans_read_buf_map.constprop () #4 0x0000557b7bbee7ef in xfs_rtbuf_get () #5 0x0000557b7bbf03d8 in libxfs_rtfree_extent () #6 0x0000557b7bbbb322 in parseproto.lto_priv () #7 0x0000557b7bbb7643 in main ()
Quoted 3 lines
And,
1) more chance happen when kernel 6.12.36, but yet not happen when kernel 6.6.93.
2) it happen on both xfsprogs-6.11.0 and xfsprogs-6.4.
You /could/ strace the mkfs process to see if it's really stuck, or just issuing IOs really slowly.
--D
Quoted 7 lines
Best Regards
Wang Yugui ([email protected])
2025/07/17
Hi,
Quoted 33 lines · Wang Yugui
Quoted 32 lines
Hi,
There seems a deadloop(?) when mkfs.xfs -o rtdev.
# mkfs.xfs -r rtdev=/dev/disk/by-id/ata-MZ7KM1T6HAJM00D3_S2CXNAAH600026 /dev/d isk/by-id/scsi-SHITACHI_HUSMH842_CLAR100_0LX0JWVA -f meta-data=/dev/disk/by-id/scsi-SHITACHI_HUSMH842_CLAR100_0LX0JWVA isize=512 agcount=20, agsize=1221072 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=0 bigtime=1 inobtcount=1 nrext64=1 = exchange=0 data = bsize=4096 blocks=24421440, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1, parent=0 log =internal log bsize=4096 blocks=41799, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =/dev/disk/by-id/ata-MZ7KM1T6HAJM00D3_S2CXNAAH600026 extsz=4096 blocks=390703446, rtextents=390703446 Discarding blocks...Done. Discarding blocks...Done.It did not fishish after 10 mins.
# pstack 5785 #0 0x00007f8df5efc01a in pread64 () from /lib64/libc.so.6 #1 0x0000557b7bbfe9c3 in __read_buf.constprop.0 () #2 0x0000557b7bbc3030 in libxfs_buf_read_map () #3 0x0000557b7bbfe1ae in libxfs_trans_read_buf_map.constprop () #4 0x0000557b7bbee7ef in xfs_rtbuf_get () #5 0x0000557b7bbf03d8 in libxfs_rtfree_extent () #6 0x0000557b7bbbb322 in parseproto.lto_priv () #7 0x0000557b7bbb7643 in main ()
Quoted 6 lines
Quoted 3 lines
And,
1) more chance happen when kernel 6.12.36, but yet not happen when kernel 6.6.93.
2) it happen on both xfsprogs-6.11.0 and xfsprogs-6.4.You /could/ strace the mkfs process to see if it's really stuck, or just issuing IOs really slowly.
Today the mkfs.xfs with rtdev all finished in 110s-160s on kernel 6.6.93/6.12.38.
the result of strace show that pread64() with 4K size so it is just slow.
pread64(3, "\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377"..., 4096, 868352) = 4096 pread64(3, "\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377"..., 4096, 864256) = 4096 pread64(3, "\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377"..., 4096, 860160) = 4096 pread64(3, "\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377"..., 4096, 856064) = 4096 pread64(3, "\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377"..., 4096, 851968) = 4096
Thanks a lot.
Best Regards
Wang Yugui ([email protected])
2025/07/18
Hi,
Quoted 44 lines
Hi,
Quoted 33 lines · Wang Yugui
Quoted 32 lines
Hi,
There seems a deadloop(?) when mkfs.xfs -o rtdev.
# mkfs.xfs -r rtdev=/dev/disk/by-id/ata-MZ7KM1T6HAJM00D3_S2CXNAAH600026 /dev/d isk/by-id/scsi-SHITACHI_HUSMH842_CLAR100_0LX0JWVA -f meta-data=/dev/disk/by-id/scsi-SHITACHI_HUSMH842_CLAR100_0LX0JWVA isize=512 agcount=20, agsize=1221072 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=0 bigtime=1 inobtcount=1 nrext64=1 = exchange=0 data = bsize=4096 blocks=24421440, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1, parent=0 log =internal log bsize=4096 blocks=41799, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =/dev/disk/by-id/ata-MZ7KM1T6HAJM00D3_S2CXNAAH600026 extsz=4096 blocks=390703446, rtextents=390703446 Discarding blocks...Done. Discarding blocks...Done.It did not fishish after 10 mins.
# pstack 5785 #0 0x00007f8df5efc01a in pread64 () from /lib64/libc.so.6 #1 0x0000557b7bbfe9c3 in __read_buf.constprop.0 () #2 0x0000557b7bbc3030 in libxfs_buf_read_map () #3 0x0000557b7bbfe1ae in libxfs_trans_read_buf_map.constprop () #4 0x0000557b7bbee7ef in xfs_rtbuf_get () #5 0x0000557b7bbf03d8 in libxfs_rtfree_extent () #6 0x0000557b7bbbb322 in parseproto.lto_priv () #7 0x0000557b7bbb7643 in main ()Quoted 6 lines
Quoted 3 lines
And,
1) more chance happen when kernel 6.12.36, but yet not happen when kernel 6.6.93.
2) it happen on both xfsprogs-6.11.0 and xfsprogs-6.4.You /could/ strace the mkfs process to see if it's really stuck, or just issuing IOs really slowly.
Today the mkfs.xfs with rtdev all finished in 110s-160s on kernel 6.6.93/6.12.38.
Quoted 8 lines
the result of strace show that
pread64()with 4K size so it is just slow.pread64(3, "\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377"..., 4096, 868352) = 4096 pread64(3, "\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377"..., 4096, 864256) = 4096 pread64(3, "\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377"..., 4096, 860160) = 4096 pread64(3, "\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377"..., 4096, 856064) = 4096 pread64(3, "\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377"..., 4096, 851968) = 4096
the offset of pread64() is in DECE order, not in ASCE order , so the readahead does not help too.
Best Regards
Wang Yugui ([email protected])
2025/07/18
Quoted 6 lines
Thanks a lot.
Best Regards
Wang Yugui ([email protected])
2025/07/18