Custom driver FS brokenness at 4GB?

DISCUSSION Rob Harris 3 participants 4 messages May 27, 2015 – May 28, 2015

Share & tools

Rob Harris

author

2015-05-27 13:56 UTC #0 lore raw

Greetings. I have an odd issue and need some ideas of where to go next -- I'm out of hair to rip out.

I'm writing a custom block device driver talking to some custom RAID hardware (>32TB) using DMA scatter-gather, with no partitions and am using make_request() to service all the BIO requests to simplify debugging. I have the driver working to the point where using DD against the block device seems to work fine (I'm setting iflag|oflag=direct to ensure it's writing to the disk). I also have the blk_queue set to only request a single 4k I/O per BIO (again to simplify debugging for now.) Also, again to debug, I have a mutex wrapping the entire make_request call to ensure that only a single request is being serviced at a time. So, this should be as "simple" as I can make the environment to debug this problem.

Once the driver is loaded, when I try to create a file system (ext4 but the same thing happens with xfs) it seems like there is some corruption occurring, but only when I set the sector size of the block device over 4GB. For instance, when I set the size to 4G, I can mkfs.ext4, but after 2 or 3 mount/umounts the FS refuses to mount anymore and the kernel log complains that the journal is missing. This was discovered running this loop...

#!/bin/sh
COUNT=4032

while [ 1 ] ; do

figlet ${COUNT}

( umount /mnt ; rmmod smc ) || true
modprobe smc capacity_in_mb=${COUNT} debug=1
mkfs.ext4 -m 0 /dev/smcd

mount /dev/smcd /mnt
cp count_512m.dat /mnt/test
umount /mnt
mount /dev/smcd /mnt
umount /mnt
mount /dev/smcd /mnt
cmp count_512m.dat /mnt/test
umount /mnt
mount /dev/smcd /mnt # ***
sync
umount /mnt
mount /dev/smcd /mnt
sleep 1
umount /mnt

COUNT=$(( COUNT + 64 ))
sleep 1

done

Sometimes I'll get in the kernel log:

May 27 09:39:01 febtober kernel: [64547.304695] EXT4-fs (smcd): ext4_check_descriptors: Checksum for group 0 failed (7009!=0)
May 27 09:39:01 febtober kernel: [64547.305744] EXT4-fs (smcd): group descriptors corrupted!

Others I'll get:

May 27 09:46:49 ryftone-smcdrv kernel: [65014.342850] EXT4-fs (smcd): no journal found

I've seen this loop fail as early as COUNT=4096, but as late as COUNT=4220; removing the sync changes the behavior. When it fails, it usually does so on the 3rd mount (***).
FYI, I effectively call: set_capacity( disk, capacity_in_mb * 2048 ); ( 2048 * 512b (kernel sector) = 1M )

Another example: if I set the sector count of the disk to 16G, I can run mkfs.ext4 but the first mount fails and I see May 27 09:07:27 febtober kernel: [62653.269387] EXT4-fs (smcd): ext4_check_descriptors: Block bitmap for group 0 not in group (block 4294967295)!

But, again, if I set the sector size < 4G, everything seems fine. I can currently DD read and write across that 4G boundary without issue -- it's ONLY the filesystem accesses. My gut is screaming there's 32/64 bit overflow condition somewhere but for the life of me I can't find it. Is there something I need to set to tell the block layer I have a 64-bit addressible device? set_capacity is always the number of LINUX KERNEL sectors (not what I set blk_queue_logical|physical_block_size to) correct?

I'm currently on 3.16.0 (Ubuntu 14.04.2 LTS) if it matters.

Any help/pointers would be greatly appreciated.

--Rob Harris

Jan Kara

2015-05-28 10:59 UTC #1 lore raw

Quoted 15 lines · Rob Harris

Greetings. I have an odd issue and need some ideas of where to go next -- I'm out of hair to rip out.
I'm writing a custom block device driver talking to some custom RAID hardware (>32TB) using DMA scatter-gather, with no partitions and am using make_request() to service all the BIO requests to simplify debugging. I have the driver working to the point where using DD against the block device seems to work fine (I'm setting iflag|oflag=direct to ensure it's writing to the disk). I also have the blk_queue set to only request a single 4k I/O per BIO (again to simplify debugging for now.) Also, again to debug, I have a mutex wrapping the entire make_request call to ensure that only a single request is being serviced at a time. So, this should be as "simple" as I can make the environment to debug this problem.

Quoted 7 lines

Once the driver is loaded, when I try to create a file system (ext4 but the same thing happens with xfs) it seems like there is some corruption occurring, but only when I set the sector size of the block device over 4GB. For instance, when I set the size to 4G, I can mkfs.ext4, but after 2 or 3 mount/umounts the FS refuses to mount anymore and the kernel log complains that the journal is missing. This was discovered running this loop...

Hard to tell exactly but with 4GB being 32-bit limit, I would first look
for some int / unsigned int number overflow. You could possibly better debug this when writing some pattern via DD that is different for each block to verify that each block indeed lands in the expected location...

Honza

Quoted 74 lines

#!/bin/sh
COUNT=4032
while [ 1 ] ; do
figlet ${COUNT}
( umount /mnt ; rmmod smc ) || true
modprobe smc capacity_in_mb=${COUNT} debug=1
mkfs.ext4 -m 0 /dev/smcd
mount /dev/smcd /mnt
cp count_512m.dat /mnt/test
umount /mnt
mount /dev/smcd /mnt
umount /mnt
mount /dev/smcd /mnt
cmp count_512m.dat /mnt/test
umount /mnt
mount /dev/smcd /mnt # ***
sync
umount /mnt
mount /dev/smcd /mnt
sleep 1
umount /mnt
COUNT=$(( COUNT + 64 ))
sleep 1
done
Sometimes I'll get in the kernel log:
May 27 09:39:01 febtober kernel: [64547.304695] EXT4-fs (smcd):
ext4_check_descriptors: Checksum for group 0 failed (7009!=0)
May 27 09:39:01 febtober kernel: [64547.305744] EXT4-fs (smcd):
group descriptors corrupted!
Others I'll get:
May 27 09:46:49 ryftone-smcdrv kernel: [65014.342850] EXT4-fs
(smcd): no journal found
I've seen this loop fail as early as COUNT=4096, but as late as COUNT=4220; removing the sync changes the behavior. When it fails, it usually does so on the 3rd mount (***). FYI, I effectively call: set_capacity( disk, capacity_in_mb * 2048 ); ( 2048 * 512b (kernel sector) = 1M )
Another example: if I set the sector count of the disk to 16G, I can run mkfs.ext4 but the first mount fails and I see May 27 09:07:27 febtober kernel: [62653.269387] EXT4-fs (smcd):
ext4_check_descriptors: Block bitmap for group 0 not in group (block 4294967295)!
But, again, if I set the sector size < 4G, everything seems fine. I can currently DD read and write across that 4G boundary without issue -- it's ONLY the filesystem accesses. My gut is screaming there's 32/64 bit overflow condition somewhere but for the life of me I can't find it. Is there something I need to set to tell the block layer I have a 64-bit addressible device? set_capacity is always the number of LINUX KERNEL sectors (not what I set blk_queue_logical|physical_block_size to) correct?
I'm currently on 3.16.0 (Ubuntu 14.04.2 LTS) if it matters.
Any help/pointers would be greatly appreciated.
--Rob Harris
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines

Jan Kara <[email protected]>
SUSE Labs, CR

Andreas Dilger

2015-05-28 17:43 UTC #2 lore raw

Quoted 24 lines · Jan Kara

Quoted 15 lines · Rob Harris
Greetings. I have an odd issue and need some ideas of where to go next -- I'm out of hair to rip out.
I'm writing a custom block device driver talking to some custom RAID hardware (>32TB) using DMA scatter-gather, with no partitions and am using make_request() to service all the BIO requests to simplify debugging. I have the driver working to the point where using DD against the block device seems to work fine (I'm setting iflag|oflag=direct to ensure it's writing to the disk). I also have the blk_queue set to only request a single 4k I/O per BIO (again to simplify debugging for now.) Also, again to debug, I have a mutex wrapping the entire make_request call to ensure that only a single request is being serviced at a time. So, this should be as "simple" as I can make the environment to debug this problem.
Quoted 7 lines
Once the driver is loaded, when I try to create a file system (ext4 but the same thing happens with xfs) it seems like there is some corruption occurring, but only when I set the sector size of the block device over 4GB. For instance, when I set the size to 4G, I can mkfs.ext4, but after 2 or 3 mount/umounts the FS refuses to mount anymore and the kernel log complains that the journal is missing. This was discovered running this loop...

Quoted 4 lines

Hard to tell exactly but with 4GB being 32-bit limit, I would first look
for some int / unsigned int number overflow. You could possibly better debug this when writing some pattern via DD that is different for each block to verify that each block indeed lands in the expected location...

We have a tool "llverdev" which does exactly this - write a pattern to each block in the block device (or in sparse regions covering the device) with a timestamp and block number to track down sources of block addressing errors:

http://git.hpdd.intel.com/fs/lustre-release.git/blob/HEAD:/lustre/utils/llverdev.c

Cheers, Andreas

Quoted 67 lines

Honza
Quoted 66 lines
#!/bin/sh
COUNT=4032
while [ 1 ] ; do
figlet ${COUNT}
( umount /mnt ; rmmod smc ) || true
modprobe smc capacity_in_mb=${COUNT} debug=1
mkfs.ext4 -m 0 /dev/smcd
mount /dev/smcd /mnt
cp count_512m.dat /mnt/test
umount /mnt
mount /dev/smcd /mnt
umount /mnt
mount /dev/smcd /mnt
cmp count_512m.dat /mnt/test
umount /mnt
mount /dev/smcd /mnt # ***
sync
umount /mnt
mount /dev/smcd /mnt
sleep 1
umount /mnt
COUNT=$(( COUNT + 64 ))
sleep 1
done
Sometimes I'll get in the kernel log:
May 27 09:39:01 febtober kernel: [64547.304695] EXT4-fs (smcd):
ext4_check_descriptors: Checksum for group 0 failed (7009!=0)
May 27 09:39:01 febtober kernel: [64547.305744] EXT4-fs (smcd):
group descriptors corrupted!
Others I'll get:
May 27 09:46:49 ryftone-smcdrv kernel: [65014.342850] EXT4-fs
(smcd): no journal found
I've seen this loop fail as early as COUNT=4096, but as late as COUNT=4220; removing the sync changes the behavior. When it fails, it usually does so on the 3rd mount (***). FYI, I effectively call: set_capacity( disk, capacity_in_mb * 2048 ); ( 2048 * 512b (kernel sector) = 1M )
Another example: if I set the sector count of the disk to 16G, I can run mkfs.ext4 but the first mount fails and I see May 27 09:07:27 febtober kernel: [62653.269387] EXT4-fs (smcd):
ext4_check_descriptors: Block bitmap for group 0 not in group (block 4294967295)!
But, again, if I set the sector size < 4G, everything seems fine. I can currently DD read and write across that 4G boundary without issue -- it's ONLY the filesystem accesses. My gut is screaming there's 32/64 bit overflow condition somewhere but for the life of me I can't find it. Is there something I need to set to tell the block layer I have a 64-bit addressible device? set_capacity is always the number of LINUX KERNEL sectors (not what I set blk_queue_logical|physical_block_size to) correct?
I'm currently on 3.16.0 (Ubuntu 14.04.2 LTS) if it matters.

Quoted 15 lines

Quoted 8 lines
Any help/pointers would be greatly appreciated.
--Rob Harris
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 6 lines
Jan Kara <[email protected]>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Cheers, Andreas

Rob Harris

author

2015-05-28 18:31 UTC #3 lore raw

Thanks for the pointers everyone. After further testing and code review, I was boneheadedly truncating a u64 to a u32 for the sector address as part of a function signature with an obscured typedef.

*facepalm*

All seems well now. Thanks for the help!
-R

Quoted 125 lines · Andreas Dilger

Quoted 23 lines · Jan Kara
Quoted 15 lines · Rob Harris
Greetings. I have an odd issue and need some ideas of where to go next -- I'm out of hair to rip out.
I'm writing a custom block device driver talking to some custom RAID hardware (>32TB) using DMA scatter-gather, with no partitions and am using make_request() to service all the BIO requests to simplify debugging. I have the driver working to the point where using DD against the block device seems to work fine (I'm setting iflag|oflag=direct to ensure it's writing to the disk). I also have the blk_queue set to only request a single 4k I/O per BIO (again to simplify debugging for now.) Also, again to debug, I have a mutex wrapping the entire make_request call to ensure that only a single request is being serviced at a time. So, this should be as "simple" as I can make the environment to debug this problem.
Quoted 7 lines
Once the driver is loaded, when I try to create a file system (ext4 but the same thing happens with xfs) it seems like there is some corruption occurring, but only when I set the sector size of the block device over 4GB. For instance, when I set the size to 4G, I can mkfs.ext4, but after 2 or 3 mount/umounts the FS refuses to mount anymore and the kernel log complains that the journal is missing. This was discovered running this loop...
Quoted 4 lines
Hard to tell exactly but with 4GB being 32-bit limit, I would first look
for some int / unsigned int number overflow. You could possibly better debug this when writing some pattern via DD that is different for each block to verify that each block indeed lands in the expected location...
We have a tool "llverdev" which does exactly this - write a pattern to each block in the block device (or in sparse regions covering the device) with a timestamp and block number to track down sources of block addressing errors:
http://git.hpdd.intel.com/fs/lustre-release.git/blob/HEAD:/lustre/utils/llverdev.c
Cheers, Andreas
Quoted 66 lines
Honza
Quoted 65 lines
#!/bin/sh
COUNT=4032
while [ 1 ] ; do
figlet ${COUNT}
( umount /mnt ; rmmod smc ) || true
modprobe smc capacity_in_mb=${COUNT} debug=1
mkfs.ext4 -m 0 /dev/smcd
mount /dev/smcd /mnt
cp count_512m.dat /mnt/test
umount /mnt
mount /dev/smcd /mnt
umount /mnt
mount /dev/smcd /mnt
cmp count_512m.dat /mnt/test
umount /mnt
mount /dev/smcd /mnt # ***
sync
umount /mnt
mount /dev/smcd /mnt
sleep 1
umount /mnt
COUNT=$(( COUNT + 64 ))
sleep 1
done
Sometimes I'll get in the kernel log:
May 27 09:39:01 febtober kernel: [64547.304695] EXT4-fs (smcd):
ext4_check_descriptors: Checksum for group 0 failed (7009!=0)
May 27 09:39:01 febtober kernel: [64547.305744] EXT4-fs (smcd):
group descriptors corrupted!
Others I'll get:
May 27 09:46:49 ryftone-smcdrv kernel: [65014.342850] EXT4-fs
(smcd): no journal found
I've seen this loop fail as early as COUNT=4096, but as late as COUNT=4220; removing the sync changes the behavior. When it fails, it usually does so on the 3rd mount (***). FYI, I effectively call: set_capacity( disk, capacity_in_mb * 2048 ); ( 2048 * 512b (kernel sector) = 1M )
Another example: if I set the sector count of the disk to 16G, I can run mkfs.ext4 but the first mount fails and I see May 27 09:07:27 febtober kernel: [62653.269387] EXT4-fs (smcd):
ext4_check_descriptors: Block bitmap for group 0 not in group (block 4294967295)!
But, again, if I set the sector size < 4G, everything seems fine. I can currently DD read and write across that 4G boundary without issue -- it's ONLY the filesystem accesses. My gut is screaming there's 32/64 bit overflow condition somewhere but for the life of me I can't find it. Is there something I need to set to tell the block layer I have a 64-bit addressible device? set_capacity is always the number of LINUX KERNEL sectors (not what I set blk_queue_logical|physical_block_size to) correct?
I'm currently on 3.16.0 (Ubuntu 14.04.2 LTS) if it matters.
Quoted 15 lines
Quoted 8 lines
Any help/pointers would be greatly appreciated.
--Rob Harris
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 6 lines
Jan Kara <[email protected]>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Cheers, Andreas