ext4fs error "ext4_mb_generate_buddy:741:group 16, 8160 clusters in bitmap, 4064 in gd" (with repro)

DISCUSSION Paolo Bonzini 3 participants 5 messages Aug 9, 2012 – Aug 16, 2012 SCSI

Share & tools

Theodore Ts'o

2012-08-09 17:06 UTC #1 lore raw

Quoted 7 lines · Paolo Bonzini

Here is how to reproduce it. It happens during fstrim. I found other occurrences of the error in the mailing list, but they were not related to trim so they may be something different.
modprobe scsi_debug dev_size_mb=256 lbpws=1
dd if=/dev/zero of=/dev/sdb bs=1M
fdisk /dev/sdb

Quoted 8 lines

Quoted 1 line
Quoted 1 line
create a new partition accepting all defaults
fdisk -lu /dev/sdb|tail -1
Quoted 1 line
Quoted 1 line
should show: /dev/sdb1 57 524285 262114+ 83 Linux
mkfs.ext4 /dev/sdb1
mkdir test
mount /dev/sdb1 test
fstrim ./test

I can confirm that this accurately reproduces file system corruption using a 3.5 kernel. It looks like some block allocation bitmap blocks is getting trimmed when it shouldn't have been. Lukas, can you take a look at this?

- Ted

Lukáš Czerner

2012-08-15 09:18 UTC #2 lore raw

Quoted 27 lines · Theodore Ts'o

Date: Thu, 9 Aug 2012 13:06:40 -0400
From: Theodore Ts'o <[email protected]>
To: Lukas Czerner <[email protected]>
Cc: Paolo Bonzini <[email protected]>,
"Linux Kernel [email protected] List" <[email protected]>, [email protected]
Subject: Re: ext4fs error
    "ext4_mb_generate_buddy:741:group 16, 8160 clusters in bitmap, 4064 in gd"
     (with repro)
Quoted 7 lines · Paolo Bonzini
Here is how to reproduce it. It happens during fstrim. I found other occurrences of the error in the mailing list, but they were not related to trim so they may be something different.
modprobe scsi_debug dev_size_mb=256 lbpws=1
dd if=/dev/zero of=/dev/sdb bs=1M
fdisk /dev/sdb
Quoted 8 lines
Quoted 1 line
Quoted 1 line
create a new partition accepting all defaults
fdisk -lu /dev/sdb|tail -1
Quoted 1 line
Quoted 1 line
should show: /dev/sdb1 57 524285 262114+ 83 Linux
mkfs.ext4 /dev/sdb1
mkdir test
mount /dev/sdb1 test
fstrim ./test

Quoted 6 lines

I can confirm that this accurately reproduces file system corruption using a 3.5 kernel. It looks like some block allocation bitmap blocks is getting trimmed when it shouldn't have been. Lukas, can you take a look at this?
- Ted

Hi Ted,

sorry for the delay, I've just got back from my vacation. I'll take a look at it.

Thanks!
-Lukas

Lukáš Czerner

2012-08-16 15:21 UTC #3 lore raw

Quoted 46 lines · Lukáš Czerner

Date: Wed, 15 Aug 2012 11:17:57 +0200 (CEST)
From: Lukáš Czerner <[email protected]>
To: Theodore Ts'o <[email protected]>
Cc: Lukas Czerner <[email protected]>, Paolo Bonzini <[email protected]>,
"Linux Kernel [email protected] List" <[email protected]>, [email protected]
Subject: Re: ext4fs error
    "ext4_mb_generate_buddy:741:group 16, 8160 clusters in bitmap, 4064 in gd"
     (with repro)
Quoted 27 lines · Theodore Ts'o
Date: Thu, 9 Aug 2012 13:06:40 -0400
From: Theodore Ts'o <[email protected]>
To: Lukas Czerner <[email protected]>
Cc: Paolo Bonzini <[email protected]>,
"Linux Kernel [email protected] List" <[email protected]>, [email protected]
Subject: Re: ext4fs error
    "ext4_mb_generate_buddy:741:group 16, 8160 clusters in bitmap, 4064 in gd"
     (with repro)
Quoted 7 lines · Paolo Bonzini
Here is how to reproduce it. It happens during fstrim. I found other occurrences of the error in the mailing list, but they were not related to trim so they may be something different.
modprobe scsi_debug dev_size_mb=256 lbpws=1
dd if=/dev/zero of=/dev/sdb bs=1M
fdisk /dev/sdb
Quoted 8 lines
Quoted 1 line
Quoted 1 line
create a new partition accepting all defaults
fdisk -lu /dev/sdb|tail -1
Quoted 1 line
Quoted 1 line
should show: /dev/sdb1 57 524285 262114+ 83 Linux
mkfs.ext4 /dev/sdb1
mkdir test
mount /dev/sdb1 test
fstrim ./test
Quoted 6 lines
I can confirm that this accurately reproduces file system corruption using a 3.5 kernel. It looks like some block allocation bitmap blocks is getting trimmed when it shouldn't have been. Lukas, can you take a look at this?
- Ted

Quoted 7 lines

Hi Ted,
sorry for the delay, I've just got back from my vacation. I'll take a look at it.
Thanks!
-Lukas

This does not seem like an ext4 problem. The code seems unable to actually discard blocks which are allocated. Moreover I was not able to reproduce the problem on the loop device with the same setting as the reported scsi_debug device (1024 bs file system on the 256MB image residing on the 1024 bs filesystem)

After a little bit of tracing with the systemtap and blktrace ext4 does not seem to be doing anything wrong and yet we get part of the block bitmap trimmed. This lead me to the scsi_debug driver itself and indeed it seems that we have off-by-one bug there in the unamp_region() which is causing the problem.

Here is the patch which fixes the problem for me, I'll resend the proper patch in a bit.

drivers/scsi/scsi_debug.c +1 -1

index 182d5a5..f4cc413 100644
@@ -2054,7 +2054,7 @@ static void unmap_region(sector_t lba, unsigned int len)
 		block = lba + alignment;
 		rem = do_div(block, granularity);
 
-		if (rem == 0 && lba + granularity <= end && block < map_size) {
+		if (rem == 0 && lba + granularity < end && block < map_size) {
 			clear_bit(block, map_storep);
 			if (scsi_debug_lbprz)
 				memset(fake_storep +

Thanks!
-Lukas

Theodore Ts'o

2012-08-16 20:00 UTC #4 lore raw

Quoted 6 lines · Lukáš Czerner

After a little bit of tracing with the systemtap and blktrace ext4 does not seem to be doing anything wrong and yet we get part of the block bitmap trimmed. This lead me to the scsi_debug driver itself and indeed it seems that we have off-by-one bug there in the unamp_region() which is causing the problem.

Thanks for finding this --- I was getting scared that ext4 users were losing data in production. It's good to know it was just a bug in the scsi_debug driver.

- Ted