Skip to contentSkip to conversation

BTRFS error count 754 after reboot on Debian kernel 6.12.17

DISCUSSION Russell Coker 3 participants 9 messages Mar 22, 2025 – Apr 2, 2025 Btrfs file system

lore
Share & tools
lore
#0 lore raw

[/dev/sdd1].write_io_errs 753
[/dev/sdd1].read_io_errs 1

I have a test system which has a strange problem where the BTRFS error count on one device (out of four) goes to 754 after a reboot.

There are no BTRFS errors in the kernel message log after booting up. There are no log entries in /var/log/kern.log about BTRFS issues. When I look at the console as it's shutting down I don't see any errors being logged, so either there are no errors logged or there are 753 errors logged in the final split second before power off or reboot so that I don't even see them.

This is repeatable and it's 754 every time.

After I get the error I remove the device from the array and add it again. I can run it for days without problem with data being written to that device and read from it without error.

But when I reboot it says 754 errors. When I swapped that device with another one in a different drive bay the same device has errors and the other device doesn't. So it's not related to the drive bay it's related to the SSD.

The system is a Dell PowerEdge T630.

The SSD could have a fault, but if so why does it only show up on reboot and why 754 errors every time?

Signature · 2 lines
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/
#1 lore raw
Quoted 5 lines · Russell Coker

[/dev/sdd1].write_io_errs 753
[/dev/sdd1].read_io_errs 1

I have a test system which has a strange problem where the BTRFS error count on one device (out of four) goes to 754 after a reboot.

Mind to provide the full dmesg just in case?

It's better to cover the initial device removal/add to be extra safe.

Quoted 10 lines

There are no BTRFS errors in the kernel message log after booting up. There are no log entries in /var/log/kern.log about BTRFS issues. When I look at the console as it's shutting down I don't see any errors being logged, so either there are no errors logged or there are 753 errors logged in the final split second before power off or reboot so that I don't even see them.

This is repeatable and it's 754 every time.

After I get the error I remove the device from the array and add it again.

How did you do the removal and add?

"btrfs device remove" then "btrfs device add"? Or just power the machine down and physically add/remove the device?

In the later case it won't reset the internal error count inside btrfs.

Quoted 7 lines

I
can run it for days without problem with data being written to that device and read from it without error.

But when I reboot it says 754 errors. When I swapped that device with another one in a different drive bay the same device has errors and the other device doesn't. So it's not related to the drive bay it's related to the SSD.

Again, how did you do the swap?

Quoted 6 lines

The system is a Dell PowerEdge T630.

The SSD could have a fault, but if so why does it only show up on reboot and why 754 errors every time?

The error counters are stored inside the fs, it records all the history errors a device hit in the past.

You need to inform btrfs by either proper btrfs device removal/add, or make btrfs to zero the counters.

Thanks,
Qu

#2 lore raw
Quoted 5 lines · On Sunday, 23 March 2025 08:36:56 AEDT Qu Wenruo wrote:
Quoted 3 lines

This is repeatable and it's 754 every time.

After I get the error I remove the device from the array and add it again.

How did you do the removal and add?

btrfs dev rem ...
btrfs dev add ...

Quoted 2 lines

"btrfs device remove" then "btrfs device add"? Or just power the machine down and physically add/remove the device?

I didn't physically remove the devices because working out which device is which /dev/sd node is not easy at all. For unrelated reasons the assignment of /dev/sd nodes is apparently random.

Quoted 10 lines
Quoted 4 lines

The system is a Dell PowerEdge T630.

The SSD could have a fault, but if so why does it only show up on reboot and why 754 errors every time?

The error counters are stored inside the fs, it records all the history errors a device hit in the past.

You need to inform btrfs by either proper btrfs device removal/add, or make btrfs to zero the counters.

Yes I had done the proper device removal, but either the counters weren't properly reset or some other errors were occuring. I even tried dding 1G of data from /dev/zero over the device after removal and that didn't help.

The problem has gone away now. I was about to do another run to get dmesg output for you but accidentally removed the wrong device. After I added that back then removed and re-added the correct device the problem went away. I can now reboot without seeing an error count of 754. I presume that forcing a bunch of metadata to be moved around changed things somehow.

Signature · 2 lines
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/
#3 lore raw
In reply to Russell Coker
Quoted 18 lines · Russell Coker

[/dev/sdd1].write_io_errs 753
[/dev/sdd1].read_io_errs 1

I have a test system which has a strange problem where the BTRFS error count on one device (out of four) goes to 754 after a reboot.

There are no BTRFS errors in the kernel message log after booting up. There are no log entries in /var/log/kern.log about BTRFS issues. When I look at the console as it's shutting down I don't see any errors being logged, so either there are no errors logged or there are 753 errors logged in the final split second before power off or reboot so that I don't even see them.

This is repeatable and it's 754 every time.

After I get the error I remove the device from the array and add it again. I can run it for days without problem with data being written to that device and read from it without error.

Quoted 8 lines

But when I reboot it says 754 errors. When I swapped that device with another one in a different drive bay the same device has errors and the other device doesn't. So it's not related to the drive bay it's related to the SSD.

The system is a Dell PowerEdge T630.

The SSD could have a fault, but if so why does it only show up on reboot and why 754 errors every time?

These are likely old errors. You'd need to check old logs to see when the write errors occurred. These statistics are just a counter. You can reset them with btrfs dev stats -z and they'll go back to zero.

It's simple counter. It could be 754 errors seen one time. Or it could be `1 error seen 754 times. Or any combination of multiple errors multiple times adding up to 754 errors.

Signature · 1 line
Chris Murphy
#4 lore raw
Quoted 7 lines · On Tuesday, 1 April 2025 15:04:20 AEDT Chris Murphy wrote:

These are likely old errors. You'd need to check old logs to see when the write errors occurred. These statistics are just a counter. You can reset them with btrfs dev stats -z and they'll go back to zero.

It's simple counter. It could be 754 errors seen one time. Or it could be `1 error seen 754 times. Or any combination of multiple errors multiple times adding up to 754 errors.

Is "btrfs dev stats -z" covered by removing the device from the set and adding it again? If so I did that but it kept recurring. The fact that the error count was there in the first place wasn't the unexpected thing, it was the fact that it kept coming back and had no log entries about it.

Signature · 2 lines
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/
#5 lore raw
Quoted 5 lines · Russell Coker
Quoted 4 lines · On Tuesday, 1 April 2025 15:04:20 AEDT Chris Murphy wrote:

These are likely old errors. You'd need to check old logs to see when the write errors occurred. These statistics are just a counter. You can reset them with btrfs dev stats -z and they'll go back to zero.

Quoted 8 lines
Quoted 3 lines

It's simple counter. It could be 754 errors seen one time. Or it could be `1 error seen 754 times. Or any combination of multiple errors multiple times adding up to 754 errors.

Is "btrfs dev stats -z" covered by removing the device from the set and adding it again? If so I did that but it kept recurring. The fact that the error count was there in the first place wasn't the unexpected thing, it was the fact that it kept coming back and had no log entries about it.

Removing it with a btrfs command? Or physically disconnecting and reconnecting?

The statistics are per device, persistently stored in the device b-tree which is metadata block group. So this metadata could be on any device in a multiple device Btrfs, not necessarily on the device that produced the errors.

I'd like to think upon btrfs device remove or btrfs replace the device's stats are also removed from dev tree. But I haven' tested it, and I'm not sure what the code says should happen.

Signature · 1 line
Chris Murphy
#6 lore raw
Quoted 10 lines · On Wednesday, 2 April 2025 03:00:33 AEDT Chris Murphy wrote:
Quoted 9 lines · Russell Coker
Quoted 7 lines · On Tuesday, 1 April 2025 15:04:20 AEDT Chris Murphy wrote:

These are likely old errors. You'd need to check old logs to see when the write errors occurred. These statistics are just a counter. You can reset them with btrfs dev stats -z and they'll go back to zero.

It's simple counter. It could be 754 errors seen one time. Or it could be `1 error seen 754 times. Or any combination of multiple errors multiple times adding up to 754 errors.

Quoted 7 lines
Quoted 4 lines

Is "btrfs dev stats -z" covered by removing the device from the set and adding it again? If so I did that but it kept recurring. The fact that the error count was there in the first place wasn't the unexpected thing, it was the fact that it kept coming back and had no log entries about it.

Removing it with a btrfs command? Or physically disconnecting and reconnecting?

btrfs commands.

Quoted 4 lines

The statistics are per device, persistently stored in the device b-tree which is metadata block group. So this metadata could be on any device in a multiple device Btrfs, not necessarily on the device that produced the errors.

It should be on the device itself and once the device is subject to a btrfs dev rem command it should be gone for good.

Quoted 3 lines

I'd like to think upon btrfs device remove or btrfs replace the device's stats are also removed from dev tree. But I haven' tested it, and I'm not sure what the code says should happen.

After doing the btrfs dev add it reports 0 errors until after reboot.

Signature · 2 lines
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/
#7 lore raw
Quoted 26 lines · Russell Coker
Quoted 10 lines · On Wednesday, 2 April 2025 03:00:33 AEDT Chris Murphy wrote:
Quoted 9 lines · Russell Coker
Quoted 7 lines · On Tuesday, 1 April 2025 15:04:20 AEDT Chris Murphy wrote:

These are likely old errors. You'd need to check old logs to see when the write errors occurred. These statistics are just a counter. You can reset them with btrfs dev stats -z and they'll go back to zero.

It's simple counter. It could be 754 errors seen one time. Or it could be `1 error seen 754 times. Or any combination of multiple errors multiple times adding up to 754 errors.

Quoted 7 lines
Quoted 4 lines

Is "btrfs dev stats -z" covered by removing the device from the set and adding it again? If so I did that but it kept recurring. The fact that the error count was there in the first place wasn't the unexpected thing, it was the fact that it kept coming back and had no log entries about it.

Removing it with a btrfs command? Or physically disconnecting and reconnecting?

btrfs commands.

Quoted 4 lines

The statistics are per device, persistently stored in the device b-tree which is metadata block group. So this metadata could be on any device in a multiple device Btrfs, not necessarily on the device that produced the errors.

Quoted 8 lines

It should be on the device itself and once the device is subject to a btrfs dev rem command it should be gone for good.

Quoted 3 lines

I'd like to think upon btrfs device remove or btrfs replace the device's stats are also removed from dev tree. But I haven' tested it, and I'm not sure what the code says should happen.

After doing the btrfs dev add it reports 0 errors until after reboot.

Uhh, well I'm confused then.

What happens if you use btrfs dev stats -z $MNT and then reboot?

Signature · 1 line
Chris Murphy
#8 lore raw
Quoted 1 line · On Wednesday, 2 April 2025 13:42:32 AEDT Chris Murphy wrote:

What happens if you use btrfs dev stats -z $MNT and then reboot?

I didn't try that and after I accidentally removed a different device the problem stopped recurring so I can't try that now.

Signature · 2 lines
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/