Skip to contentSkip to conversation

SanDisk Extreme Pro w/btrfs Frozen: nodiscard per Western Digital

DISCUSSION John L. Poole 1 participant 1 message Oct 13, 2023 1 message not in the archive Btrfs file system

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

[My replies to Damien Le Moal's reply inline below. This email is my second attempt to respond in "text" format, my prior response was accidentally in HTML and rejected by the mail server.]

Quoted 151 lines · Damien Le Moal
Quoted 150 lines · John L. Poole

** Issue: I have 3 SanDisk Extreme Pro micro SD cards using btrfs which are readable, but not writable; they are frozen.

** Background:  I purchased from Amazon (1) 5 SanDisk Extreme Pro (64GB) microSDXC(tm) UHS-I cards in August, 2023.  They have been used to test/build-out the GenPi64 Gentoo Linux for the Raspberry Pi(2). I selected the Extreme Pro series because they had faster I/O ratings. This is the first time I have used the Extreme Pro series, previously, I have used other cards including the SanDisk Ultra and I have not experienced any problems.

The GenPi64 image uses the btrfs file system.  /etc/fstab has:

PARTUUID="426c28a1-02"  /       btrfs   noatime, [LINE BREAK] compress=zstd,ssd,discard,x-systemd.growfs      0       0

During my build-out I ran into the problem that my session's file changes did not persist on reboots.  I'd install a Gentoo package with "emerge", and/or I'd create/edit a file, and after reboot those changes were missing.  Although dmesg revealed some warnings, there was nothing in the start-up post or system to suggests there was any problem.  So a non-technical user would not be alerted to any problems and on reboot suddenly and without fanfare learn that the files from the previous session are not saved.  This would be btrfs doing its job: not integrating that last set of changes because it detected a corruption.

My build-out has been on a Raspberry Pi 4B with 8 MB(3) and the Raspberry Pi forums suggest that the SD card I/O has a limitation of 50 MBs/second.(4)

What seemed to trigger the problem was when I emerged the package sys-kernel/raspberrypi-sources.  During the processing, there are over 1 GB of files that are to be staged and the emerge would fail during the final transfer of those files.

Here's a lsblk output:

eos /tmp # date; lsblk --output NAME,FSSIZE,FSUSED,FSTYPE,FSVER,LABEL,UUID,MODE,STATE,DISC-ALN Thu Oct 12 10:44:29 PDT 2023
NAME        FSSIZE FSUSED FSTYPE FSVER LABEL
UUID                                 MODE       STATE DISC-ALN mmcblk0 brw-rw----              0
+-mmcblk0p1               vfat   FAT16 932D-ABE2 brw-rw----        3145728
+-mmcblk0p2  59.2G   2.2G btrfs d15fd42d-7572-4d9f-b5fb-eadbe3d70d5c brw-rw----              0

I do not know what DISC-ALN means other than "discard alignment offset" which "lsblk --help" tells me, but share the following. The GenPi64 image that I burned was a 4GB file. When I boot up, there is a script at boot-up which will expand the root (btrfs) file system to use up any available space on the card, hence a one-time expansion occurs changing the root file system from its less than 4 GB size to 59.2 GB. The DISC-ALN value for the vfat system, 3145728, concerns me since the vfat is suppose to be 256 MBs. I'm just wondering if that 3145728 value is somewhat in interference with the expanded btrfs system and may be contributing to the frozen read-only mode of the card.

I began to use the "emerge sys-kernel/raspberrypi-sources" as a test which would cause the SD card to become unwritable.

I opened a ticket with Western Digital and their response was:

    [--- start response ---]

    Please also refer to the following command to run BEFORE doing any major writes to card after installing OS.

    Using the appropriate format command can avoid the Read-Only mode issue. File system format tools such as mkfs and mke2fs have the nodiscard flag.

    Format SanDisk card with ext4 FS:

      Linux Host is needed
      Native microSD/SD support.
      Discard feature supported on Host and card Use the nodiscard flag.
      Command: $ mkfs.ext4 -E nodiscard -F /dev/mmcblk0p1 [--- end response ---]

I altered my /etc/fstab on a newly burned enhanced image and changed "discard" to "nodiscard" and I successfully installed "sys-kernel/raspberrypi-sources".  Furthermore, subsequent activity on the image seems normal and it appears the mount specification of "nodiscard" has solved my problem... for a micro disk not already frozen.

Western Digital has offered to replace whatever disks with like kind; I had suggested they simply provide me Ultras and refund the price difference, but they insist that they can only provide Extreme Pro replacements. Therefore, I am stuck with five Extreme Pro Cards that have
a propensity to become unusable.

I thought I would try to resurrect the three frozen cards before resolving my open ticket with Western Digital.  To that end and working with the principal developer of the GenPi64 project, we devised these steps to test whether the cards could be salvaged:

     # need to be root to write to card
     su
     # create a test file of 512 zeros
     date; time dd if=/dev/zero of=/tmp/1_block-512_zeros.raw count=1
     # write test file to card
     date; time dd if=/tmp/1_block-512_zeros.raw  of=/dev/mmcblk0
     # copy the first 512 bytes out to a file
     date; time dd if=/dev/mmcblk0 of=/tmp/1_block-512_from_card_A.raw count=1
     # view the files with hexdump abbreviate with head
     hexdump 1_block-512_from_card_A.raw |head -n 1
     hexdump 1_block-512_zeros.raw | head -n 1

    eos /tmp #      # create a test file of 512 zeros date; time dd if=/dev/zero of=/tmp/1_block-512_zeros.raw count=1 Thu Oct 12 10:26:11 PDT 2023
    1+0 records in
    1+0 records out
    512 bytes copied, 8.6744e-05 s, 5.9 MB/s

    real    0m0.001s
    user    0m0.000s
    sys     0m0.001s
    eos /tmp #      # write test file to card
    date; time dd if=/tmp/1_block-512_zeros.raw  of=/dev/mmcblk0 Thu Oct 12 10:26:23 PDT 2023
    1+0 records in
    1+0 records out
    512 bytes copied, 0.0211458 s, 24.2 kB/s

    real    0m0.023s
    user    0m0.002s
    sys     0m0.000s
    eos /tmp #      # copy the first 512 bytes out to a file date; time dd if=/dev/mmcblk0 of=/tmp/1_block-512_from_card_A.raw count=1 Thu Oct 12 10:26:34 PDT 2023
    1+0 records in
    1+0 records out
    512 bytes copied, 0.0202807 s, 25.2 kB/s

    real    0m0.022s
    user    0m0.002s
    sys     0m0.000s
    eos /tmp #      # view the files with hexdump abbreviate with head hexdump 1_block-512_from_card_A.raw |head -n 1
    hexdump 1_block-512_zeros.raw | head -n 1
    0000000 b8fa 1000 d08e 00bc b8b0 0000 d88e c08e
    0000000 0000 0000 0000 0000 0000 0000 0000 0000
    eos /tmp #

Quoted 4 lines
Quoted 2 lines

The above shows that my attempts to write to the card fail; both rows should be zeros.

Anything in dmesg ? Any IO error message showing up there ? For these tests or when using btrfs ?

I do not want to flood this list, but I will leave it to Mr. Le Moal to determine what is appropriate for inclusion.  The URL (which probably will not work for the general public) and identifier to my "incident" [Bug/ticket] with Western Digital is:
https://support-en.wd.com/app/account/questions/detail/i_id/7298003

In my incident at 28-Sep-2023 11:00:00 AM, I provided this pastebin URL:
https://pastebin.com/mPv4fNgH
which contains dmesg output.  See especially line 69.
I regret that I do not have the dmesg output from the
moment the problem occurred as I rebooted before I realized this was something more than a problem with the image/build.

As you will see in the incident report, I have provided a link to the GenPi64 system image so Western Digital can burn their own disks to experiment with and other helpful information
to allow engineers to assess and test with.

I also posted to Gentoo's Forum:

"Help troubleshooting btrfs" at:
https://forums.gentoo.org/viewtopic-t-1165138-highlight-.html

This problem has really thrown me to the point that I am having problem retaining all the avenues I investigated to try and move forward with confidence.  You'll see in my incident a substantial delay and even notice of closure which I am guessing was AI generated -- I ended up asking for a human being to review what I had submitted.

Quoted 15 lines
Quoted 6 lines

Moreover, I've tried using fdisk and deleting the partitions and writing my changes and fdisk appears to execute the task without error or warning, yet the partitions persist and were not deleted.

There are currently pending 3 class action lawsuits against Western Digital over their SSD drives where customers would experience lost data. (5)

That is irrelevant information here. Not the same product nor even the same device class.

Quoted 6 lines

Western Digital has a web page "Format Using Linux, Raspberry Pi, or Android Host Results in Read Only Mode on a SanDisk SD or microSD Memory Card "(6).

** Questions:

Quoted 7 lines
Quoted 2 lines

1) Is there some way to write to the card, or some tools I can use so I can resurrect the cards and restore them to normal use?

This is a device bug. What (if anything) can be done about it depends on the device FW. For extreme problems, the card could be "bricked". So I suggest you accept the offer to get replacements and use nodiscard option if you get the exact same model, or use discard if you get a fixed version (the vendor will have that information).

I will request five replacements even though two currently work, one of these two had "discard" specified in several sessions so my opinion is that it could be compromised until someone from Western Digital can state with certainty is not.  And the remaining last card did boot
initially with the "discard" in the /etc/fstab until I could change the value to "nodiscard" in the initial session.

The GenPi64 developer believes inclusion of "discard" as a mount option is desirable, so I am going to recommend in an Issue filed with that project that users not use SanDisk Extreme Pro micro cards until word from Western Digital comes forth stating that the Extreme Pro micro cards are not subject to freezing due to the presence of "discard" in the image's /etc/fstab. Moreover, I will suggest users using SanDisk Extreme Pro micros change the "discard" to "nodiscard".

Quoted 25 lines
Quoted 2 lines

2) It looks like the use "discard" for SanDisk Extreme Pro cards causes problems?  I am told btrfs uses discard by default.

Yes.

Quoted 2 lines

3) Is there some assay anyone wants to suggest I try to see if I can trigger the read-only condition such as I have stumbled upon three times?

Given that this seems to lock the drives in read-only mode, I would not recommend that. You will not be able to repeat that test without a method to get out of the RO state. And without being able to repeat, the test value will be limited.

Quoted 3 lines

I have 2 cards that I can still write to.  I am willing to try anything someone suggests that would help shed light onto this problem.  I was hoping to garner the attention of an engineer from Western Digital.

You have. But I am not developing SD card firmware, so no clue. But again, I do not recommend trying to trigger the issue. WD seems to be aware of the problem, so it is understood and being worked on.

Quoted 7 lines

Before I accept Western Digital's offer to replace the cards, I'd like to know what is happening or what is causing them to lock up.  I do not want to encounter the problem again. What is troubling is that
an inexperienced user will not detect anything wrong until after a reboot when they
discover everything from the previous session did not persist. This kind of media failure could discourage the use of the btrfs based image.

Quoted 7 lines

The issue does not seem to be limited to btrfs but rather to the use of discard commands. I am not aware of the exact issue or its trigger and even if I were, I would likely not be able to describe it in much details without revealing confidential information about the device design...

I will do some poking around internally to check the status of this.

[Footnotes to my original posting were ommitted the teply and may be accessed at:
https://marc.info/?l=linux-btrfs&m=169714218815709&w=2
]

Thank you,

John Poole
bcc: Damian Le Moal